eval-driven-development

Gate LLM prompt changes by running an offline eval suite that measures aggregated pass-rates against a representative dataset, ensuring proposed iterations meet a predefined regression budget before merging. This approach applies to prompt, model, retrieval, and scaffolding changes.

A regression budget in LLM evaluation defines the acceptable threshold of metric drop when iterating on stochastic model outputs. It works with held-out sets and multi-metric reporting to prevent overfitting and mitigate Goodharting during prompt or model updates.

Model-graded evaluations use a language model to judge stochastic LLM outputs against defined criteria, functioning as one of several judgment mechanisms including programmatic, human-graded, and hybrid approaches to account for statistical rather than binary results.

Use human calibration in offline LLM evaluation suites when validating automated or model-graded judgment mechanisms, ensuring the aggregated pass-rate signals accurately reflect system quality and reducing the risk of Goodharting through periodic refresh cycles.

Public benchmarks like MMLU, HumanEval, BIG-bench, GAIA, and MT-Bench provide context and selection grounding, but system-specific offline eval suites with representative datasets and regression budgets are required to reliably gate changes for your LLM-integrated system.

Prevent Goodharting when iterating on LLM systems by applying anti-Goodhart safeguards: maintain held-out datasets, enforce regression budgets, implement calibration cycles, and use multi-metric reporting to avoid overfitting to the evaluation target.