evaluating-code-models

To evaluate code generation models on HumanEval and MBPP, use an execution-based harness to run functional correctness tasks and report pass@k scores. This provides consistent, reproducible metrics for comparing model quality.

The pass@k metric in code model benchmarking measures functional correctness by executing generated code against test cases. It reports the probability that at least one of k generated samples passes, enabling objective model comparison.

Yes, you can benchmark multi-language code synthesis models using the MultiPL-E dataset. The evaluation harness supports testing instruction-tuned models and fill-in-the-middle workflows across multiple programming languages.

You can create custom benchmark definitions for code evaluation by adapting the harness to new datasets and task definitions. Reference guides assist with task creation and troubleshooting to ensure accurate execution-based scoring.

Supported datasets for execution-based code evaluation include HumanEval, MBPP, MultiPL-E, APPS, and DS-1000. These standard benchmarks measure functional correctness and generate pass@k results.

Execution-based evaluation is preferred over ad hoc prompting because it provides consistent, reproducible measurements of functional correctness. It prevents biased assessments by running generated code against standardized test suites.