evaluating-code-models

Run accelerate launch main.py with --model pointing to your HuggingFace model, --tasks humaneval, --n_samples 200, --temperature 0.8, and --allow_code_execution. Results are saved as JSON containing pass@1, pass@10, and pass@100 scores.

HumanEval+ uses the same 164 problems as HumanEval but with 80 times more test cases per problem, catching solutions that pass original tests but fail on edge cases. Execution takes longer due to the additional tests.

Use MultiPL-E tasks named multiple-{lang} for 18 languages including JavaScript, Java, C++, Go, and Rust. Generate solutions on the host with --generation_only, then evaluate inside the official MultiPL-E Docker image for safe execution.

Mismatches usually come from wrong n_samples (use 200 for accurate pass@k), incorrect temperature (0.2 for pass@1, 0.8 for higher k), inexact task names, or missing instruction tokens for chat models. Verify each parameter against the reference configuration.

Enable quantization with --load_in_8bit or --load_in_4bit, reduce --batch_size to 1, set --max_memory_per_gpu to a limit like 20GiB, or use half precision with --precision fp16. A 7B model needs about 14GB VRAM in fp16 or 6GB in 4-bit.

Use BigCode Evaluation Harness for code generation benchmarks requiring functional correctness testing with pass@k metrics. Use lm-evaluation-harness for general LLM benchmarks like MMLU, GSM8K, and HellaSwag that do not involve code execution.