evaluating-llms-harness

Install lm-eval and run lm_eval with --model hf, --model_args pretrained=your-model, --tasks mmlu,gsm8k, and --num_fewshot 5. Results with accuracy metrics and standard errors are saved to a JSON output file.

lm-evaluation-harness focuses on standardized academic benchmarks like MMLU and GSM8K with reproducible prompts, while HELM from Stanford covers broader dimensions including fairness, efficiency, and calibration. Use lm-eval for standard model comparisons.

Yes, use --model openai-chat-completions or --model anthropic-chat with your API key exported. Note that chat models lack logprobs, so they only support generation tasks like MMLU and GSM8K, not perplexity tasks.

Switch to the vLLM backend with --model vllm for 5-10x faster inference, reduce few-shot examples with --num_fewshot 0, or evaluate a subset like mmlu_stem instead of the full 57-subject MMLU benchmark.

Code-executing tasks like HumanEval and MBPP require the explicit --confirm_run_unsafe_code flag because they execute generated code. Without this flag, lm-eval refuses to run the task rather than silently skipping execution.

A 70B model in bfloat16 needs roughly 168GB, so it requires tensor parallelism across 4-8 GPUs using vLLM's tensor_parallel_size or HuggingFace's parallelize=True option. Quantization to 8-bit reduces requirements significantly.