evaluating-llms-harness

Install lm-eval via pip, then run lm_eval with --model hf, --model_args pretrained=your-model, --tasks mmlu, and --num_fewshot 5 for the standard 5-shot setting. Results are saved as JSON with accuracy scores and standard errors.

lm-evaluation-harness focuses on standardized academic benchmarks like MMLU and GSM8K with reproducible prompts, making it the industry standard used by EleutherAI and HuggingFace. HELM from Stanford offers broader evaluation covering fairness, efficiency, and calibration.

Yes, use --model openai-chat-completions or --model anthropic-chat with your API key exported as an environment variable. Note that chat models lack logprobs, so they only support generation tasks, not perplexity or loglikelihood tasks.

Switch to the vLLM backend with --model vllm for 5-10x faster inference, reduce few-shot examples with --num_fewshot 0, or evaluate a subset like mmlu_stem instead of the full 57-subject MMLU. Full MMLU on a 7B model takes about 2 hours on one A100.

Code-executing tasks like HumanEval and MBPP are gated behind an explicit safety flag. You must pass --confirm_run_unsafe_code on the command line, otherwise lm-eval refuses to run the task rather than silently skipping code execution.

A 70B model in bfloat16 requires roughly 168GB of VRAM, so it will not fit on a single GPU. Use tensor parallelism with vLLM (tensor_parallel_size=4 or 8) or HuggingFace parallelize=True to shard the model across multiple GPUs.