evaluating-llms-harness

Run lm_eval with the hf model backend, specifying the pretrained model, the mmlu task, and num_fewshot 5 for the standard setting. Add batch_size auto and an output_path to save results as JSON with accuracy and standard error.

lm-evaluation-harness focuses on standardized academic benchmarks like MMLU, GSM8K, and HumanEval with reproducible prompts. HELM from Stanford covers broader dimensions including fairness, efficiency, and calibration, while AlpacaEval and MT-Bench target instruction-following and conversation.

Yes, use the openai-chat-completions or anthropic-chat model types with your API key exported. Chat APIs lack logprobs, so they only support generation tasks like MMLU and GSM8K, not perplexity or loglikelihood evaluations.

Full MMLU takes about 2 hours on a 7B model with the HuggingFace backend. Switch to the vLLM backend for 5-10x faster inference, reduce num_fewshot to 0, or evaluate a subset like mmlu_stem instead of all 57 subjects.

Code-executing tasks like HumanEval and MBPP are gated behind an explicit safety flag. You must pass --confirm_run_unsafe_code, otherwise lm-eval refuses to run the task rather than silently skipping code execution.

Use tensor parallelism to shard weights across GPUs: pass parallelize=True with the hf backend or tensor_parallel_size=4 with vLLM. A 70B model in bfloat16 needs about 168GB total, so 4-8 GPUs are typically required.