evaluating-llms-harness

Run lm_eval with the hf model backend, passing pretrained model args, the mmlu task, and a few-shot count. For example: lm_eval --model hf --model_args pretrained=meta-llama/Llama-2-7b-hf --tasks mmlu --num_fewshot 5 --batch_size 8.

lm-evaluation-harness focuses on standardized academic benchmarks with reproducible prompts, making it the industry standard for reporting results. HELM from Stanford covers broader dimensions like fairness, efficiency, and calibration.

Yes, use the openai-chat-completions or anthropic-chat model types with your API key exported. Note that chat APIs without logprobs only support generation tasks like MMLU and GSM8K, not perplexity or loglikelihood tasks.

Switch to the vLLM backend for 5-10x faster inference, reduce few-shot examples with --num_fewshot 0, or evaluate a subset like mmlu_stem instead of full MMLU. Multi-GPU data parallelism with accelerate also provides near-linear speedup.

Check that your few-shot count matches the paper (usually 5-shot), verify the exact task name, and confirm the model and tokenizer match. Sampling temperature and seed settings also affect generation-based task scores.

A 70B model in bfloat16 needs roughly 168GB, so it requires tensor parallelism across 4-8 GPUs or quantization. With vLLM tensor_parallel_size=4, each GPU needs about 35GB.