evaluating-llms-harness

Run lm_eval with the hf model backend, specifying the pretrained model, the mmlu task, and num_fewshot 5 for the standard setting. Add batch_size auto and an output_path to save JSON results with accuracy scores and standard errors.

Use the vLLM backend instead of HuggingFace for 5-10x faster inference by passing --model vllm with tensor_parallel_size. You can also reduce num_fewshot to 0, use batch_size auto, or evaluate MMLU subsets like mmlu_stem.

Yes, it supports openai-chat-completions and anthropic-chat model types using your API keys. Chat APIs lack logprobs, so they only work with generation tasks like MMLU and GSM8K, not perplexity or loglikelihood tasks.

Mismatches usually come from wrong fewshot counts (most papers use 5-shot), incorrect task names, or a tokenizer that does not match the model. Verify num_fewshot, the exact task identifier, and that pretrained and tokenizer arguments point to the same model.

Use tensor parallelism with vLLM by setting tensor_parallel_size to split weights across GPUs, or HuggingFace's parallelize=True option. A 70B model in bfloat16 needs roughly 168GB, so 4-8 GPUs are typically required.

Use HELM for broader evaluation covering fairness, efficiency, and calibration, or MT-Bench for multi-turn conversational assessment. lm-evaluation-harness is best for standardized academic benchmarks and reproducible model comparisons.