evaluating-llms-harness

Run lm_eval with --model hf, --model_args pretrained=your-model, --tasks mmlu, and --num_fewshot 5 for the standard 5-shot setting. Add --batch_size auto and --output_path to save JSON results.

Use the vLLM backend with --model vllm and tensor_parallel_size set to your GPU count, which runs 5-10x faster than the HuggingFace backend. You can also reduce num_fewshot to 0 or evaluate MMLU subsets like mmlu_stem.

Yes, it supports openai-chat-completions and anthropic-chat model types using your API keys. Note that chat APIs lack logprobs, so they only work with generation tasks like MMLU and GSM8K, not perplexity tasks.

Large models exceed single-GPU VRAM; a 7B model needs about 16GB in bf16. Reduce batch size, use load_in_8bit quantization, or enable parallelize=True to shard the model across multiple GPUs.

Score differences usually come from mismatched few-shot settings (most papers use 5-shot), wrong task variants, or tokenizer mismatches. Verify num_fewshot, exact task names, and that the tokenizer matches the model.

Use HELM for broader evaluation covering fairness, efficiency, and calibration, or MT-Bench for multi-turn conversational assessment. lm-evaluation-harness is best for standardized academic benchmarks and reproducible model comparisons.