evaluating-llms-harness

Run lm_eval with the hf model backend, passing the HuggingFace model path and a comma-separated task list such as mmlu,gsm8k,hellaswag. Use --num_fewshot 5 for the standard setting and --output_path to save JSON results with accuracy and standard error.

lm-evaluation-harness focuses on standardized academic benchmarks like MMLU and GSM8K with reproducible prompts, while HELM from Stanford covers broader dimensions such as fairness, efficiency, and calibration. Use lm-eval for comparable benchmark scores and HELM for holistic model assessment.

Yes, use the openai-chat-completions or anthropic-chat model types with your API key exported as an environment variable. Chat APIs lack logprobs, so they only support generation tasks like MMLU and GSM8K, not perplexity or loglikelihood tasks.

Full MMLU covers 57 subjects and takes about 2 hours for a 7B model on one GPU. Switch to the vLLM backend for 5-10x speedup, reduce few-shot examples to zero, or evaluate only a subset like mmlu_stem.

Reduce batch size to 1 or use --batch_size auto, enable 8-bit quantization with load_in_8bit=True, or shard the model across GPUs with parallelize=True. A 7B model needs roughly 16GB VRAM in bfloat16 or 8GB in 8-bit.

Avoid it for conversational multi-turn evaluation, where MT-Bench fits better, and for instruction-following judged by LLMs, where AlpacaEval is standard. Domain-specific needs are better served by custom tasks or bespoke evaluation scripts.