lm-evaluation-harness

Benchmarking language models evaluates their performance on standardized tasks like MMLU, GSM8K, and HumanEval. This Skill runs your model against 60+ academic benchmarks to measure reasoning, code, and knowledge capabilities, producing comparable results across different model architectures and sizes.

Yes. This Skill supports diverse model backends including HuggingFace, vLLM, and API-based models from OpenAI and Anthropic, plus custom endpoints. You configure your model source once and run the same benchmark suite across all of them.

Training-progress workflows let you evaluate checkpoints across multiple benchmarks without full re-evaluation each time. Configure batch processing and few-shot settings to run fast-pass evaluations, generating serialized results that track quality gains incrementally.

Yes. The 60+ benchmark suite includes specialized tasks: HumanEval for code generation, GSM8K for mathematical reasoning, and MMLU for broad knowledge. Each benchmark targets different model capabilities so you get a full quality picture.

Yes. Beyond the 60+ built-in tasks, you can define custom tasks with your own datasets and evaluation metrics, then run them alongside standard benchmarks. Results serialize in the same format for unified analysis.

Benchmark evaluation scales to both GPU and CPU environments. GPU acceleration speeds large-scale runs, but CPU evaluation is supported. Parallel and distributed execution options let you tune resource use for your infrastructure.