evaluating-llms-harness

What problem does it solve? Evaluating and comparing language models requires standardized prompts, metrics, and reproducible setups. This Skill provides workflows for running the EleutherAI lm-evaluation-harness so you can benchmark models on MMLU, GSM8K, HumanEval, HellaSwag, TruthfulQA, and 60+ other tasks without building evaluation infrastructure yourself. ## Core Features & Use Cases - Standard Benchmark Suites: Run core reasoning, math, code, and truthfulness benchmarks with a single command against HuggingFace, vLLM, or API-based models. - Training Progress Tracking: Evaluate checkpoints periodically during training and plot learning curves from saved JSON results. - Model Comparison: Batch-evaluate multiple models and generate comparison tables of accuracy and exact-match scores. - Use Case: You are preparing a model release and need academic-standard numbers. Run the standard suite (mmlu, gsm8k, hellaswag, truthfulqa, arc_challenge) with 5-shot prompting, then compare results against published baselines like Llama 2 and Mistral. ## Quick Start Ask the AI to evaluate a HuggingFace model on MMLU, GSM8K, and HellaSwag using lm_eval with 5-shot prompting and save the results to a JSON file.