skills.rest
loteiron
loteironCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

evaluating-llms-harness

Benchmark LLMs across 60+ academic tasks using lm-evaluation-harness.

Updated Sep 10, 2026
Search Tags:#benchmarking#llm evaluation#model comparison#vLLM#GSM8K#MMLU#lm-eval
One-click install
npx skills add https://github.com/loteiron/ZeusAgent --skill evaluating-llms-harness-loteiron
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: evaluating-llms-harness
Source: https://github.com/loteiron/ZeusAgent/tree/main/optional-skills/mlops/evaluation/evaluating-llms-harness
Command: npx skills add https://github.com/loteiron/ZeusAgent --skill evaluating-llms-harness-loteiron

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires lm-eval, transformers, vllm, and includes references (resource) components.

What problem does it solve? Evaluating and comparing language models requires standardized prompts, metrics, and benchmarks; ad-hoc testing produces inconsistent, non-reproducible results that cannot be compared across models or papers. ## Core Features & Use Cases - Standardized Benchmarking: Run 60+ academic benchmarks (MMLU, GSM8K, HumanEval, TruthfulQA, HellaSwag, ARC) with consistent prompts and metrics. - Multiple Backends: Evaluate HuggingFace models, vLLM-served models, and API models (OpenAI, Anthropic, local OpenAI-compatible servers). - Training Progress Tracking: Automate periodic checkpoint evaluation and plot learning curves during training runs. - Use Case: Compare Llama-2-7B, Mistral-7B, and Phi-2 on MMLU, GSM8K, and HellaSwag, then generate a markdown comparison table of scores for a model selection report. ## Quick Start Ask the agent to evaluate a HuggingFace model on MMLU and GSM8K with 5-shot prompting and save the results to a JSON file.

Frequently Asked Questions about evaluating-llms-harness

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I benchmark an LLM on MMLU with lm-evaluation-harness?▼

Run lm_eval with --model hf, --model_args pretrained=your-model, --tasks mmlu, and --num_fewshot 5 for the standard 5-shot setting. Results with accuracy and standard error are saved to a JSON output path.

What is the difference between lm-evaluation-harness and HELM or AlpacaEval?▼

lm-evaluation-harness focuses on standardized academic benchmarks with reproducible prompts. HELM covers broader dimensions like fairness and efficiency, while AlpacaEval uses LLM judges for instruction-following evaluation.

Can lm-eval evaluate OpenAI or Anthropic API models?▼

Yes, use openai-chat-completions or anthropic-chat model types with the appropriate API key exported. Chat APIs lack logprobs, so they only support generation tasks like MMLU and GSM8K, not perplexity tasks.

Why is my MMLU evaluation so slow and how do I speed it up?▼

Full MMLU takes about 2 hours on a 7B model with the HuggingFace backend. Switch to the vLLM backend for 5-10x faster inference, reduce fewshot examples, or evaluate a subset like mmlu_stem.

Why does HumanEval refuse to run in lm-eval?▼

Code-executing tasks like HumanEval and MBPP are gated behind an explicit safety flag. Pass --confirm_run_unsafe_code to run tasks that execute generated code, otherwise lm-eval refuses rather than silently skipping.

How much GPU memory do I need to evaluate a 70B model?▼

A 70B model in bf16 needs roughly 168GB, so it will not fit on a single GPU. Use vLLM tensor_parallel_size=4 or HuggingFace parallelize=True to shard weights across multiple GPUs.

Related Skills

Explore More Data & Analytics Skills→
rishikanthcrishikanthc

ml-huggingface-models

Verify, download, fine-tune, and integrate HuggingFace pre-trained models in PyTorch projects.

Community
Advanced
domattiolidomattioli

huggingface-best

Query HuggingFace benchmark leaderboards and aggregate model performance data into a device-aware comparison.

Community
Advanced
duanbiao2000duanbiao2000
3

llm-evaluation

Evaluate LLM applications with automated metrics, human judgments, and benchmarking.

Community
Advanced
HermeticOrmusHermeticOrmus

Model Evaluation Patterns

Generate classification metrics, calibration analysis, and fairness evaluations for machine learning models.

Community
Advanced