skills.rest
AvaTar-ArTs
AvaTar-ArTsCOMMUNITY
Explore all skills by this author →
GitHub

evaluating-llms-harness

Benchmarks LLMs across 60+ academic tasks using the lm-evaluation-harness framework.

Updated Jul 10, 2026
Search Tags:#benchmarking#llm evaluation#model comparison#vllm#gsm8k#mmlu#lm-eval
One-click install
npx skills add https://github.com/AvaTar-ArTs/.Agent-skills --skill evaluating-llms-harness-avatar-arts
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: evaluating-llms-harness
Source: https://github.com/AvaTar-ArTs/.Agent-skills/tree/main/skills/mlops/evaluation/lm-evaluation-harness
Command: npx skills add https://github.com/AvaTar-ArTs/.Agent-skills --skill evaluating-llms-harness-avatar-arts

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires lm-eval, transformers, vllm, and includes references (resource) components.

What problem does it solve? Evaluating and comparing language models requires standardized prompts, metrics, and reproducible setups. This Skill provides structured workflows for benchmarking LLMs with the EleutherAI lm-evaluation-harness, covering model comparison, training progress tracking, and result analysis. ## Core Features & Use Cases - Standard Benchmark Evaluation: Run MMLU, GSM8K, HellaSwag, TruthfulQA, ARC, HumanEval, and 60+ other tasks against HuggingFace, vLLM, or API-based models. - Training Progress Tracking: Evaluate checkpoints periodically during training and plot learning curves from saved JSON results. - Multi-Model Comparison: Batch-evaluate a list of models and generate comparison tables from the output files. - Use Case: A researcher fine-tunes a 7B model and needs to report MMLU and GSM8K scores comparable to published baselines; this Skill provides the exact commands, few-shot settings, and result parsing steps. ## Quick Start Ask the AI to evaluate a HuggingFace model on MMLU and GSM8K with 5-shot prompting using lm_eval and save the results to a JSON file.

Frequently Asked Questions about evaluating-llms-harness

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I benchmark an LLM on MMLU with lm-evaluation-harness?▼

Install lm-eval and run lm_eval with --model hf, --model_args pretrained=your-model, --tasks mmlu, and --num_fewshot 5. Results are saved as JSON with accuracy scores and standard errors per task.

How can I speed up LLM evaluation with vLLM?▼

Use --model vllm instead of hf and pass tensor_parallel_size in model_args. vLLM runs evaluation 5-10x faster than standard HuggingFace inference, reducing full MMLU on a 7B model from about 2 hours to 15-20 minutes.

Can lm-evaluation-harness evaluate OpenAI or Anthropic API models?▼

Yes, it supports openai-chat-completions and anthropic-chat model types via API keys. Chat models lack logprobs, so they only work with generation tasks like MMLU and GSM8K, not perplexity-based tasks.

Why is my evaluation running out of GPU memory?▼

Reduce batch size to 1 or auto, enable 8-bit quantization with load_in_8bit=True, or use tensor parallelism across multiple GPUs. A 7B model needs about 16GB VRAM in bfloat16 or 8GB in 8-bit.

When should I use HELM or MT-Bench instead of lm-evaluation-harness?▼

Use HELM for broader evaluation covering fairness, efficiency, and calibration, or MT-Bench for multi-turn conversational assessment. lm-evaluation-harness is best for standardized academic benchmarks and reproducible model comparisons.

Related Skills

Explore More Data & Analytics Skills→
MLT-OSSMLT-OSS
14

fine-tuning-with-trl

Fine-tune language models with TRL using SFT, DPO, PPO, and GRPO training workflows.

Official
Advanced
akiyoshiakiyoshi

benchmark-models

Compare the same prompt across multiple AI models for latency, cost, and quality.

Community
Advanced
duanbiao2000duanbiao2000
3

llm-evaluation

Evaluate LLM applications with automated metrics, human judgments, and benchmarking.

Community
Advanced
amanhsnamanhsn

interview-system-designer

Generate role-specific interview loops, question banks, and scoring rubrics.

Community
Advanced