skills.rest
Junkfooooood
JunkfoooooodCOMMUNITY
Explore all skills by this author →
GitHub

evaluating-llms-harness

Benchmark LLMs across 60+ academic tasks using the lm-evaluation-harness framework.

Updated May 4, 2026
Search Tags:#benchmarking#llm evaluation#model comparison#vllm#gsm8k#mmlu#lm-eval
One-click install
npx skills add https://github.com/Junkfooooood/hermes-governance --skill evaluating-llms-harness-junkfooooood
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: evaluating-llms-harness
Source: https://github.com/Junkfooooood/hermes-governance/tree/main/skills/mlops/evaluation/lm-evaluation-harness
Command: npx skills add https://github.com/Junkfooooood/hermes-governance --skill evaluating-llms-harness-junkfooooood

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires lm-eval, transformers, vllm, and includes references (resource) components.

What problem does it solve? Evaluating and comparing large language models requires standardized prompts, metrics, and benchmarks; running ad-hoc tests produces inconsistent, non-reproducible results that cannot be compared across models or papers. ## Core Features & Use Cases - Standardized Benchmarking: Evaluate models on 60+ academic benchmarks including MMLU, GSM8K, HumanEval, TruthfulQA, HellaSwag, and ARC with consistent prompts and metrics. - Multiple Backends: Run evaluations via HuggingFace transformers, vLLM for 5-10x faster inference, or API-based models like OpenAI GPT-4 and Anthropic Claude. - Training Progress Tracking: Automate periodic checkpoint evaluation during training and plot learning curves over time. - Use Case: Compare Llama-2-7B, Mistral-7B, and Phi-2 on MMLU, GSM8K, and HellaSwag, then generate a markdown comparison table of accuracy scores for a model selection report. ## Quick Start Ask the AI to evaluate a HuggingFace model like meta-llama/Llama-2-7b-hf on the MMLU and GSM8K benchmarks using lm_eval with 5-shot prompting.

Frequently Asked Questions about evaluating-llms-harness

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I evaluate an LLM on MMLU with lm-evaluation-harness?▼

Run lm_eval with --model hf, --model_args pretrained=your-model, --tasks mmlu, and --num_fewshot 5 for the standard 5-shot setting. Results are saved as JSON with accuracy scores and standard errors per subject.

How can I speed up LLM benchmark evaluation?▼

Use the vLLM backend with --model vllm and tensor_parallel_size set to your GPU count, which runs 5-10x faster than standard HuggingFace inference. You can also reduce num_fewshot to 0 or evaluate MMLU subsets like mmlu_stem.

Can lm-evaluation-harness evaluate OpenAI or Anthropic API models?▼

Yes, it supports openai-chat-completions and anthropic-chat model types using your API keys. Note that chat APIs lack logprobs, so they only work with generation tasks like MMLU and GSM8K, not perplexity tasks.

Why does my evaluation run out of GPU memory?▼

Large models exceed single-GPU VRAM; a 7B model needs about 16GB in bfloat16. Reduce batch size, use load_in_8bit=True quantization, or enable tensor parallelism with parallelize=True or vLLM tensor_parallel_size.

Why do my benchmark results differ from published numbers?▼

Mismatches usually come from wrong few-shot counts (most papers use 5-shot), incorrect task names, or tokenizer mismatches. Verify --num_fewshot, use exact task names like mmlu, and confirm the tokenizer matches the model.

When should I use HELM or MT-Bench instead of lm-evaluation-harness?▼

Use HELM for broader evaluation covering fairness, efficiency, and calibration, or MT-Bench for multi-turn conversational assessment. lm-evaluation-harness is best for standardized academic benchmarks and reproducible model comparisons.

Related Skills

Explore More Data & Analytics Skills→
azaanalirazaazaanaliraza

fine-tuning-with-trl

Fine-tune language models with TRL using SFT, DPO, PPO, and GRPO training methods.

Community
Advanced
AlexiosBluffMaraAlexiosBluffMara
2

evaluating-llms-harness

Run language model evaluations across 60+ benchmarks with HuggingFace, vLLM, and API backends.

Community
Advanced
robertbr123robertbr123

evaluating-llms-harness

Evaluate LLMs against 60+ academic benchmarks using lm-eval.

Community
Advanced
HermeticOrmusHermeticOrmus

Model Evaluation Patterns

Generate classification metrics, calibration analysis, and fairness evaluations for machine learning models.

Community
Advanced