skills.rest
perasyudha
perasyudhaCOMMUNITY
Explore all skills by this author →
Website𝕏@Nyxora_AIGitHub

evaluating-llms-harness

Benchmark LLMs across 60+ academic tasks using the lm-evaluation-harness framework.

5|2|Updated May 26, 2026
Search Tags:#benchmarking#llm evaluation#model comparison#vllm#gsm8k#mmlu#lm-eval
One-click install
npx skills add https://github.com/perasyudha/Nyxora --skill evaluating-llms-harness-perasyudha
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: evaluating-llms-harness
Source: https://github.com/perasyudha/Nyxora/tree/main/packages/core/playbooks/mlops/evaluation/lm-evaluation-harness
Command: npx skills add https://github.com/perasyudha/Nyxora --skill evaluating-llms-harness-perasyudha

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires lm-eval, transformers, vllm, and includes references (resource) components.

What problem does it solve? Evaluating and comparing large language models requires standardized prompts, metrics, and benchmarks; running ad-hoc tests produces inconsistent, non-reproducible results that cannot be compared across models or papers. ## Core Features & Use Cases - Standardized Benchmarking: Evaluate models on 60+ academic benchmarks including MMLU, GSM8K, HumanEval, TruthfulQA, HellaSwag, and ARC with consistent prompts and metrics. - Multiple Backends: Run evaluations via HuggingFace transformers, vLLM for 5-10x faster inference, or API-based models like OpenAI GPT-4 and Anthropic Claude. - Training Progress Tracking: Automate periodic checkpoint evaluation during training and plot learning curves over time. - Use Case: Compare Llama-2-7B, Mistral-7B, and Phi-2 on MMLU, GSM8K, and HellaSwag, then generate a markdown comparison table of accuracy scores for a model selection report. ## Quick Start Ask the agent to evaluate a HuggingFace model on MMLU and GSM8K with 5-shot prompting and save the results to a JSON file.

Frequently Asked Questions about evaluating-llms-harness

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I benchmark an LLM on MMLU with lm-evaluation-harness?▼

Install lm-eval via pip, then run lm_eval with --model hf, --model_args pretrained=your-model, --tasks mmlu, and --num_fewshot 5. Results with accuracy and standard error are saved to a JSON output file.

What is the difference between lm-evaluation-harness and HELM or MT-Bench?▼

lm-evaluation-harness focuses on standardized academic benchmarks with reproducible prompts. HELM covers broader dimensions like fairness and efficiency, while MT-Bench evaluates multi-turn conversation using LLM judges.

Can I evaluate OpenAI or Anthropic API models with lm-eval?▼

Yes, set OPENAI_API_KEY or ANTHROPIC_API_KEY and use the openai-chat-completions or anthropic-chat model types. Chat APIs lack logprobs, so they only support generation tasks like MMLU and GSM8K, not perplexity tasks.

Why is my MMLU evaluation so slow and how do I speed it up?▼

Full MMLU takes about 2 hours on a 7B model with the HuggingFace backend. Switch to the vLLM backend for 5-10x speedup, reduce few-shot examples, or evaluate only a subset like mmlu_stem.

How do I fix out-of-memory errors during evaluation?▼

Reduce batch size to 1 or use --batch_size auto, load the model in 8-bit or 4-bit quantization, or enable CPU offloading with device_map=auto. For large models, use tensor parallelism across multiple GPUs.

Why do my benchmark results differ from published numbers?▼

Check that your few-shot count matches the paper (usually 5-shot), verify the exact task name, and confirm the model and tokenizer match. Sampling temperature and prompt formatting also affect scores.

Related Skills

Explore More Data & Analytics Skills→
MLT-OSSMLT-OSS
14

fine-tuning-with-trl

Fine-tune language models with TRL using SFT, DPO, PPO, and GRPO training workflows.

Official
Advanced
AlexiosBluffMaraAlexiosBluffMara
2

evaluating-llms-harness

Run language model evaluations across 60+ benchmarks with HuggingFace, vLLM, and API backends.

Community
Advanced
NOMARJNOMARJ

llm-evaluation

Evaluate LLM applications with automated metrics, human feedback, and benchmarking.

Official
Advanced
HermeticOrmusHermeticOrmus

Model Evaluation Patterns

Generate classification metrics, calibration analysis, and fairness evaluations for machine learning models.

Community
Advanced