skills.rest
MLT-OSS
MLT-OSSOFFICIAL
Explore all skills by this author →
WebsiteGitHub

evaluating-llms-harness

Evaluates LLMs across 60+ academic benchmarks using standardized prompts and metrics.

14|5|Updated Apr 9, 2026
Search Tags:#benchmarking#llm evaluation#model comparison#vllm#gsm8k#mmlu#lm-eval
One-click install
npx skills add https://github.com/MLT-OSS/hermes-agent-go --skill evaluating-llms-harness-mlt-oss
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: evaluating-llms-harness
Source: https://github.com/MLT-OSS/hermes-agent-go/tree/main/skills/mlops/evaluation/lm-evaluation-harness
Command: npx skills add https://github.com/MLT-OSS/hermes-agent-go --skill evaluating-llms-harness-mlt-oss

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires lm-eval, transformers, vllm, and includes references (resource) components.

What problem does it solve? Benchmarking LLMs consistently is hard: different prompts, few-shot settings, and metrics make results incomparable. This Skill standardizes evaluation across 60+ academic benchmarks (MMLU, GSM8K, HumanEval, TruthfulQA, HellaSwag) so you can compare models, track training progress, and report reproducible results. ## Core Features & Use Cases - Standardized Benchmark Suites: Run MMLU, GSM8K, HellaSwag, TruthfulQA, ARC, HumanEval, and 60+ tasks with one command using industry-standard prompts. - Multiple Model Backends: Evaluate HuggingFace models, vLLM-served models (5-10x faster), quantized checkpoints, and API models (OpenAI, Anthropic, local OpenAI-compatible servers). - Training Progress Tracking: Automate periodic checkpoint evaluation and plot learning curves to monitor model quality during training. - Use Case: Compare Llama-2-7B, Mistral-7B, and Phi-2 on MMLU/GSM8K/HellaSwag and generate a markdown comparison table for a model selection decision. ## Quick Start Ask the agent to evaluate a HuggingFace model like meta-llama/Llama-2-7b-hf on MMLU, GSM8K, and HellaSwag with 5-shot prompting and save the results to a JSON file.

Frequently Asked Questions about evaluating-llms-harness

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I evaluate an LLM on MMLU and GSM8K benchmarks?▼

Run lm_eval with the hf model backend, specify the HuggingFace model path, and list tasks like mmlu,gsm8k,hellaswag. Use --num_fewshot 5 for standard 5-shot evaluation and --batch_size auto to optimize throughput.

What is the difference between lm-evaluation-harness and HELM?▼

lm-evaluation-harness focuses on standardized academic benchmarks with reproducible prompts, used by EleutherAI and HuggingFace leaderboards. HELM from Stanford covers broader dimensions like fairness, efficiency, and calibration.

Can I evaluate OpenAI or Anthropic API models with lm-eval?▼

Yes, use the openai-chat-completions or anthropic-chat model types with your API key set as an environment variable. Note that chat APIs lack logprobs, so they only support generation tasks, not perplexity or loglikelihood tasks.

How do I speed up slow LLM benchmark evaluation?▼

Switch to the vLLM backend for 5-10x faster inference, reduce few-shot examples with --num_fewshot 0, or use data parallelism across multiple GPUs with accelerate launch. For MMLU, evaluate subsets like mmlu_stem instead of all 57 subjects.

Why do my benchmark results differ from published numbers?▼

Check that your few-shot count matches the paper (usually 5-shot for MMLU), verify the exact task name, and confirm the model and tokenizer match. Sampling temperature and prompt formatting differences also cause variance.

How much GPU memory do I need to evaluate a 70B model?▼

A 70B model in bfloat16 needs roughly 140GB, requiring tensor parallelism across 4-8 GPUs via vLLM tensor_parallel_size or HuggingFace parallelize=True. Quantization to 8-bit or 4-bit reduces memory requirements significantly.

Related Skills

Explore More Data & Analytics Skills→
DeepLink-orgDeepLink-org
7

nvidia-nlp-finetune-llama-7b

Benchmarks Alpaca-LoRA LLaMA-7B fine-tuning throughput on NVIDIA GPUs.

Official
Advanced
Andrew-GirgisAndrew-Girgis

huggingface-best

Recommend top HuggingFace models by benchmark scores and device constraints.

Community
Advanced
DiologIRDiologIR

bench-corpus-review

Re-run whole-corpus quality reviews of diolog-swe-bench and diff against previous reviews.

Official
Advanced
rfdiosuaorfdiosuao
30

llm-finetuning

Guide fine-tuning large language models with LoRA and QLoRA techniques.

Community
Advanced