skills.rest
Chensihakniroth
ChensihaknirothCOMMUNITY
Explore all skills by this author →
GitHub

evaluating-llms-harness

Benchmark LLMs across 60+ academic tasks using the lm-evaluation-harness framework.

Updated Jun 7, 2026
Search Tags:#benchmarking#llm evaluation#model comparison#vLLM#GSM8K#MMLU#lm-eval
One-click install
npx skills add https://github.com/Chensihakniroth/ANAKOT-AGENT --skill evaluating-llms-harness-chensihakniroth
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: evaluating-llms-harness
Source: https://github.com/Chensihakniroth/ANAKOT-AGENT/tree/main/skills/mlops/evaluation/lm-evaluation-harness
Command: npx skills add https://github.com/Chensihakniroth/ANAKOT-AGENT --skill evaluating-llms-harness-chensihakniroth

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires lm-eval, transformers, vllm, and includes references (resource) components.

What problem does it solve? Evaluating and comparing large language models requires standardized prompts, metrics, and benchmarks; running ad-hoc tests produces inconsistent, non-reproducible results that cannot be compared across models or papers. ## Core Features & Use Cases - Standardized Benchmarking: Evaluate models on 60+ academic benchmarks including MMLU, GSM8K, HumanEval, TruthfulQA, HellaSwag, and ARC with consistent prompts and metrics. - Multiple Backends: Run evaluations against HuggingFace models, vLLM for 5-10x faster inference, or API-based models like OpenAI GPT-4 and Anthropic Claude. - Distributed Evaluation: Scale across multiple GPUs using data parallelism, tensor parallelism, or pipeline parallelism for large models. - Use Case: Compare Llama-2-7B, Mistral-7B, and Phi-2 on MMLU, GSM8K, and HellaSwag, then generate a markdown comparison table to decide which base model to fine-tune for your project. ## Quick Start Ask the agent to evaluate a HuggingFace model on MMLU and GSM8K with 5-shot prompting and save the results to a JSON file.

Frequently Asked Questions about evaluating-llms-harness

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I benchmark an LLM on MMLU with lm-evaluation-harness?▼

Install lm-eval via pip, then run lm_eval with --model hf, --model_args pretrained=your-model, --tasks mmlu, and --num_fewshot 5. Results with accuracy and standard error are saved to a JSON output file.

What is the difference between lm-evaluation-harness and HELM?▼

lm-evaluation-harness focuses on standardized academic benchmarks with reproducible prompts, while HELM from Stanford covers broader dimensions like fairness, efficiency, and calibration. Use lm-eval for standard model comparisons and papers.

Can lm-eval evaluate OpenAI or Anthropic API models?▼

Yes, it supports openai-chat-completions and anthropic-chat model types via API keys. Note that chat APIs without logprobs only work on generation tasks like MMLU and GSM8K, not perplexity tasks.

How do I speed up slow LLM evaluation?▼

Use the vLLM backend with --model vllm for 5-10x faster inference, reduce few-shot examples with --num_fewshot 0, or run data-parallel evaluation across multiple GPUs with accelerate launch.

Why does evaluation run out of GPU memory?▼

Large models exceed VRAM at full precision. Reduce batch size, use 8-bit or 4-bit quantization via load_in_8bit, or shard the model across GPUs with parallelize=True or vLLM tensor_parallel_size.

How do I create a custom evaluation task?▼

Define a YAML file with dataset_path, doc_to_text, doc_to_target, and metric_list, then run lm_eval with --include_path pointing to your task directory. Python functions in utils.py handle complex preprocessing or custom metrics.

Related Skills

Explore More Data & Analytics Skills→
MLT-OSSMLT-OSS
14

fine-tuning-with-trl

Fine-tune language models with TRL using SFT, DPO, PPO, and GRPO training workflows.

Official
Advanced
AlexiosBluffMaraAlexiosBluffMara
2

evaluating-llms-harness

Run language model evaluations across 60+ benchmarks with HuggingFace, vLLM, and API backends.

Community
Advanced
agentscope-aiagentscope-ai
809

metric-design

Select, configure, and combine evaluation graders into executable OpenJudge pipeline code.

Official
Advanced
wuyoscarwuyoscar
934

aiml-sentiment-calibration

Evaluate multilingual sentiment classifiers against target star ratings using Python scripts.

Community
Intermediate