evaluating-llms-harness

Evaluate LLMs across 60+ academic benchmarks using lm-eval harnesses.

Updated Oct 23, 2024

Search Tags:#llm #benchmarking #model evaluation #gsm8k #mmlu

Ready to run

npx skills add https://github.com/lenadlm/docker --skill evaluating-llms-harness-lenadlm

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: evaluating-llms-harness
Source: https://github.com/lenadlm/docker/tree/main/hermes-skills/skills/mlops/evaluation/lm-evaluation-harness
Command: npx skills add https://github.com/lenadlm/docker --skill evaluating-llms-harness-lenadlm

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires lm-eval, transformers, vllm, and includes scripts (resource) and references (resource) and assets (resource) components.

What problem does it solve?

This Skill provides a comprehensive framework for benchmarking and evaluating Large Language Models (LLMs) across a wide range of academic benchmarks, allowing users to compare model performance, track training progress, and report standardized metrics.

Core Features & Use Cases

Benchmarking: Evaluate LLMs across 60+ academic benchmarks including MMLU, GSM8K, HumanEval, and more.
Training Progress Tracking: Automatically evaluate checkpoints during training to track progress.
Model Comparison: Compare multiple models using a standardized benchmark suite.
vLLM Support: Use the vLLM backend for significantly faster evaluation.
Use Case: Imagine you are training a new LLM for a language understanding task. Use this Skill to benchmark your model against existing baselines and track its progress over time.

Quick Start

To evaluate the model 'gpt-4' on the MMLU benchmark, run the following command:

lm-evaluation-harness --model hf --model_args pretrained=gpt-4 --tasks mmlu

Frequently Asked Questions about evaluating-llms-harness

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I benchmark LLMs across academic benchmarks like MMLU and GSM8K?▼

You can benchmark LLMs across 60+ academic benchmarks like MMLU and GSM8K by running the lm-evaluation-harness command with HuggingFace models. Specify the model name and task arguments to evaluate performance and report standardized metrics.

Can I use vLLM to speed up LLM evaluation?▼

Yes, you can use the vLLM backend for LLM evaluation to achieve significantly faster processing speeds. The framework supports vLLM alongside HuggingFace and local models for benchmarking and training progress tracking.

What is the best way to track LLM training progress with standardized metrics?▼

Tracking LLM training progress involves automatically evaluating model checkpoints against a standardized benchmark suite. This allows you to compare performance over time and report metrics across 60+ academic benchmarks.

Do I need specific Python packages to run LLM model evaluation?▼

Yes, you need the lm-eval, transformers, and vllm Python packages installed to run LLM model evaluation. These dependencies enable benchmarking, model loading, and accelerated evaluation backends.

How do I compare multiple models using a standardized benchmark suite?▼

Comparing multiple models requires evaluating each model against the same standardized benchmark suite of 60+ academic benchmarks. This generates comparable metrics for analyzing relative performance and baselines.

Does LLM benchmarking work with local models or only HuggingFace?▼

LLM benchmarking works with both local models and HuggingFace models. The evaluation framework supports multiple backends, allowing you to evaluate local models directly or load pretrained models from HuggingFace.