skills.rest
camoneart
camoneartCOMMUNITY
Explore all skills by this author →
GitHub

llm-evaluation

Evaluate LLM performance using automated metrics, human judgments, and LLM-as-Judge workflows.

5|Updated Aug 23, 2025
Search Tags:#human feedback#llm-as-judge#rag#ai testing#benchmarking#llm evaluation#metrics
One-click install
npx skills add https://github.com/camoneart/claude-code --skill llm-evaluation
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: llm-evaluation
Source: https://github.com/camoneart/claude-code/tree/main/skills/llm-evaluation
Command: npx skills add https://github.com/camoneart/claude-code --skill llm-evaluation

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires nltk, rouge-score, bert-score, transformers, detoxify, scipy, numpy, scikit-learn, and includes assets (resource) and references (resource) components.

What problem does it solve?

Systematically evaluating Large Language Models (LLMs) and their applications is crucial for ensuring performance, reliability, and safety, but it's a complex task. This Skill provides a comprehensive guide to automated metrics, human feedback, and LLM-as-Judge techniques.

Core Features & Use Cases

  • Automated Metrics: Covers BLEU, ROUGE, BERTScore for text generation, and MRR/NDCG for retrieval.
  • Human Evaluation: Guides on setting up annotation tasks and measuring inter-rater agreement.
  • LLM-as-Judge: Explains how to use stronger LLMs (e.g., GPT-4) to evaluate outputs from other models.
  • Use Case: When developing a new LLM-powered chatbot, this Skill helps you set up an evaluation framework to compare different prompt variations, detect performance regressions, and ensure the chatbot provides accurate and helpful responses.

Quick Start

Example: Basic LLM evaluation suite

This demonstrates defining metrics and running an evaluation on test cases.

from llm_eval import EvaluationSuite, Metric

suite = EvaluationSuite([ Metric.accuracy(), Metric.bleu(), Metric.bertscore(), Metric.custom(name="groundedness", fn=check_groundedness) ])

test_cases = [ { "input": "What is the capital of France?", "expected": "Paris", "context": "France is a country in Europe. Paris is its capital." }, # ... more test cases ]

results = suite.evaluate(model=your_model, test_cases=test_cases)

Frequently Asked Questions about llm-evaluation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I evaluate LLM performance across multiple metrics?▼

LLM evaluation combines automated metrics (BLEU, ROUGE, BERTScore), human judgment, and LLM-as-Judge workflows to quantify AI quality. This Skill provides standardized metrics, human evaluation frameworks, and code examples to measure text generation, classification, and retrieval accuracy across model comparisons and prompt variations.

What metrics should I use to benchmark text generation quality?▼

Text generation benchmarking uses BLEU, ROUGE, METEOR, and BERTScore for semantic similarity; perplexity for likelihood; and custom metrics for domain-specific criteria. This Skill covers how to apply each metric, interpret results, and combine them into reproducible evaluation pipelines for chatbots, summarization, and RAG systems.

Can I use LLMs like GPT-4 to evaluate other model outputs?▼

Yes. LLM-as-Judge uses stronger models to rate outputs on accuracy, coherence, relevance, fluency, safety, and helpfulness. This Skill explains the mechanism, provides runnable examples, and guides integration into evaluation workflows for prompt ablations, regression testing, and model selection.

How do I set up human evaluation for my LLM application?▼

Human evaluation requires defining annotation tasks, recruiting raters, and measuring inter-rater agreement to validate LLM outputs. This Skill covers annotation task design, quality control, and how to combine human judgments with automated metrics for comprehensive performance assessment.

What's the difference between automated metrics and human evaluation?▼

Automated metrics (BLEU, ROUGE, BERTScore, MRR, NDCG) scale quickly but may miss nuance; human evaluation catches semantic quality and safety concerns but is slower and costly. This Skill explains when to use each, how to layer them together, and when LLM-as-Judge bridges the gap.

How do I detect performance regressions in my LLM pipeline?▼

Regression testing compares current model outputs against baseline metrics and human judgments using standardized thresholds. This Skill provides frameworks for continuous evaluation, statistical testing, and automated alerting to catch quality degradation across prompt changes, model updates, or data shifts.

Related Skills

Explore More Software Engineering Skills→
omicverseomicverse
32

data-transform

Clean, reshape, normalize and engineer features from CSV/TSV/Parquet inputs using pandas and numpy locally.

Official
Advanced
snoodleboot-iosnoodleboot-io
2

model-monitoring

Monitor machine learning model performance and detect prediction or data drift.

Official
Intermediate
ollieb89ollieb89

llm-evaluation

Evaluate LLM outputs with automated metrics and LLM-as-Judge workflows.

Community
Advanced
sunghwanismsunghwanism

evaluating-llms-harness

Evaluate large language models on benchmarks like MMLU and GSM8K.

Community
Intermediate