skills.rest
patronus-ai
patronus-aiOFFICIAL
Explore all skills by this author →
GitHub

llm-evaluation

Compute automated metrics and aggregate scores for LLM evaluation suites.

2|2|Updated Mar 26, 2026
Search Tags:#llm-as-judge#benchmarking#llm evaluation#regression testing#ab testing#automated metrics#human annotation
One-click install
npx skills add https://github.com/patronus-ai/skill-inject --skill llm-evaluation-patronus-ai
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: llm-evaluation
Source: https://github.com/patronus-ai/skill-inject/tree/main/data/skills/llm-evaluation
Command: npx skills add https://github.com/patronus-ai/skill-inject --skill llm-evaluation-patronus-ai

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

It solves the challenge of assessing whether LLM outputs are accurate, reliable, and safe using repeatable methods instead of ad-hoc inspection.

Core Features & Use Cases

  • Automated metrics: Computes scores for text generation quality (BLEU, ROUGE, METEOR, BERTScore, perplexity), classification performance (accuracy, precision/recall/F1, confusion matrix, AUC-ROC), and retrieval performance (MRR, NDCG, precision@K, recall@K).
  • Human evaluation: Provides a practical framework for rating answers on accuracy, coherence, relevance, fluency, safety, and helpfulness when automation is insufficient.
  • LLM-as-judge: Supports pointwise, pairwise, reference-based, and reference-free evaluation patterns to compare outputs when gold labels are limited or absent.
  • Experimental rigor: Includes patterns for A/B testing and regression detection to track improvements over time and catch performance drops.
  • Benchmarking workflows: Defines how to run models over datasets and aggregate results with summary statistics for each metric.

Quick Start

Use llm-evaluation to set up an evaluation suite for your model and compute automated metric scores for a curated set of test cases.

Frequently Asked Questions about llm-evaluation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I evaluate LLM outputs using automated metrics like BLEU and ROUGE?▼

To evaluate LLM outputs using automated metrics, compute text generation scores like BLEU, ROUGE, METEOR, BERTScore, and perplexity by defining an evaluation suite, preparing structured test cases, and aggregating per-metric scores for decision-making.

What is LLM-as-judge evaluation and when should I use it?▼

LLM-as-judge evaluation uses an LLM to compare outputs when gold labels are limited or absent. Use pointwise, pairwise, reference-based, or reference-free evaluation patterns to assess accuracy, coherence, relevance, fluency, safety, and helpfulness.

Can I run A/B testing and regression detection for LLM applications?▼

Yes, you can run A/B testing and regression detection for LLM applications by applying experimental rigor patterns. Track performance improvements over time and catch drops by running models over datasets and aggregating summary statistics.

How do I measure retrieval performance for LLM applications?▼

Measure retrieval performance for LLM applications by computing scores using metrics like MRR, NDCG, precision@K, and recall@K. Define evaluation suites and prepare structured test cases to compute per-metric scores with aggregation.

What's the best way to evaluate LLM classification performance?▼

The best way to evaluate LLM classification performance is computing accuracy, precision/recall/F1, confusion matrix, and AUC-ROC scores. Run models over datasets and aggregate results with summary statistics for each metric.

When should I use human evaluation instead of automated metrics?▼

Use human evaluation instead of automated metrics when automation is insufficient. Apply a practical framework to rate answers on accuracy, coherence, relevance, fluency, safety, and helpfulness to assess whether outputs are accurate, reliable, and safe.

Related Skills

Explore More Data & Analytics Skills→
dieu-donneedieu-donnee

prompt-engineer

Design prompts, evaluation rubrics, and test suites for LLM applications.

Community
Advanced
aisa-groupaisa-group
90

llm-evaluation

Compute automated metrics and statistical tests for LLM evaluation.

Official
Advanced
ccfccf

llm-evaluation

Automate LLM evaluation with automated metrics, human feedback, and benchmarking.

Community
Advanced
simplysmartaisimplysmartai

llm-evaluation

Evaluate LLM applications with automated metrics, human feedback, and LLM-as-judge patterns.

Community
Advanced