skills.rest
Dianshu-Liao
Dianshu-LiaoCOMMUNITY
Explore all skills by this author →
GitHub

llm-evaluation

Implement LLM evaluation with automated metrics and LLM-as-judge patterns.

2|Updated May 20, 2026
Search Tags:#llm-as-judge#llm#benchmarking#metrics#evaluation#ab testing
One-click install
npx skills add https://github.com/Dianshu-Liao/SkilLGuard --skill llm-evaluation-dianshu-liao
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: llm-evaluation
Source: https://github.com/Dianshu-Liao/SkilLGuard/tree/main/data/skills/llm-evaluation
Command: npx skills add https://github.com/Dianshu-Liao/SkilLGuard --skill llm-evaluation-dianshu-liao

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

LLMs often change behavior with new prompts, models, or pipelines, and teams need a repeatable way to measure quality, compare variants, and detect regressions before shipping.

Core Features & Use Cases

  • Automated Metrics: Compute quantitative scores for text generation, classification, and retrieval (e.g., BLEU/ROUGE/METEOR/BERTScore, accuracy/F1/AUC, MRR/NDCG/Precision@K).
  • Human Evaluation: Define consistent rating dimensions such as accuracy, coherence, relevance, fluency, safety, and helpfulness, plus annotation structures.
  • LLM-as-Judge: Apply pointwise or pairwise judging patterns using stronger models to score or compare outputs when ground truth is limited.
  • A/B Testing and Regression Detection: Run statistical tests and track baseline vs. current metric shifts to flag meaningful drops.

Quick Start

Ask the skill to help you set up an evaluation suite with BLEU plus a custom groundedness metric, then run it against your test cases for a given model.

Frequently Asked Questions about llm-evaluation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I set up an LLM evaluation suite with automated metrics and judge patterns?▼

To set up an LLM evaluation suite, define metric functions like BLEU or BERTScore, prepare test cases, and optionally call external LLM judge APIs to compute scores and record results for model comparison.

What is the LLM-as-judge pattern and when should I use it for evaluation?▼

The LLM-as-judge pattern uses stronger models to score or compare outputs pointwise or pairwise when ground truth is limited, providing automated quality measurement for benchmarking and A/B testing.

How do I detect prompt regressions when updating models or pipelines?▼

Detect prompt regressions by running A/B testing with statistical tests to track baseline versus current metric shifts, flagging meaningful drops in accuracy, coherence, or other evaluation dimensions before shipping.

Can I evaluate retrieval tasks using automated metrics?▼

Yes, you can evaluate retrieval tasks using automated quantitative metrics such as MRR, NDCG, and Precision@K to measure ranking quality alongside classification metrics like accuracy and F1.

What rating dimensions should I use for human evaluation of LLM outputs?▼

For human evaluation of LLM outputs, define consistent rating dimensions such as accuracy, coherence, relevance, fluency, safety, and helpfulness, structured within clear annotation frameworks.

Does this evaluation approach work without ground truth data?▼

Yes, when ground truth is limited, you can apply LLM-as-judge patterns using pointwise or pairwise judging to score outputs, allowing quality measurement and variant comparison without labeled datasets.

Related Skills

Explore More Education & Research Skills→
patronus-aipatronus-ai
2

llm-evaluation

Compute automated metrics and aggregate scores for LLM evaluation suites.

Official
Advanced
AdeitasunaAdeitasuna

eval-running

Run Loa evaluation suites to detect regressions and benchmark skill quality.

Community
Intermediate
ollieb89ollieb89

llm-evaluation

Evaluate LLM outputs with automated metrics and LLM-as-Judge workflows.

Community
Advanced
simplysmartaisimplysmartai

llm-evaluation

Evaluate LLM applications with automated metrics, human feedback, and LLM-as-judge patterns.

Community
Advanced