skills.rest
escotilha
escotilhaCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

llm-eval

Build an end-to-end LLM evaluation pipeline with synthetic data and CI regression checks.

Updated Aug 27, 2026
Search Tags:#llm-as-judge#llm#evaluation#synthetic data#rag eval#ci regression testing
One-click install
npx skills add https://github.com/escotilha/claude-public --skill llm-eval-escotilha
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: llm-eval
Source: https://github.com/escotilha/claude-public/tree/main/skills/llm-eval
Command: npx skills add https://github.com/escotilha/claude-public --skill llm-eval-escotilha

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill prevents unreliable or unsafe LLM behavior from reaching production by creating and validating an evaluation pipeline that mirrors real failure modes.

Core Features & Use Cases

  • Failure analysis & error taxonomy: identifies the most important ways LLM outputs fail (e.g., hallucinations, format violations, instruction drift) to focus eval effort.
  • Synthetic data & golden sets: generates diverse test cases and human-verified examples to cover happy paths and edge/adversarial cases.
  • LLM-as-judge with calibration: builds targeted binary judges per failure type, then calibrates them against human labels and reports agreement metrics.
  • RAG eval support: separates retrieval quality from generation quality when retrieval-augmented generation is involved.
  • CI automation: generates an eval runner and integrates regression detection so PRs that touch AI features can be assessed automatically.

Quick Start

Run /llm-eval full pipeline to audit existing evals and build a complete, calibrated evaluation setup for all detected AI features in your codebase.

Frequently Asked Questions about llm-eval

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I build an LLM evaluation pipeline for regression testing?▼

Build an LLM evaluation pipeline by analyzing failure modes, generating synthetic test data, designing calibrated LLM-as-judge criteria, and automating CI regression checks to prevent unsafe behavior in production.

What is LLM-as-judge calibration and how does it work?▼

LLM-as-judge calibration builds targeted binary judges per failure type, then validates them against human-verified golden sets and reports agreement metrics to ensure reliable automated evaluation.

How do I evaluate RAG retrieval and generation quality separately?▼

Evaluate RAG quality by separating retrieval metrics from generation metrics, applying specialized evaluation criteria to isolate index retrieval failures from hallucinations in the final output.

How do I generate synthetic test data for LLM edge cases?▼

Generate synthetic test data by creating diverse test cases and human-verified golden sets that cover happy paths, edge cases, and adversarial scenarios to mirror real failure modes.

Can I automate LLM feature regression checks in CI?▼

Automate LLM feature regression checks in CI by generating an eval runner that integrates regression detection, allowing pull requests that touch AI features to be assessed automatically.

What is the best way to audit existing LLM evaluations?▼

Audit existing LLM evaluations by running a full pipeline workflow that detects AI features in your codebase and builds a complete, calibrated evaluation setup for all identified failure modes.

Related Skills

Explore More Software Engineering Skills→
MantisWareMantisWare

error-analysis

Analyze LLM traces and logs to categorize failures into a structured error taxonomy.

Community
Advanced
HouseofmvpsHouseofmvps
115

evals

Generate characterization tests and Promptfoo eval suites for CI ship-gates.

Community
Advanced
hamelsmuhamelsmu
1.6k

evaluate-rag

Evaluate RAG pipelines by measuring retrieval metrics and generation quality.

Community
Advanced
rohunvorarohunvora
5

advanced-evaluation

Automate LLM-as-a-judge scoring, pairwise comparison, and rubric generation with bias mitigation.

Community
Advanced