skills.rest
nshaikhs
nshaikhsCOMMUNITY
Explore all skills by this author →
GitHub

evaluation

Evaluate agent outputs against multi-dimensional rubrics to compute scores and detect regressions.

Updated Apr 25, 2026
Search Tags:#metrics#evaluation#rubric#llm-judge#production-monitoring#test-sets
One-click install
npx skills add https://github.com/nshaikhs/claude-code-for-operators --skill evaluation-nshaikhs
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: evaluation
Source: https://github.com/nshaikhs/claude-code-for-operators/tree/main/skills/context-engineering/skills/evaluation
Command: npx skills add https://github.com/nshaikhs/claude-code-for-operators --skill evaluation-nshaikhs

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve?

This skill provides a structured framework to evaluate agent outputs using multi-dimensional rubrics, enabling consistent benchmarking, regression detection, and quality gates across development and production.

Core Features & Use Cases

  • Flexible rubrics with per-dimension weights to produce an overall score and pass/fail verdict.
  • Test-set management and automated evaluation pipelines, including generation of per-dimension scores, average statistics, and failure traces.
  • Production monitoring support via sampling and alerting to maintain quality over time.

Quick Start

Use an AgentEvaluator with a TestSet and run an EvaluationRunner to generate a production-ready quality summary.

Frequently Asked Questions about evaluation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I evaluate agent outputs against a custom rubric?▼

To evaluate agent outputs against a custom rubric, you configure a multi-dimensional rubric with per-dimension weights. The evaluation runner processes your test sets to produce an overall score, pass/fail verdicts, and per-dimension averages.

What is the best way to monitor LLM agent quality in production?▼

Monitoring LLM agent quality in production involves continuous sampling of outputs and alerting against a structured rubric. This approach detects regressions over time by computing pass rates and dimension averages to maintain quality gates.

Can I use a multi-dimensional rubric for benchmarking agent configurations?▼

Yes, you can use flexible rubrics with per-dimension weights for benchmarking agent configurations. The framework quantifies quality across different setups, generating per-dimension scores and average statistics to compare performance consistently.

Do I need a test set to run an automated evaluation pipeline?▼

Yes, an automated evaluation pipeline requires a test set to evaluate outputs effectively. You use an AgentEvaluator with a TestSet and run an EvaluationRunner to generate a production-ready quality summary with failure traces.

How does per-dimension scoring detect agent regressions?▼

Per-dimension scoring detects agent regressions by quantifying specific quality dimensions over time. By tracking dimension averages and pass rates through production sampling, the system triggers alerts when performance drops below defined thresholds.

Related Skills

Explore More Data & Analytics Skills→
FVossebeldFVossebeld

evaluation

Evaluate agent outputs with weighted rubric criteria and per-dimension scores.

Community
Advanced
wangx7wangx7

agent-platform-alert-configuration

Configure dynamic Vertex AI Agent Platform alert policies with Terraform and PromQL.

Community
Advanced
enigmaicon-engenigmaicon-eng
1

evaluation

Score agent outputs across quality dimensions with pass/fail thresholds.

Community
Advanced
GamezarGamezar

sadd-judge

Launch an isolated sub-agent judge to score conversation work with evidence-based rubrics.

Community
Advanced