evaluation

Evaluate AI agent performance using rubrics and degradation testing.

1|Updated Feb 4, 2026

Search Tags:#testing #quality assurance #llm #evaluation #rubrics #agent performance

Ready to run

npx skills add https://github.com/bthillerup/bens-garage-session-2 --skill evaluation-bthillerup

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: evaluation
Source: https://github.com/bthillerup/bens-garage-session-2/tree/main/.github/skills/evaluation
Command: npx skills add https://github.com/bthillerup/bens-garage-session-2 --skill evaluation-bthillerup

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill provides a framework and methodologies for systematically evaluating the performance and quality of AI agents, addressing the unique challenges posed by their non-deterministic nature.

Core Features & Use Cases

Performance Measurement: Systematically test agent performance and validate context engineering choices.
Quality Gates: Build quality gates for agent pipelines and compare different agent configurations.
Use Case: A product manager wants to ensure that a new version of their customer support agent meets predefined quality standards before deploying it to production. They use this skill to run a suite of tests and analyze the results against a rubric.

Quick Start

Use the evaluation skill to build a test framework for agent quality.

Frequently Asked Questions about evaluation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I evaluate AI agent performance when outputs are non-deterministic?▼

Evaluating AI agent performance requires methodologies that address non-determinism by using rubrics, LLM-as-Judge, and end-state evaluation. This allows you to systematically test agent performance and validate context engineering choices despite varying outputs.

What is the best way to build quality gates for LLM agent pipelines?▼

Building quality gates for LLM agent pipelines involves applying systematic evaluation rubrics and complexity stratification to test results. You can compare different agent configurations against predefined quality standards before deploying to production.

How does LLM-as-Judge work for testing agent quality?▼

LLM-as-Judge works by using a language model to evaluate agent responses against specific rubrics and quality standards. It provides a scalable methodology to measure agent performance and validate context engineering without relying solely on human evaluation.

Can I use degradation testing to measure context dependency in agent systems?▼

Yes, degradation testing measures context dependency by stratifying complexity and evaluating the end-state of agent systems. This approach identifies how agent performance degrades under varying conditions and validates context engineering choices.

When should I use human evaluation instead of automated rubrics for AI agents?▼

Human evaluation is necessary when assessing nuanced context-dependency and subjective quality standards that automated rubrics or LLM-as-Judge cannot capture. It complements systematic testing by validating agent performance against human expectations.

What are the limitations of using end-state evaluation for non-deterministic agents?▼

End-state evaluation limitations include overlooking intermediate reasoning errors in non-deterministic agents and missing context-dependency issues. Degradation testing and complexity stratification are required to capture the full scope of agent performance variations.