evaluation

Evaluate agent outputs with multi-dimensional rubrics, test sets, and production monitoring.

Updated Apr 7, 2026
One-click install
npx skills add https://github.com/NT-boop-star/BRMV-tract --skill evaluation-nt-boop-star
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: evaluation
Source: https://github.com/NT-boop-star/BRMV-tract/tree/main/antigravity/skills/evaluation
Command: npx skills add https://github.com/NT-boop-star/BRMV-tract --skill evaluation-nt-boop-star

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve? Agent systems are non-deterministic and lack single correct answers, making traditional software testing inadequate. This Skill provides a framework for measuring agent quality across multiple dimensions, catching regressions, and validating context engineering decisions. ## Core Features & Use Cases - Multi-Dimensional Rubric Scoring: Score agent outputs on factual accuracy, completeness, citation accuracy, source quality, and tool efficiency with configurable weights and pass thresholds. - Test Set Management: Build complexity-stratified test sets (simple to very complex) with tag-based filtering and distribution analysis. - Production Monitoring: Sample live agent interactions, track pass rates over time, and trigger alerts at warning (0.85) and critical (0.70) thresholds. - Use Case: Before deploying an updated research agent, run a 50-case stratified test set through the EvaluationRunner, compare per-dimension scores against the baseline, and block deployment if any dimension regresses. ## Quick Start Ask the agent to evaluate a test set of agent outputs using the default five-dimension rubric and report per-dimension scores with an overall pass rate.

Frequently Asked Questions about evaluation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I evaluate LLM agent performance with rubrics?

Define weighted dimensions such as factual accuracy, completeness, and tool efficiency, then score each output per dimension and compute a weighted overall score. The AgentEvaluator class applies a default five-dimension rubric with a 0.7 pass threshold.

How many test cases do I need for reliable agent evaluation?

Start with 20-30 cases during early development, then scale to at least 50 cases for reliable signal. Stratify tests across simple, medium, complex, and very complex levels so easy cases do not inflate overall scores.

Should I use LLM-as-judge or human evaluation for agent testing?

Use LLM-as-judge for scalable evaluation across large test sets, but supplement with human review for edge cases and hallucinations. Use a different model family as the judge than the model being evaluated to avoid self-enhancement bias.

How do I monitor agent quality in production?

Sample a fraction of live interactions with the ProductionMonitor class, score each sample, and track pass rate over time. Set alerts when pass rate drops below 0.85 (warning) or 0.70 (critical).

Why does a single overall score hide agent quality problems?

A single aggregate number averages away dimension-specific failures, such as high accuracy with poor tool efficiency. Always report per-dimension scores alongside the aggregate and fail the evaluation if any dimension falls below its minimum threshold.