What problem does it solve? Agent systems are non-deterministic and lack single correct answers, making traditional software testing inadequate. This Skill provides a structured methodology and Python framework for measuring agent quality through outcome-based rubrics, regression detection, and continuous production monitoring. ## Core Features & Use Cases - Multi-Dimensional Rubric Scoring: Score agent outputs across factual accuracy, completeness, citation accuracy, source quality, and tool efficiency with configurable weights and pass thresholds. - Test Set Management: Build complexity-stratified test sets (simple to very complex) with tag-based filtering and distribution analysis to prevent easy cases from inflating scores. - Production Monitoring: Sample live interactions at configurable rates, track pass rates against warning (0.85) and critical (0.70) thresholds, and generate alerts for quality degradation. - Use Case: Before deploying an updated research agent, run a 50-case stratified test set through the EvaluationRunner, compare per-dimension scores against the baseline, and block deployment if factual accuracy regresses. ## Quick Start Use the evaluation skill to score my agent's outputs against a multi-dimensional rubric and summarize pass rates across a stratified test set.
