Evals-Driven Development

What problem does it solve?

This Skill provides a structured approach to evaluating and iterating on deep agent systems, ensuring they meet their intended goals with high accuracy and reliability.

Core Features & Use Cases

• Define Scenarios: Create detailed scenarios from Job-To-Be-Done descriptions, covering happy paths, edge cases, and failures.
• Evaluate Agents: Use snapshot testing, smoke testing, and full eval reviews to assess agent performance and functionality.
• Iterate and Expand: Add new scenarios, refine existing ones, and expand the dataset for continuous improvement.
• Multi-Agent Systems: Evaluate routing, subagent execution, and system-level integration with hierarchical evaluation patterns.
• Safety and Security: Ensure agents handle edge cases, provide correct responses, and comply with security protocols.
• Use Case: Imagine you are developing an AI agent for customer support. Use this Skill to design test scenarios, evaluate its performance, and ensure it handles edge cases like account recovery and order tracking.

Quick Start

Run the /design-evals command to generate a baseline eval suite from your JTBD.