eval-generator

What problem does it solve?

Generates concrete eval test cases for an agent by using an eval suite plan or a plain-English agent description, enabling structured evaluation across lifecycle stages.

Core Features & Use Cases

• Use plan-based generation: leverages the eval suite plan's scenario table, evaluation methods, quality signals, and tags to produce test inputs and configurations.
• Fallback generation: if no plan exists, creates 6-8 test cases from a plain-English agent description, including happy-path, edge, and adversarial scenarios.
• Output formats: produces a Copilot Studio test set table, a CSV for single-response import, and a docx-style report for human review.
• Lifecycle integration: supports subsequent steps in the cycle with /eval-result-interpreter and /eval-triage-and-improvement.

Quick Start

Run /eval-generator with your agent description to generate test cases.