eval-generator

Name: eval-generator
Availability: InStock
Author: Microsoft

Generate eval test cases from an eval suite plan or agent description.

123|20|Updated Mar 30, 2026

Search Tags:#testing #test-generation #eval #agent-evaluation #copilot-studio #conversational-eval

Ready to run

npx skills add https://github.com/microsoft/eval-guide --skill eval-generator

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: eval-generator
Source: https://github.com/microsoft/eval-guide/tree/main/skills/eval-generator
Command: npx skills add https://github.com/microsoft/eval-guide --skill eval-generator

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Generates concrete eval test cases for an agent by using an eval suite plan or a plain-English agent description, enabling structured evaluation across lifecycle stages.

Core Features & Use Cases

Use plan-based generation: leverages the eval suite plan's scenario table, evaluation methods, quality signals, and tags to produce test inputs and configurations.
Fallback generation: if no plan exists, creates 6-8 test cases from a plain-English agent description, including happy-path, edge, and adversarial scenarios.
Output formats: produces a Copilot Studio test set table, a CSV for single-response import, and a docx-style report for human review.
Lifecycle integration: supports subsequent steps in the cycle with /eval-result-interpreter and /eval-triage-and-improvement.

Quick Start

Run /eval-generator with your agent description to generate test cases.

Frequently Asked Questions about eval-generator

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I generate test cases for an agent using a plain-English description?▼

To generate test cases from an agent description, provide a clear agent task and success criteria. The tool creates 6-8 test cases covering happy-path, edge, and adversarial scenarios without requiring a formal plan.

What output formats can I get for agent evaluation test sets?▼

Agent evaluation test sets are output as a Copilot Studio test set table, a CSV file for single-response imports, and a docx-style report for human review.

Do I need an eval suite plan to generate conversational eval test cases?▼

No, an eval suite plan is not required. While plan-based generation uses a scenario table and evaluation methods, you can fallback to generating tests from a plain-English agent description.

Can I generate multi-turn evaluation scenarios for conversational agents?▼

Yes, multi-turn evaluation is supported. You can generate conversational eval test cases using either an eval suite plan or a plain-English agent description, though CSV output is limited to single-response mode.

What is the best way to create adversarial test scenarios for Copilot Studio agents?▼

The best way to create adversarial scenarios is using the fallback generation mode, which automatically derives 6-8 tests including adversarial cases from a plain-English agent description and success criteria.

Why does my agent evaluation CSV import only support single-response tests?▼

The CSV output format is restricted to single-response imports by design. Multi-turn evaluation scenarios are supported in the Copilot Studio test set table and the docx-style report instead.