eval-suite-planner

Plan an AI agent evaluation suite by providing a plain-English agent description to generate a structured eval plan, mapping to core business, capability, safety, edge-case, and variation scenarios. This grounds the plan in Microsoft's MS Learn agent evaluation guidance.

Stage 1 Define planning establishes the foundational eval suite structure before execution. It routes business problems to capability scenario types like Information Retrieval and Knowledge Grounding, mapping to core plan outputs including scenario tables and quality signals.

Generate test scenarios for Copilot Studio agent evaluation by inputting an agent description. The system produces a scenario table featuring core business, capability, edge-case, and variation tests, alongside primary and secondary evaluation methods per scenario.

Yes, the MS Learn evaluation framework supports Information Retrieval and Knowledge Grounding agents by routing Step 0 to specific capability scenario types. It generates quality signals mapping and pass/fail thresholds tailored to these business problems.

An eval plan for AI agent evaluation includes a detailed scenario table, quality signals mapping, pass/fail thresholds, and planning rationale. It supports optional assets, scripts, and references while satisfying frontmatter constraints.

Set pass/fail thresholds for AI evaluation scenarios by generating a structured eval plan from an agent description. The plan automatically produces accompanying quality signals mapping and thresholds that explain how the plan enables Stage 2 execution.