generate-evaluation-test-sets

What problem does it solve?

Manually creating evaluation test sets for Copilot Studio agents is time-consuming, requires deep familiarity with the agent's topic structure, and is prone to formatting errors that break deployment. This skill automates the entire workflow, from analyzing agent topics to generating categorized, standards-compliant test cases and pushing them directly to Copilot Studio via Dataverse.

Core Features & Use Cases

• Automated Topic Analysis: Classifies all agent topics into user-facing and system triggers, groups them by functional area, and extracts key details for test generation.
• Categorized Test Generation: Produces test sets for 7 standard categories (Topic Triggering, Ambiguous Prompts, Responsible AI, Sensitive Topics, Emotional Intelligence, Integration Data, Multi-Turn) with positive, boundary, and negative cases per topic.
• Direct Deployment: Pushes validated test sets directly to Copilot Studio via Dataverse, eliminating manual YAML editing and upload steps. Use Case: For an HR self-service agent with 25+ employee-facing topics, this skill generates 100+ test cases across all categories in minutes, replacing 4+ hours of manual work and ensuring consistent test coverage.

Quick Start

Use the generate evaluation test sets skill to create a full evaluation test suite for your Copilot Studio agent and push it directly to your Dataverse environment.