skills.rest
microsoft
microsoftOFFICIAL
Explore all skills by this author →
Website𝕏@OpenAtMicrosoftGitHub

generate-evaluation-test-sets

Generates categorized evaluation test sets for Microsoft Copilot Studio agents.

25|8|Updated Apr 8, 2026
Search Tags:#quality assurance#test generation#agent testing#dataverse#copilot studio#ess agent#evaluation tests
One-click install
npx skills add https://github.com/microsoft/Employee-Self-Service-Agent-Developer-Kit --skill generate-evaluation-test-sets
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: generate-evaluation-test-sets
Source: https://github.com/microsoft/Employee-Self-Service-Agent-Developer-Kit/tree/main/solutions/ess-maker-skills/src/skills/evaluations/create
Command: npx skills add https://github.com/microsoft/Employee-Self-Service-Agent-Developer-Kit --skill generate-evaluation-test-sets

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) components.

What problem does it solve?

Manually creating evaluation test sets for Copilot Studio agents is time-consuming, requires deep familiarity with the agent's topic structure, and is prone to formatting errors that break deployment. This skill automates the entire workflow, from analyzing agent topics to generating categorized, standards-compliant test cases and pushing them directly to Copilot Studio via Dataverse.

Core Features & Use Cases

  • Automated Topic Analysis: Classifies all agent topics into user-facing and system triggers, groups them by functional area, and extracts key details for test generation.
  • Categorized Test Generation: Produces test sets for 7 standard categories (Topic Triggering, Ambiguous Prompts, Responsible AI, Sensitive Topics, Emotional Intelligence, Integration Data, Multi-Turn) with positive, boundary, and negative cases per topic.
  • Direct Deployment: Pushes validated test sets directly to Copilot Studio via Dataverse, eliminating manual YAML editing and upload steps. Use Case: For an HR self-service agent with 25+ employee-facing topics, this skill generates 100+ test cases across all categories in minutes, replacing 4+ hours of manual work and ensuring consistent test coverage.

Quick Start

Use the generate evaluation test sets skill to create a full evaluation test suite for your Copilot Studio agent and push it directly to your Dataverse environment.

Frequently Asked Questions about generate-evaluation-test-sets

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I automate evaluation test generation for Copilot Studio agents?▼

Automate evaluation test generation for Copilot Studio agents by analyzing agent topic files to produce categorized test cases. This skill generates standards-compliant tests for topic triggering, ambiguous prompts, responsible AI, sensitive topics, emotional intelligence, integration data, and multi-turn conversations, then pushes them to Dataverse.

What categories of test cases are needed for Microsoft Copilot Studio quality assurance?▼

Copilot Studio quality assurance requires 7 test categories: topic triggering, ambiguous prompts, responsible AI, sensitive topics, emotional intelligence, integration data, and multi-turn conversations. Each category includes positive, boundary, and negative test cases per topic to ensure comprehensive agent evaluation.

Can I push test sets directly to Copilot Studio via Dataverse without manual YAML editing?▼

Yes, you can push validated test sets directly to Copilot Studio via Dataverse without manual YAML editing. The skill automates the entire workflow from topic analysis to deployment, eliminating manual upload steps and preventing formatting errors that break deployment.

Does automated test generation work for agents with 25 or more topics?▼

Automated test generation works efficiently for agents with 25 or more topics, producing 100-plus test cases across all categories in minutes. It replaces 4-plus hours of manual work by classifying user-facing and system triggers, grouping them by functional area, and extracting key details for comprehensive test coverage.

What's the best way to create multi-turn conversation tests for Copilot Studio agents?▼

The best way to create multi-turn conversation tests is to automate generation by analyzing agent topic files. This skill produces categorized test sets covering multi-turn scenarios alongside topic triggering, ambiguous prompts, responsible AI, sensitive topics, emotional intelligence, and integration data tests, all pushed directly to Dataverse.

Why do manually created evaluation test sets fail during Copilot Studio deployment?▼

Manually created evaluation test sets fail during Copilot Studio deployment due to formatting errors and inconsistent test coverage. Manual creation requires deep familiarity with the agent's topic structure and is prone to YAML formatting errors that the automated skill prevents through validated direct Dataverse integration.

Related Skills

Explore More Product & Management Skills→
javiarmestojaviarmesto

bc-topic-patterns

Provide topic patterns and AdaptiveDialog templates for Business Central scenarios in Copilot Studio.

Community
Intermediate
javiarmestojaviarmesto

copilot-studio-troubleshoot

Validate YAML syntax and diagnose topic routing in Copilot Studio configurations.

Community
Intermediate
yuseidemoyuseidemo

ai-builder-prompt-skill

Build AI Builder GPT Dynamic Prompts and publish them as Copilot Studio agent tools.

Community
Advanced
taipt1504taipt1504
3

skill-creator

Create, modify, and improve Claude skills for workflow automation.

Community
Advanced