ESS Eval Quality Validator

What problem does it solve?

Building Microsoft Employee Self-Service (ESS) agents requires high-quality evaluation files to accurately test agent performance, but manual review of eval files is time-consuming and prone to oversight. This Skill automates the quality scoring of evaluation YAML files to catch issues like poor coverage, unrealistic inputs, and redundant test cases before they skew agent testing results or make it to production.

Core Features & Use Cases

• Automated Quality Scoring: Runs a Python script that calls the Copilot API to score eval files across 8 key quality dimensions in seconds, with a manual fallback option for offline use.
• Actionable Triage: Generates a structured report with clear pass/review/fail gates, flagging specific test cases that contribute to low scores so you can fix issues quickly.
• Use Case: A maker building an ESS HR agent can run this validator on their topic-triggering evaluation set to identify low-diversity inputs and missing boundary cases before pushing the agent to production.

Quick Start

Use this skill to validate the quality of your ESS evaluation YAML files and get a prioritized list of issues to fix before deploying your agent.