m365-agent-evaluator

You can evaluate Microsoft 365 Copilot agents by running the @microsoft/m365-copilot-eval CLI to analyze outputs for relevance, coherence, groundedness, and similarity. This requires setting up schema-compatible eval datasets to test specific agent behaviors.

Testing M365 Copilot declarative agents requires a Microsoft 365 Copilot license, a configured agent, and Azure OpenAI configuration. You also need Node.js to run the @microsoft/m365-copilot-eval CLI package for executing the evaluation suites.

Yes, the M365 Copilot evaluation CLI supports both single-turn and multi-turn evaluations for declarative agents. It processes schema version 1.2.0 documents to simulate conversational flows and analyze iterative agent performance.

You create eval datasets by designing schema-compatible JSON documents that define specific prompts for your declarative agents. These datasets are then passed to the evaluation CLI to measure agent behaviors across various test scenarios.

The best way to analyze Copilot agent evaluation results is by reviewing the generated output scores for relevance, coherence, groundedness, and similarity. You can then apply iterative fixes to your agent based on these metrics to improve production performance.

M365 Copilot agent evaluation requires Azure OpenAI configuration to process and score the groundedness and similarity of generated responses. This setup ensures the evaluation CLI can accurately measure agent performance against expected behaviors.