auto-eval

Run offline evaluation against pre-captured fixtures to verify AI agent quality without hitting live ADO or LLM APIs. This catches regressions from prompt or rule changes by testing against saved inputs and outputs locally using Node.js.

Offline evaluation is the process of testing AI automation agents against pre-captured fixtures to ensure quality. It verifies that changes to prompts, rules, or agent steps do not regress behavior without requiring live API connections.

Yes, you need a local environment with Node.js installed to execute the evaluation framework. You run the tests by executing node eval/run.js from the .ai/automation directory where the required fixtures are located.

Yes, you can enable an optional tier-2 mode for real LLM checks during offline evaluation. This allows you to perform full LLM checks alongside the standard fixture-based tests for agents like dor, pr-review, and pr-answer.

Apply the offline evaluation framework after making changes to prompts, rules, or agent steps. It runs tests against pre-captured fixtures across all agent types, including pr-review, to catch behavioral regressions before deployment.

You should avoid live APIs when testing prompt or rule changes to prevent regressions and save API costs. Use offline evaluation with pre-captured fixtures instead to validate agent behavior safely before deployment.