model-ab-test

You can automate A/B testing for model downgrades by running parallel tests between current and proposed models, scoring outputs against predefined criteria, and generating a formal pass/fail verdict.

Automated A/B testing for model evaluation is a process that dispatches parallel subagents to compare current and proposed models, scoring their outputs against defined criteria to quantify performance differences.

You can run parallel A/B tests across agents by orchestrating two subagents within a single prompt, evaluating outputs against a YAML test-definition file that covers multiple test cases.

Yes, automated model evaluation requires a YAML test-definition file to validate proposed models across skills or agents, applying predefined criteria to multiple test cases for accurate scoring.

You generate evidence files for model evaluation by running automated A/B tests that write per-test evidence documentation, ensuring every output comparison and verdict is formally recorded.

Binary scoring in A/B model evaluation strictly scores each criterion as pass or fail, which simplifies the final verdict but may lack the nuance for partial performance differences between models.