bat-story-eval

Benchmark MCP tool behavior by comparing target runs against a baseline using pre-built and custom stories with diff-based evaluation. This highlights regressions and behavioral drift for traceable regression detection.

Diff-based evaluation for MCP regression testing compares the outputs of baseline and target tool versions using structured stories. It validates analysis workflows to ensure traceable detection of behavioral changes.

Triage changes by selecting representative pre-built stories based on diff impact, or design custom stories to exercise new code paths. This ensures targeted evaluation of potential regressions across versions.

Yes, you can document and compare results across baseline and target versions. The evaluation workflow validates story catalogs and reporting workflows to ensure traceability throughout the regression detection process.

You need custom stories when triaging changes and pre-built stories are insufficient to exercise new code paths. Designing custom tales allows you to target potential regressions and validate specific behavior across baseline and target comparisons.

Diff-based story evaluation requires both a baseline and target run to compare MCP tool behavior. Without establishing a baseline, the evaluation cannot detect regressions or validate story catalogs for traceable reporting.