What problem does it solve?
Choosing between competing tools, libraries, or implementations often relies on anecdote or vendor claims rather than controlled evidence. This Skill enforces a rigorous, reproducible A/B benchmarking process so decisions rest on measured correctness and performance data instead of guesswork.
Core Features & Use Cases
- Controlled Experiment Design: Freezes scenarios, oracles, metrics, repetition counts, and decision rules before any candidate runs, preventing biased or post-hoc conclusions.
- Symmetric Harness and Activation Proof: Runs every candidate through identical isolated worktrees, runners, and logging, and verifies each candidate actually executed rather than silently falling back.
- Validity-First Analysis: Grades correctness before speed, separates setup from steady-state cost, reports spread and failure classes, and produces WIN, TIE, INCONCLUSIVE, or BLOCKED verdicts with cleanup confirmation.
- Use Case: When deciding between two HTTP client libraries, define representative request workloads and a correctness oracle, run both candidates repeatedly in clean worktrees, and receive a verdict backed by raw per-run evidence.
Quick Start
Use the benchmark comparator skill to design and run a controlled A/B comparison between these two implementations and tell me which one to adopt.
