What problem does it solve? Most prompt, config, and agent tuning changes are unverified guesses, and stochastic LLM outputs make "it looks better" almost always noise. This Skill enforces a disciplined loop: state a hypothesis and observable, measure a baseline, change exactly one thing, run multiple rollouts, and keep or revert based on evidence. ## Core Features & Use Cases - Structured experiment loop: Hypothesis plus concrete observable, baseline measurement, one isolated change, N rollouts, and an explicit keep/revert/inconclusive verdict with a recorded result. - Real-agent A/B testing: Drives the actual Claude or Codex TUI through the index Python harness (tui.harness.Claude) in a PTY, so runs are live-visible on the web dashboard and rollouts are a simple asyncio gather loop. - Confound control: Enforces same-model baselines, neutral non-leading tasks, fresh agents per rollout, and rate-based comparison instead of single samples. - Use Case: You tweak a global skill prompt and want to know if it actually helps. Apply the change, run 5 baseline and 5 candidate rollouts of a representative task through the TUI harness, compare success rates, and keep the change only if it measurably wins. ## Quick Start Ask the agent to A/B test your prompt change by running five baseline and five post-change rollouts of a representative task through the TUI harness and reporting the keep-or-revert verdict with rates.
