pop-benchmark-runner

Automated side-by-side benchmark trials compare PopKit-enabled Claude Code against a baseline by orchestrating parallel runs across separate workspaces, collecting detailed recordings, and generating comprehensive reports with statistical analysis like t-tests and Cohen's d.

Benchmark performance analysis uses t-tests, Cohen's d, and confidence intervals to evaluate statistical significance between paired trials, producing objective measurements delivered in both markdown and HTML report formats for stakeholders or CI pipelines.

Shareable benchmark reports are generated automatically in markdown and HTML formats after orchestrating trials and collecting recordings, providing objective performance measurements suitable for CI pipelines, team dashboards, or stakeholder distribution.

The benchmark orchestrator supports task definitions and automated responses to ensure repeatable, automated performance analytics across tasks and environments, allowing software engineers to maintain consistent testing parameters across multiple benchmark runs.

Yes, pyyaml is a required dependency for running the benchmark orchestrator, which automates trial orchestration, recording collection, statistical analysis, and report generation across paired PopKit and baseline environments.