ab-test-zephyr

Create git worktrees at the merge base (control) and PR head (treatment), then launch matched zephyr_benchmark Iris jobs against the same immutable sample with identical pool shapes. Compare per-stage CPU time, elapsed time, and memory from Finelog before posting a verdict.

Use cpu_time_total as the primary efficiency signal since it excludes scheduling noise, and treat elapsed as a secondary latency signal sensitive to stragglers. Verify items and bytes_processed match to confirm workload equivalence, and watch mem_peak_bytes_max for OOM regressions.

Collect execution IDs from Iris job logs, then run a Finelog query against the zephyr.stage namespace filtering by those IDs and status END. Aggregate CPU, elapsed, items, and bytes per arm and join control to treatment by stage_name to compute deltas.

Yes, but only if the baseline report contains the same workload fingerprint (sample, stage range, sources, pool shape, cluster) and its Finelog execution IDs remain queryable. Otherwise launch a fresh matching control to keep the comparison valid.

Skip the remote benchmark with reviewer agreement when the change only touches documentation, tests, types, or log text. For code changes, run local Zephyr and Datakit tests first before paying for remote workers.

Comparisons are inconclusive when workload fingerprints differ, a stage fails, item or byte counts mismatch, or infrastructure churn like preemptions and stragglers distorts elapsed time. Re-run under matched conditions, interleaving control trials if the verdict depends on wall time.