empirical-systems-evaluation

What problem does it solve?

Provides a complete, reproducible methodology to design, run, and report experiments that measure multi-agent coordination systems with statistical rigor, ensuring claims are supported by confidence intervals, effect sizes, and honest threats-to-validity.

Core Features & Use Cases

• Precise metric definitions for latency, throughput, recovery time, and proportions with instrumentation guidance.
• Statistical decision trees to choose parametric vs non-parametric tests, paired vs independent analyses, and multiple-comparison corrections.
• Human evaluation protocols including anchored rubrics, rater calibration, and inter-rater reliability requirements.
• Sample size planning and bootstrap CI procedures (B ≥ 10,000 recommended) and worked examples for crash-recovery benchmarks.
• Use Case: comparing salvage protocols for crash recovery across coordination protocols with automated latency metrics and blinded human fidelity ratings.

Quick Start

Design an experiment comparing your coordination protocols by defining metrics (salvage latency, recovery fidelity), choosing baselines, performing a power-based sample size calculation, pre-registering an analysis plan with bootstrapped CIs and effect sizes, and running blinded human ratings with inter-rater reliability checks.