eval-robuste

What problem does it solve?

Measures the variance in stochastic skill evaluations (skill-review) by running N parallel instances, aggregating results deterministically, and producing a clear verdict against a reference to decide if a perceived change is noise or meaningful.

Core Features & Use Cases

• Parallel robust runs: launches N skill-review evaluations concurrently and collects raw scores.
• Deterministic aggregation: delegates all stats to a dedicated aggregator to compute median, standard deviation, confidence-like interval, and verdict.
• Baseline management: compares to archived baselines and emits a clear signal whether a change is BRUIT, REGRESSION, or AMELIORATION.
• Use Case: when a skill's score oscillates across passes, enabling a validated baseline σ before declaring regression.

Quick Start

Invoke the robust evaluation on a target skill to quantify evaluator noise and determine if observed changes indicate a true regression.