eval-before-optimize

What problem does it solve?

Verifying eval precision before claiming improvements from RL/ES post-training to prevent mistaking noise for learning.

Core Features & Use Cases

• Validate eval noise floor by repeating evaluations across seeds and runs to establish a robust baseline.
• Compute required sample size and baseline variance to determine when results are statistically meaningful.
• Apply guardrails before proceeding with post-training optimizations to avoid pursuing spurious gains.

Quick Start

Run a baseline eval to measure noise and then decide whether the observed improvement is confident enough to pursue post-training optimization.