eval-analysis

You can detect eval regressions by comparing baseline versus candidate run data, pulling per-scenario observations, and reading prompt configurations to identify signal quality drops and potential regressions for targeted adjustments.

Anti-bias gating in eval analysis applies safeguard rules to proposed prompt and config changes, ensuring modifications are safe and targeted while documenting counter-scenarios to prevent skewed evaluation outcomes.

Diagnose signal quality by gathering recent eval run IDs and timestamps, drilling into affected scenarios to inspect inputs and scoring criteria, then reading relevant prompts in src/engine/prompts and config defaults.

Yes, comparing baseline versus candidate eval results surfaces data-driven findings that guide safe prompt and config modifications, outputting actionable recommendations structured as data, hypothesis, and action.

The best way to structure eval findings is a three-part format: data representing observed signal quality, hypothesis explaining the regression, and actionable recommendations such as adding scenarios or modifying prompts and configs.

Investigate eval scenario details when you need to inspect specific inputs, scoring criteria, and notes alongside config defaults in src/engine/config.ts to understand anomalies before safely adjusting prompts.