os-eval-runner

You can automate skill evaluation loops using headless Python scripts that score iterations and apply KEEP or DISCARD gating decisions. The engine reads a target directory and evals.json to emit objective metrics without side effects.

Validating a proposed skill change is best done using single-shot QA mode, which empirically scores the specific mutation before applying it. The evaluation engine checks routing accuracy and structural heuristics against the baseline.

Yes, running autonomous evaluation loops requires git and Python 3.8 or higher. Git manages the repository for reverting discarded changes, while Python executes the evaluation and scoring scripts.

The evaluation gate decides to keep or revert skill improvements by scoring iterations against an enforced baseline. It evaluates routing accuracy, structural heuristics, and an F1 guard, then appends results to a per-target ledger.

Yes, you can initialize standard templates for autoresearch experiments using a scaffold script. It deploys program, evals, and results templates into your experiment directory to structure the evaluation loop.

Stateless skill evaluation is limited by its lack of side effects, meaning it only emits objective scores from evals.json without modifying the target. It requires a git repository to manage baseline enforcement and reverts for discarded iterations.