exp-pilot-eval

Pilot experiment results are evaluated by reading pilot JSON results, pilot logs, and a pilot spec YAML file to compare metrics against baselines, generating a pass, fail, or inconclusive verdict.

Update the idea wiki page by writing the pilot_result, appending a mandatory [pilot] prefix to the failure_reason on failure, and modifying the status field to reflect the evaluated outcome.

Generate a persistent experiment verdict report by writing to experiments/pilot/{slug}/report.md and appending an audit entry to wiki/log.md, while also printing the PILOT_VERDICT_REPORT to the terminal.

Yes, you can automate pilot evaluation by including the --auto flag when running the evaluation, allowing the workflow to proceed without pausing for manual confirmation.

Forward-only lifecycle rules are enforced during pilot evaluation to prevent reverting research idea statuses, ensuring that once a pilot verdict is reached, the idea progresses sequentially without backward state transitions.

You need the specific idea slug, the pilot spec YAML file defining success criteria, the pilot JSON results, and pilot logs to execute the evaluation and determine the experiment verdict.