eval-optimize

To automate iterative skill improvements, this tool analyzes evaluation results, reads judge rationales and transcripts, edits the SKILL.md file to fix identified issues, and re-runs validations until success criteria are met or max iterations are reached.

The best way to fix judge failures is to use an automated loop that reads judge rationales and traces, applies targeted fixes to the SKILL.md entry point, and re-runs the eval suite to verify that regressions are caught and resolved.

Automatic skill editing works by reading and updating the skill's entry point, coordinating with eval.yaml configurations to support per-case and batch modes, and running validation cycles to verify improvements without requiring user prompts.

Yes, you can run validation cycles without manual intervention by triggering the optimization workflow, which autonomously analyzes eval results, edits the skill, and re-runs validations until all judges pass or the maximum iteration limit is reached.

The main limitation of automating skill edits is that the improvement cycle stops when the maximum iterations are reached, meaning if complex judge failures require more cycles, the skill may not fully pass all validation criteria autonomously.

Your skill optimization loop stops before all judges pass because it operates within a maximum iteration limit to prevent infinite cycles, halting automatically if success criteria are not met within the configured bounds.