autoresearch

Automate prompt optimization by running autonomous experimentation loops that test a SKILL.md against binary evaluation criteria, applying targeted prompt mutations one change at a time to iteratively improve outputs. You provide test prompts and a binary eval suite to guide the process.

Binary eval scoring translates success criteria into yes/no signals to guide prompt mutations. The autonomous loop evaluates mutated SKILL.md outputs against this fixed suite, preserving successful mutations and discarding failures to build better skills over time.

Start the autonomous optimization loop by providing the file path to your target SKILL.md, a set of varied test prompts for different use-case scenarios, and a defined binary eval suite to score the outputs.

The skill mutation process tracks changes by generating a results.tsv log, a changelog.md, and a live dashboard (results.json + dashboard.html) that refreshes after each experiment to monitor progress and preserve baseline tracking.

Yes, the approach applies to any Claude Code skill by running autonomous experiments against your defined binary evaluation criteria. It tests varied test prompts and use-case scenarios to close performance gaps in the target SKILL.md.

Binary eval criteria restrict scoring to yes/no signals, meaning nuanced performance gradients are lost. Prompt mutations are applied one change at a time, which ensures baseline tracking but may slow down optimization for complex skills requiring multi-variable adjustments.