What problem does it solve?
Manual code optimization via trial and error requires constant manual test execution, result tracking, and change rollback, which is slow and prone to human error. This skill automates the entire experiment loop so agents can focus solely on code modifications while all infrastructure and evaluation tasks are handled automatically.
Core Features & Use Cases
- Karpathy AutoResearch Pattern: Enforces a strict agent role limited to code changes, eliminating accidental infrastructure modifications or manual test runs
- Git-Based Experiment Tracking: Uses dedicated branches for each experiment, with automatic keep/discard/crash verdicts and full result logging to a TSV file
- Autonomous Iteration: Runs continuous experiment loops, parses structured JSON results, and iterates on code changes without user intervention until predefined goals are met
- Use Case: Optimize a speech-to-text model's character error rate, tune machine learning hyperparameters, or improve algorithm performance with minimal manual overhead
Quick Start
Use the auto skill to run a 2 hour autonomous experiment loop to optimize the character error rate of the local speech-to-text project.