auto-research

To automate metric optimization, you need to define a specific target metric, identify modifiable code sections, provide a runnable experiment command, and ensure available compute resources like local or remote GPUs are accessible for parallel execution.

It requires a defined metric, modifiable code sections, a runnable experiment command, and available compute resources to systematically test changes, retain improvements, and discard failures for continuous optimization.

It supports any scenario where you have a defined metric, modifiable code sections, a runnable experiment command, and available compute resources, making it suitable for compiler optimization and algorithm tuning.

The Skill supports both local and remote GPUs, allowing you to leverage available compute resources to run parallel experiments for optimizing your defined metric efficiently.

The Skill automates this process by exploring different parameters like learning rates and batch sizes, keeping successful changes that improve the metric, and reverting failures, allowing you to wake up to an improved model.

It is limited by the quality of the defined metric and the availability of compute resources, as the process relies on systematically running experiments and evaluating metric improvements to retain or discard code changes.