autoresearch

Automate ML experimentation loops by editing train.py and logging results.

40|9|Updated Jan 4, 2026

Search Tags:#automation #closed-loop #overnight #ml-experiments #autoresearch #git-ratcheting #results.tsv

Ready to run

npx skills add https://github.com/akillness/oh-my-skills --skill autoresearch-akillness

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: autoresearch
Source: https://github.com/akillness/oh-my-skills/tree/main/.agent-skills/autoresearch
Command: npx skills add https://github.com/akillness/oh-my-skills --skill autoresearch-akillness

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve?

Autonomously coordinates a closed-loop ML experimentation workflow by editing train.py, running fixed-time experiments, and recording results to produce monotonically improving models.

Core Features & Use Cases

Closed-loop automation of ML experiments: agent edits code, runs 300-second experiments, logs val_bpb and memory usage, and performs git-based ratcheting on improvements.
Overnight experimentation and baseline evaluation: enables continuous exploration of architecture/hyperparameter options on single-GPU setups.
Reproducible research with program.md and results.tsv: maintains a tamper-evident history of experiments and decisions.

Quick Start

Install uv, clone this repository, then run the setup and run-loop scripts to start automated autonomous research experiments.

Frequently Asked Questions about autoresearch

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I automate autonomous ML experiments to run overnight on a single GPU?▼

Automate autonomous ML experiments by using a closed-loop workflow that edits train.py, runs 300-second experiments, and logs results. This enables continuous overnight hyperparameter exploration on single-GPU Linux setups.

What is the best way to ensure reproducible ML research and prevent overwriting improved models?▼

Ensure reproducible ML research by using git ratcheting to lock in model improvements and a results.tsv log to record val_bpb and memory usage. A program.md directive and locked dependencies maintain a tamper-evident history.

Can I run architecture ablations and baseline evaluation without manually editing code each time?▼

Yes, you can run architecture ablations and baseline evaluation without manual edits. The agent autonomously edits train.py, executes fixed-time experiments, and performs git-based ratcheting to monotonically improve models.

Do I need to lock dependencies in pyproject.toml before starting closed-loop ML experimentation?▼

Yes, you need to lock dependencies in pyproject.toml before starting closed-loop ML experimentation. The immutable evaluation harness and reproducible setup rely exclusively on these locked dependencies.

What are the limitations of running closed-loop ML experimentation with a 300-second budget?▼

The 300-second budget limits each experiment iteration to short architecture and hyperparameter tests. This fixed-time constraint is designed for single-GPU Linux setups and may not suit long-running training regimes.

Why does my autonomous ML experiment loop require an immutable evaluation harness?▼

An immutable evaluation harness is required to ensure fair, reproducible comparisons across all iterations. It prevents evaluation logic changes from confounding the val_bpb results recorded in your results.tsv log.