autoresearch-loop

Automating LLM optimization experiments involves generating hypotheses and concrete patches, evaluating them against target eval questions, and retaining top-performing improvements. This Skill iterates through that loop and stores all experiment rounds in a JSONL state file.

Tracking experiment state for iterative LLM optimization requires storing each round's hypotheses, patches, and evaluation scores. This Skill uses a JSONL state file to maintain complete experiment history, supports dry-runs, and generates summaries of the optimization process.

Running automated LLM evaluation loops requires the Bun runtime, the @anthropic-ai/sdk package, and a valid ANTHROPIC_API_KEY. The Skill uses these dependencies to generate patches and evaluate improvements against your targets.

Iterative optimization can be applied to targets stored in a JSONL file by passing the file path to the script. It processes targets by generating improvements, evaluating them against predefined questions, and keeping only those that beat the best score.

The best way to generate and evaluate patches is through an iterative loop that proposes changes via an LLM and scores them. This Skill automates this by generating concrete patches, evaluating them against target criteria, and saving only improvements that outperform previous best scores.