experiment-queue

Orchestrate SSH-based ML experiment batches with stateful queueing and OOM-aware retries.

Updated Apr 29, 2026
One-click install
npx skills add https://github.com/jkfee/Auto-Research --skill experiment-queue
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: experiment-queue
Source: https://github.com/jkfee/Auto-Research/tree/main/skills/experiment-queue
Command: npx skills add https://github.com/jkfee/Auto-Research --skill experiment-queue

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

SSH-based orchestration of large batches of ML experiments enables scalable, reproducible runs on remote GPU clusters by coordinating jobs, tracking state, and handling failures automatically.

Core Features & Use Cases

  • Wave-based scheduling to maximize GPU utilization without memory contention.
  • OOM-aware retry to recover from CUDA memory errors while preserving experiment progress.
  • Stale-screen cleanup to detect and recover from detached sessions and ensure progress.
  • Manifest/grid-driven orchestration to generate and run large grids of experiments from concise specifications.

Quick Start

Describe your experiment grid and run /experiment-queue to launch a scheduled, wave-based batch across multiple GPUs.

Frequently Asked Questions about experiment-queue

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I orchestrate large ML experiment grids across remote GPUs via SSH?

You can orchestrate large ML experiment grids by running scheduled, wave-based batches across multiple remote GPUs. The system coordinates SSH-based jobs, tracks persistent queue state, and handles failures automatically.

How do I recover ML experiments from CUDA out of memory errors?

To recover ML experiments from CUDA out of memory errors, the system uses OOM-aware retries. It automatically recovers from memory errors during wave transitions while preserving prior experiment progress.

What is wave-based scheduling for machine learning jobs?

Wave-based scheduling for machine learning jobs is a method to maximize remote GPU utilization without memory contention. It processes batches of experiments in sequential waves to ensure stable resource allocation.

How do I run multi-seed and multi-config grid searches on a remote cluster?

You run multi-seed and multi-config grid searches by parsing concise manifest specifications. The orchestrator generates and executes large experiment grids from these manifests with persistent state tracking.

Can I restart interrupted SSH batch jobs without losing experiment progress?

Yes, you can restart interrupted SSH batch jobs without losing progress. The orchestration system enforces persistent queue state and robust restartability, allowing recovery from detached or stale screen sessions.

When should I use a dedicated experiment queue instead of a single run command?

You should use a dedicated experiment queue instead of a single run command when handling large grid expansions, teacher-student phase workflows, or multi-seed configurations that exceed standard execution capacity.