experiment-queue

Running multi-seed ML experiments on remote GPU servers without crashing requires OOM-aware retry logic that waits for memory release or switches GPUs. This orchestration skill automates large job batches over SSH, retrying failed tasks and cleaning up stale screens to prevent memory exhaustion.

The best way to orchestrate wave transitions for teacher and student chains is using a queue manager that enforces sequential wave execution. This skill schedules wave 1, waits, then transitions to wave 2 without race conditions, automating the distillation pipeline across mixed seed grids.

Yes, you can batch ten or more ML jobs across multiple GPUs using SSH. This skill handles batch experiment orchestration for large job counts, distributing tasks across available remote GPUs and re-running failed cells in mixed seed grids.

Yes, you need specific scripts to automate large ML experiment sweeps on remote servers. This skill requires queue_manager.py for scheduling and build_manifest.py for manifest generation, which together coordinate multi-seed sweeps and prevent wave-transition races.

SSH screen sessions become stale during long GPU experiment runs due to unhandled process terminations or memory bottlenecks. This skill prevents stale screens by incorporating cleanup mechanisms and OOM-aware retry logic to maintain active session integrity.

To re-run failed cells in a mixed seed grid, you need an orchestration tool that tracks job success states and automatically retries failures. This skill identifies failed cells within 21 seeds x 12 cells grids and re-executes them while preventing wave-transition races.