What problem does it solve? Running large batches of ML experiments (multi-seed sweeps, grid searches, teacher-student chains) on remote GPU servers involves tedious manual orchestration: stale screen sessions block new waves, CUDA OOM failures require manual retries, and dependent jobs launch before their prerequisites finish. This Skill automates that entire scheduling layer. ## Core Features & Use Cases - Batch Job Scheduling: Expands declarative grid specs (e.g., N × seeds × dataset sizes) into explicit job manifests and distributes jobs across free GPUs via screen sessions. - OOM-Aware Retry & Stale-Screen Cleanup: Detects CUDA out-of-memory failures from per-job logs, requeues jobs after a configurable delay, and kills screens whose Python process has exited. - Phase Dependencies & Wave Transitions: Enforces depends_on ordering (e.g., train teachers before distilling students) and only launches the next wave after GPU memory settles below threshold. - Crash-Safe State: Writes queue_state.json atomically every poll cycle, so the scheduler can be killed and restarted without losing progress. - Use Case: You need to run 42 distillation experiments (2 model sizes × 4 dataset sizes × 3 seeds, plus teacher pretraining). Describe the grid, and the Skill deploys a detached scheduler on your SSH server that runs all jobs over ~6 hours and produces a summary report. ## Quick Start Ask the agent to queue your multi-seed experiment grid on your SSH GPU server, for example: "Run N in {64,128,256} × seeds {42,200,201} distillation experiments on gpu-server using the experiment queue."
