What problem does it solve? Running large batches of ML experiments (multi-seed sweeps, grid searches, teacher-student chains) on remote GPU servers is plagued by stale screen sessions, CUDA OOM failures, wave-transition races, and lost state when connections drop. This Skill deploys a detached scheduler that handles all of that orchestration automatically. ## Core Features & Use Cases - Declarative job manifests: Define jobs via YAML manifest, grid spec (Cartesian product expansion), or natural language, with per-phase dependencies (e.g., distill students only after teachers finish). - Automatic failure handling: Detects CUDA OOM from logs and retries with backoff, kills stale screens whose Python process exited, and parks unrecoverable jobs as stuck. - Persistent state and resume: Writes queue_state.json continuously, so a crashed scheduler can be relaunched idempotently without losing progress. - Use Case: You need to run 42 jobs (2 teacher trainings, then 24 distillation runs across 3 seeds and 4 dataset sizes, plus 16 validation runs) across 8 GPUs overnight. The scheduler assigns jobs to free GPUs, retries OOMs, chains the phases, and produces a summary report. ## Quick Start Ask the AI to queue and run your multi-seed experiment grid on your SSH GPU server, for example: "Run N in {64,128,256} x 4 dataset sizes x 3 seeds on gpu-server, with distillation depending on teacher checkpoints."
