What problem does it solve?
Running 10 or more machine learning experiments (such as multi-seed sweeps, grid searches, or teacher-student training chains) manually requires constant monitoring for stale screen sessions, out-of-memory errors, and wave transition timing, leading to wasted wall-clock time, lost state on scheduler crashes, and repeated engineering work for each new experiment batch.
Core Features & Use Cases
- Automated Batch Scheduling: Runs 10+ parallel jobs across multiple GPUs with automatic wave transitions, eliminating the need to manually coordinate job launches and GPU allocation.
- Resilient Failure Handling: Auto-detects CUDA out-of-memory errors to retry jobs after a configurable delay, cleans up stale screen sessions where Python processes have exited, and persists full state to disk to recover from scheduler crashes or SSH disconnections.
- Phase Dependency Support: Enforces sequential workflows like teacher model training followed by student distillation, so later jobs only launch when their required checkpoint outputs exist.
- Use Case: If you need to run 42 experiments across 2 training phases (teacher training then student distillation) with 3 random seeds per configuration, this skill automatically schedules jobs across available GPUs, retries any OOM failures, and generates a summary report when all jobs finish.
Quick Start
Use the experiment-queue skill to run your multi-seed vertebrae segmentation experiment grid with automatic wave transitions and OOM retry.
