experiment-queue

You can automate multi-seed ML experiment sweeps by providing a YAML manifest or grid-derived job specs to a queue system that orchestrates SSH batches, manages GPU availability thresholds, and enforces wave dependencies. This eliminates manual troubleshooting and automatically handles stale screen sessions across remote runs.

To automatically retry ML experiments failing with CUDA OOM errors, the system detects the error from per-job logs, kills the failing run, waits for GPU memory to release, and retries up to a configured maximum. This OOM-aware retry mechanism prevents batches from stalling after memory crashes.

The best way to orchestrate teacher student distillation chains is using a job state machine that enforces phase dependencies. This ensures downstream student jobs wait until upstream teacher phases complete, preventing wave-phase race conditions in multi-phase experiment workflows.

To clean up stale SSH screen sessions when a python process exits but the screen persists, the system detects the mismatch, verifies expected outputs, and cleanly transitions the job state. This persistent queue state management keeps the experiment grid running smoothly without manual intervention.

Yes, you need a YAML manifest or grid-derived job specs to run experiment grids with automatic OOM retry. The queue requires these specs to bootstrap remote helpers, gate jobs by GPU-free thresholds, and execute wave-based orchestration end-to-end.