experiment-queue

To run a large multi-seed ML grid search on remote SSH GPUs without losing progress to OOM crashes, use an orchestration skill that automatically detects CUDA OOM, kills the stalled screen job, waits, and requeues the experiment with a bounded retry limit. It persists state in a queue_state.json file, so you resume from the exact failure point instead of restarting the entire batch.

To schedule a teacher to student ML pipeline so that student distillation only starts after all teacher checkpoints exist, use wave-based scheduling with dependency enforcement. This approach checks precondition outputs from the teacher phase and only launches the student phase jobs once all required prior-phase checkpoints are verified as ready.

An SSH GPU screen job shows as running when the Python process already exited from an out of memory error because the screen terminal persists after the active process dies. Stale screen detection resolves this by identifying orphaned screens, checking for expected output files, and automatically cleaning up the dead session.

Yes, you can batch 10+ ML experiments across multiple GPUs using a single manifest. A manifest-driven workflow schedules all jobs across available remote GPUs with multi-GPU parallelism, handling queue bootstrapping, state persistence, and automatic wave transitions to eliminate wasted wall-clock time during large runs.

The best way to recover an ML experiment queue after a server disconnection is to rely on persisted state. A crash-resume mechanism reads the existing queue_state.json file and continues the batch orchestration from the last recorded point without regenerating run identifiers or duplicating completed jobs.

No, you do not need to manually restart failed ML jobs when running a large parameter sweep. The orchestration skill caps retry attempts and automatically handles OOM recovery, stale screen cleanup, and phase ordering mistakes, ensuring the grid expansion completes without continuous manual intervention.