experiment-queue

To run a large batch of ML experiments across remote GPUs, define a grid spec and generate a manifest.json file, then start the remote queue manager to orchestrate 10+ concurrent jobs via SSH with stateful scheduling and per-job logs.

The best way to automatically retry failed ML jobs that run out of memory is using a manifest-driven queue manager that provides OOM-aware retries, stale-screen cleanup, and persistent state tracking to resume wave-based execution without losing progress.

Yes, you can orchestrate multi-seed sweeps and dependent multi-phase workflows over SSH by applying a grid parser that expands configurations and schedules sequential waves, coordinating tasks like teacher-to-student transitions across remote machines.

Wave-based execution prevents race conditions during GPU scheduling by applying stateful coordination that tracks each job's status, ensuring wave transitions only occur when all concurrent jobs in the current batch complete or fail before launching the next set.

You do not need a specific external job scheduler to manage 10+ concurrent grid experiments; a remote launcher using SSH and screen sessions provides persistent queue state and resume capability to coordinate jobs without a dedicated cluster manager.

Remote screen sessions become stale during long experiment chains due to network drops or unhandled job failures, which is why automated stale-screen cleanup is required to prevent orphaned processes and maintain stable coordination across wave transitions.