experiment-queue

Batch scheduling for machine learning experiments on remote GPUs is automated by queuing jobs across multiple devices, tracking state, and applying retry logic for memory overflow failures. You provide a grid specification and the orchestration handles execution.

Yes, failed ML jobs from GPU memory overflows are retried automatically. The retry logic detects memory overflow errors and reschedules the affected jobs, preventing stuck states and ensuring your multi-seed grid experiments complete without manual intervention.

Multi-seed and multi-config experiments are run by submitting a manifest or grid specification to the queue. The orchestrator schedules jobs across available remote GPUs, batches them efficiently, and tracks the status of each configuration to completion.

Yes, orchestrating SSH jobs for ML experiments requires Bash, Read, Grep, Glob, Edit, Write, and Agent capabilities, plus access to run-experiment and monitor-experiment skills. These dependencies handle remote execution, file management, and state tracking.

State tracking for large batches of ML experiments is managed by monitoring job status across all remote GPUs, ensuring no jobs are left running or stuck. This is ideal for scenarios with over 10 jobs requiring orchestration and retry logic.

An experiment queue should be used when you have over 10 machine learning jobs requiring orchestration, such as multi-seed grid experiments. It eliminates manual management of batch scheduling, retry logic for memory issues, and state tracking on remote SSH GPUs.