nemo-mbridge-multi-node-slurm

To scale single-node scripts to a multi-node Slurm sbatch workflow, you convert them using a two-phase launch pattern that populates a shared cache before full distributed execution. This automatically derives distributed variables from the Slurm environment to minimize manual configuration.

NCCL timeouts across nodes during multi-node distributed training often occur due to misconfigured network rendezvous or missing shared filesystem mounts. Enforcing shared storage for logs and caches, alongside proper SLURM rendezvous variables, resolves these cross-node communication failures.

Yes, a shared filesystem is required for multi-node Slurm distributed training. You must mount shared storage for repos, data, logs, HF_HOME, UV_CACHE_DIR, and NEMO_HOME to ensure consistency and prevent synchronization failures across all nodes.

The two-phase srun pattern is a launch workflow that populates a shared cache before full multi-node execution. It is the preferred launch method for scalable Megatron Bridge scripts, ensuring reliable distributed execution and easier debugging of cross-node failures.

Yes, you can use Enroot containers with multi-node Slurm jobs. The conversion process includes Enroot container support, enforcing necessary container mounts for repo, data, and logs to ensure consistent execution across the GPU cluster.

The best way to configure RANK and WORLD_SIZE for a distributed training sbatch job is to derive them directly from the SLURM environment. This approach automatically extracts distributed variables like LOCAL_RANK, MASTER_ADDR, and MASTER_PORT, minimizing manual configuration errors.