multi-node-slurm

To convert single-node PyTorch distributed launches into multi-node Slurm sbatch jobs, you can generate a two-phase script that warms the uv cache on one process and then launches an srun-based distributed run with MASTER_ADDR, MASTER_PORT, and ranks auto-derived from SLURM variables.

To fix NCCL timeout errors during multi-node distributed training on Slurm, diagnose rank and environment issues, mitigate NCCL rendezvous and barrier failures, resolve port collisions, and ensure correct WORLD_SIZE setup derived from SLURM variables.

The best way to run containerized Megatron Bridge training across multiple Slurm nodes is using an srun-native or legacy torch.distributed approach with a valid multi-node execution template, ensuring shared filesystem caches for all containerized artifacts.

Yes, you need a shared filesystem to run multi-node Slurm jobs with enroot containers so that NEMO_HOME, HF caches, UV caches, and warmup steps are consistently visible to all nodes, preventing container cache inconsistencies.

To debug multi-node Slurm failures and MoE OOM errors interactively, use an salloc and srun workflow to iteratively reproduce and fix multi-node issues across the allocated GPUs and nodes.

Choosing between srun-native and torch.distributed.run for multi-node PyTorch training on Slurm depends on your workflow needs; srun-native integrates directly with Slurm task configuration, while the legacy approach wraps torch.distributed.run commands.