run-on-slurm

To launch Megatron-LM multi-node training on a SLURM cluster, use an sbatch skeleton that runs torch.distributed.run with correct MASTER_ADDR, MASTER_PORT, WORLD_SIZE, and node-rank settings for proper rank rendezvous.

Configure CUDA_DEVICE_MAX_CONNECTIONS based on GPU hardware and parallelism mode, setting it to 1 for certain cases, avoiding it in others, or using 32 for MoE communication overlap to prevent known failures.

SLURM multi-node training jobs fail with NCCL errors due to device connectivity issues or misconfigured environment variables; diagnose by inspecting rank-by-rank stderr to classify NCCL failures alongside OOM and import errors.

Yes, torch.distributed.run works with SLURM by configuring nnodes, nproc-per-node, node-rank, master-addr, and master-port correctly so ranks rendezvous across multiple nodes using shared filesystems.

The best way to debug rank failures in SLURM distributed training is inspecting rank-by-rank stderr to classify OOM, shape divisibility errors, import errors, and NCCL connectivity issues.

Yes, you need a shared filesystem for SLURM multi-node training configurations to ensure container or venv visibility and access to the shared Megatron worktree across all allocated nodes.