Launch distributed Megatron-LM training jobs on SLURM with torch.distributed.run.
npx skills add https://github.com/NVIDIA/Megatron-LM --skill run-on-slurm-nvidia
Please help me install this Agent Skill. Skill: run-on-slurm Source: https://github.com/NVIDIA/Megatron-LM/tree/main/skills/run-on-slurm Command: npx skills add https://github.com/NVIDIA/Megatron-LM --skill run-on-slurm-nvidia
It solves the problem of reliably running multi-node, multi-GPU Megatron-LM (via torch.distributed.run) on SLURM without misconfigured environment variables, inconsistent filesystem paths, or hardware/parallelism-related runtime failures.
Write an sbatch script that exports MASTER_ADDR and MASTER_PORT, computes WORLD_SIZE from SLURM_NNODES and GPUs per node, and starts pretrain_gpt.py using uv run python -m torch.distributed.run with matching master address/port and node-rank across all ranks.
High-intent search queries and answers about installing and using this skill.
To launch Megatron-LM distributed training on SLURM, write an sbatch script that exports MASTER_ADDR and MASTER_PORT, computes WORLD_SIZE from SLURM_NNODES and GPUs per node, and starts pretrain_gpt.py using torch.distributed.run with matching master address and node-rank across all ranks.
Correct torch.distributed.run parameters for multi-node rendezvous require matching nnodes, nproc-per-node, node-rank, master-addr, and master-port across all ranks to ensure successful distributed initialization.
NCCL failures during multi-node SLURM training often stem from misconfigured environment variables or inconsistent shared filesystem paths; systematic per-rank failure diagnosis helps identify specific OOM, shape, or import errors causing the issue.
CUDA_DEVICE_MAX_CONNECTIONS handling depends on hardware and parallelism mode, requiring conditional settings to prevent assertion failures and avoid incorrect configurations for FSDP versus non-FSDP modes and specific Blackwell behavior.
Yes, a minimal SLURM sbatch skeleton can launch a unified srun and torch.distributed.run across nodes by setting MASTER_ADDR, MASTER_PORT, NNODES, and WORLD_SIZE correctly within the script.
Multi-node SLURM training jobs require consistent shared filesystem paths across all nodes to prevent runtime failures and ensure correct distributed rendezvous configuration.
yo-stevenConvert single-node PyTorch distributed launches into multi-node Slurm sbatch jobs.
yo-stevenLaunch distributed Megatron-LM training on SLURM with torch.distributed.run.
sayalinvidiaOrchestrate distributed Megatron-LM training on SLURM clusters with sbatch skeletons.
sayalinvidiaConvert single-node Megatron Bridge scripts into multi-node Slurm sbatch workflows.
