mcore-run-on-slurm

What problem does it solve?

Launch and manage distributed Megatron-LM training on SLURM clusters, providing a repeatable pattern that reduces setup time and configuration errors.

Core Features & Use Cases

• Reusable sbatch skeleton for multi-node GPU training with Megatron-LM.
• Environment and launcher guidance for torch.distributed.run, including MASTER_ADDR/MASTER_PORT and WORLD_SIZE calculations.
• CUDA_DEVICE_MAX_CONNECTIONS recommendations across hardware generations and parallelism modes.
• Container deployment considerations and monitoring guidance.
• Use Case: Validate a new SLURM cluster by quickly spinning up a Megatron-LM job to test scaling and fault diagnosis.

Quick Start

Create and submit a SLURM sbatch script using the provided skeleton to start a multi-node Megatron-LM training job.