mcore-run-on-slurm

Orchestrate distributed Megatron-LM training on SLURM clusters with sbatch skeletons.

Updated Apr 16, 2026

Search Tags:#gpu #cluster #distributed-training #slurm #megatron-lm #sbatch

Ready to run

npx skills add https://github.com/sayalinvidia/sayali-skills-test --skill mcore-run-on-slurm

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: mcore-run-on-slurm
Source: https://github.com/sayalinvidia/sayali-skills-test/tree/main/skills/mcore-run-on-slurm
Command: npx skills add https://github.com/sayalinvidia/sayali-skills-test --skill mcore-run-on-slurm

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Launch and manage distributed Megatron-LM training on SLURM clusters, providing a repeatable pattern that reduces setup time and configuration errors.

Core Features & Use Cases

Reusable sbatch skeleton for multi-node GPU training with Megatron-LM.
Environment and launcher guidance for torch.distributed.run, including MASTER_ADDR/MASTER_PORT and WORLD_SIZE calculations.
CUDA_DEVICE_MAX_CONNECTIONS recommendations across hardware generations and parallelism modes.
Container deployment considerations and monitoring guidance.
Use Case: Validate a new SLURM cluster by quickly spinning up a Megatron-LM job to test scaling and fault diagnosis.

Quick Start

Create and submit a SLURM sbatch script using the provided skeleton to start a multi-node Megatron-LM training job.

Frequently Asked Questions about mcore-run-on-slurm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I launch multi-node Megatron-LM training on a SLURM cluster?▼

You can launch multi-node Megatron-LM training on a SLURM cluster by creating and submitting an sbatch script built from a reusable skeleton that configures torch.distributed.run across allocated GPUs.

How do I configure MASTER_ADDR and WORLD_SIZE for distributed training in SLURM?▼

Configuring MASTER_ADDR, MASTER_PORT, and WORLD_SIZE for SLURM distributed training involves calculating these values dynamically from SLURM environment variables to establish the torch.distributed.run rendezvous endpoints.

What should CUDA_DEVICE_MAX_CONNECTIONS be set to for Megatron-LM?▼

CUDA_DEVICE_MAX_CONNECTIONS settings for Megatron-LM depend on your specific hardware generation and parallelism modes, requiring tailored recommendations to optimize multi-node GPU communication throughput.

Does this SLURM sbatch skeleton support container deployment and monitoring for GPU jobs?▼

Yes, the SLURM sbatch skeleton supports container integration and provides monitoring guidance to track distributed GPU training jobs, alongside per-rank failure diagnosis workflows for troubleshooting.

Why use a reusable sbatch skeleton for Megatron-LM instead of a custom SLURM script?▼

Using a reusable sbatch skeleton for Megatron-LM provides a repeatable pattern that reduces setup time and configuration errors when orchestrating multi-node GPU training on SLURM clusters.

Can I validate a new SLURM cluster by running a Megatron-LM job?▼

Yes, you can validate a new SLURM cluster by quickly spinning up a Megatron-LM job to test scaling, verify environment setup, and perform fault diagnosis across multi-node GPU hardware.