skills.rest
Tgoldi
TgoldiCOMMUNITY
Explore all skills by this author →
GitHub

training-llms-megatron

Configures distributed LLM training with Megatron-Core using tensor, pipeline, and expert parallelism.

1|Updated May 10, 2026
Search Tags:#distributed training#megatron-core#fp8#tensor parallelism#llm training#h100#mixture of experts
One-click install
npx skills add https://github.com/Tgoldi/claude-skills --skill training-llms-megatron-tgoldi
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: training-llms-megatron
Source: https://github.com/Tgoldi/claude-skills/tree/main/training-llms-megatron
Command: npx skills add https://github.com/Tgoldi/claude-skills --skill training-llms-megatron-tgoldi

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires megatron-core, torch, apex, transformer-engine, and includes references (resource) components.

What problem does it solve? Training large language models from 2B to 462B parameters requires coordinating complex parallelism strategies across hundreds of GPUs, and misconfiguration leads to out-of-memory errors, low GPU utilization, or diverging loss. ## Core Features & Use Cases - Parallelism Configuration: Provides tested tensor, pipeline, context, sequence, and expert parallelism settings matched to model size and GPU topology. - Production Training Recipes: Includes complete hyperparameter configurations for LLaMA 3 (8B-405B), Mixtral MoE, and DeepSeek-V3 architectures. - Performance Optimization: Covers FP8 training on H100, Flash Attention, and micro-batch tuning to reach 40-47% MFU. - Use Case: A team training a 70B LLaMA-style model on 64 H100 GPUs uses the provided TP=4/PP=4 configuration and launch scripts to start distributed training without weeks of trial and error. ## Quick Start Ask the assistant to generate a Megatron-Core training configuration for a 70B LLaMA model on 64 H100 GPUs with tensor and pipeline parallelism.

Frequently Asked Questions about training-llms-megatron

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I train a 70B LLaMA model on multiple GPUs?▼

Use Megatron-Core with tensor parallelism 4 and pipeline parallelism 4 across 64 GPUs, with micro-batch size 1 and global batch size 1024. Launch with torchrun across 8 nodes of 8 H100 GPUs each, enabling sequence parallelism and Transformer Engine.

What parallelism strategy should I use for large model training?▼

Match strategy to model size: data parallelism alone under 1B, tensor parallelism 2-4 for 1-10B, TP plus pipeline parallelism for 10-175B, and 3D or 4D parallelism with context parallelism above 175B. Keep tensor parallelism within a single NVLink-connected node.

Megatron-Core vs DeepSpeed vs PyTorch FSDP for LLM training?▼

Megatron-Core targets models above 10B parameters needing maximum efficiency and fine-grained parallelism control on NVIDIA GPUs. DeepSpeed offers easier setup under 100B, while PyTorch FSDP suits models under 70B with a simpler native PyTorch API.

Does Megatron-Core support Mixture of Experts training?▼

Yes, Megatron-Core supports MoE through expert parallelism, which distributes experts across GPUs to reduce memory by up to 75%. Configure num-experts, moe-router-topk, and expert-model-parallel-size for models like Mixtral 8x7B.

Why is my GPU utilization below 30% MFU during training?▼

Low MFU usually comes from a micro-batch size that is too small, excessive parallelism overhead, or missing optimizations. Increase micro-batch size, enable Flash Attention and sequence parallelism, and reduce tensor parallelism if it exceeds 8 or crosses nodes.

How do I fix out-of-memory errors during Megatron training?▼

Increase tensor or pipeline parallelism to split the model across more GPUs, enable gradient checkpointing with recompute-granularity full, or reduce micro-batch size. You can also offload the optimizer to CPU using the cpu-optimizer flag.

Related Skills

Explore More Software Engineering Skills→
lucifertrjlucifertrj
17

nltk-nlp-text-cleaning

Clean, tokenize, and normalize English text with NLTK for NLP pipelines.

Community
Advanced
RachasumanthRachasumanth

inference-deploy

Deploy trained machine learning models for inference using vLLM, TGI, Ollama, and llama.cpp.

Community
Advanced
JKhyroJKhyro

lambda-labs-gpu-cloud

Provision and manage Lambda Labs GPU instances for ML training.

Community
Advanced
kngender5kngender5

colab-distributed

Distribute machine learning training across GPUs in Google Colab using DDP, DeepSpeed ZeRO, and FSDP.

Community
Advanced