training-llms-megatron

Distributed training for large language models using Megatron-Core is configured by selecting a mix of tensor, pipeline, sequence, context, and expert parallelism. This skill provides benchmark-backed recipes and production configuration examples to validate your cluster setup.

The best way to plan GPU memory and parallelism for a 70B pretraining run is to apply tested configurations for GPT and LLaMA architectures. This skill offers memory optimization guidance and throughput tuning to avoid communication bottlenecks across multi-GPU and multi-node workflows.

Yes, Megatron-Core supports mixture of experts training for models like Mixtral and DeepSeek. It provides specific expert parallel requirements and production recipes to manage high-throughput pretraining for these specialized architectures.

You can optimize model flop utilization on H100 and A100 clusters by applying practical performance tuning recommendations. This skill helps improve throughput and memory usage during dense or mixture-of-experts model training at scale.

Sequence and context parallelism are needed alongside tensor and pipeline parallelism when training very large language models that exceed single-device memory limits. This skill helps you choose the right parallelism strategy combination for your specific hardware cluster.

Large-scale LLM training runs face communication bottlenecks across multiple nodes due to suboptimal parallelism strategies and batch sizes. This skill provides benchmark-backed recipes to tune configurations and eliminate memory or communication performance issues.