training-llms-megatron

To train large language models with Megatron-Core, you apply tensor, pipeline, sequence, context, and expert parallelism across multi-GPU clusters. This Skill provides reference configurations to optimize end-to-end workflows for models ranging from 2B to 462B parameters.

Optimizing MFU on H100 clusters for LLM training requires combining tensor, pipeline, sequence, context, and expert parallelism. The Skill provides production-ready workflows and reference recipes to maximize hardware utilization for GPT, LLaMA, and Mixtral architectures.

Yes, you can use this Megatron-Core training workflow for DeepSeek and Nemotron deployments. The Skill includes production-ready guidance and end-to-end training recipes specifically tailored for these architectures alongside LLaMA configurations.

Yes, you need transformer-engine and apex along with megatron-core and torch to run multi-billion-parameter model training. These dependencies are required to execute the advanced parallelism configurations and achieve optimal MFU on H100 hardware.

The best way to configure expert parallelism for MoE models is by using the provided Mixtral reference recipes. These configurations leverage Megatron-Core to distribute expert layers across multi-GPU clusters, maximizing training efficiency for large-scale architectures.

Standard data parallelism fails to scale efficiently for 400B parameter models due to severe memory bottlenecks. Combining data, tensor, pipeline, context, and expert parallelism using Megatron-Core resolves these constraints, enabling successful 462B parameter training.