training-llms-megatron

To scale LLM training across multi-node GPU clusters, this Skill uses Megatron-Core to distribute workloads with tensor, pipeline, context, and expert parallelism. It supports training models from 2B to 462B parameters on NVIDIA hardware with NVLink and InfiniBand.

Training Mixture-of-Experts models at scale requires expert parallelism alongside tensor, pipeline, and context parallelism. This Skill implements these advanced parallelism strategies via Megatron-Core to efficiently train MoE architectures and improve Model Flops Utilization.

Yes, you need NVIDIA GPUs with NVLink and InfiniBand to achieve optimal performance when training 2B to 462B parameter models. This Skill requires this specific hardware configuration to execute efficient multi-node distributed training workflows.

The best way to train LLaMA-style models with advanced model parallelism is using Megatron-Core. This Skill provides production-grade workflows that combine tensor, pipeline, and context parallelism to accelerate large-scale distributed training.

No, you cannot use this Skill for distributed training without PyTorch. Megatron-Core requires PyTorch to run distributed training workflows across multi-node GPU clusters for large language models.