skills.rest
Orchestra-Research
Orchestra-ResearchOFFICIAL
Explore all skills by this author →
GitHub

training-llms-megatron

Trains large-scale language models using NVIDIA Megatron-Core with advanced parallelism strategies.

11.5k|842|Updated Nov 3, 2025
Search Tags:#distributed training#parallelism#megatron-core#llm training#h100#gpu efficiency
One-click install
npx skills add https://github.com/Orchestra-Research/AI-Research-SKILLs --skill training-llms-megatron-orchestra-research
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: training-llms-megatron
Source: https://github.com/Orchestra-Research/AI-Research-SKILLs/tree/main/08-distributed-training/megatron-core
Command: npx skills add https://github.com/Orchestra-Research/AI-Research-SKILLs --skill training-llms-megatron-orchestra-research

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires megatron-core, torch, apex, transformer-engine, and includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill addresses the challenge of training extremely large language models (LLMs) efficiently by leveraging advanced parallelism techniques to maximize GPU utilization and minimize training time.

Core Features & Use Cases

  • Large-Scale LLM Training: Train models from 2B to 462B+ parameters using NVIDIA Megatron-Core.
  • Advanced Parallelism: Implements Tensor, Pipeline, Sequence, Context, and Expert Parallelism for optimal performance.
  • High GPU Efficiency: Achieves up to 47% Model FLOP Utilization (MFU) on H100 GPUs.
  • Production-Ready: Framework used for training models like Nemotron, LLaMA, and DeepSeek.
  • Use Case: Train a 70B parameter LLaMA model on a cluster of 64 H100 GPUs using a combination of Tensor and Pipeline Parallelism for maximum throughput and memory efficiency.

Quick Start

Launch a simple distributed training job for a Megatron-Core model using the provided example script.

Frequently Asked Questions about training-llms-megatron

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I train large language models using advanced parallelism on H100 GPUs?▼

To train large language models using advanced parallelism on H100 GPUs, you can use this Megatron-Core Skill to implement Tensor, Pipeline, Sequence, Context, and Expert Parallelism. It achieves up to 47% MFU.

What is the best way to scale distributed training for a 70B parameter LLaMA model?▼

The best way to scale distributed training for a 70B parameter LLaMA model is using Megatron-Core with a combination of Tensor and Pipeline Parallelism. This maximizes throughput and memory efficiency across GPU clusters.

Can I use Megatron-Core to train models up to 462B parameters in production?▼

Yes, you can use Megatron-Core to train models from 2B to 462B+ parameters in production. It is a production-ready framework previously used to train Nemotron, LLaMA, and DeepSeek models.

Which parallelism strategies should I use to maximize GPU efficiency for LLM training?▼

To maximize GPU efficiency for LLM training, you should use Tensor, Pipeline, Sequence, Context, and Expert Parallelism. These strategies optimize hardware utilization and minimize training time on NVIDIA hardware.

Do I need Apex and Transformer-Engine to run distributed LLM training jobs?▼

Yes, you need Apex and Transformer-Engine dependencies to run distributed LLM training jobs. The Skill also requires PyTorch and Megatron-Core to execute advanced parallelism scripts on NVIDIA hardware.

Related Skills

Explore More Software Engineering Skills→
ovachieverovachiever
52

training-llms-megatron

Train large language models with Megatron-Core using configurable 3D and 4D parallelism.

Community
Advanced
Lord1EgyptLord1Egypt
2

transformers

Load, run, and fine-tune pre-trained transformer models for NLP, vision, and audio tasks.

Community
Advanced
devMoezdevMoez
1

nemo-curator

Filter, deduplicate, and sanitize LLM training datasets with GPU acceleration and privacy controls.

Community
Advanced
zhuangbiaoweizhuangbiaowei
2

training-llms-megatron

Train large-scale language models using NVIDIA Megatron-Core with advanced parallelism.

Community
Advanced