skills.rest
hhhi21g
hhhi21gCOMMUNITY
Explore all skills by this author →
GitHub

training-llms-megatron

Trains LLMs with Megatron-Core parallelism strategies for GPU efficiency.

Updated Apr 11, 2026
Search Tags:#large language models#parallelism#efficient training#GPU utilization#Megatron-Core
One-click install
npx skills add https://github.com/hhhi21g/HealthCenter --skill training-llms-megatron-hhhi21g
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: training-llms-megatron
Source: https://github.com/hhhi21g/HealthCenter/tree/main/.codex/skills/megatron-core
Command: npx skills add https://github.com/hhhi21g/HealthCenter --skill training-llms-megatron-hhhi21g

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires megatron-core, torch, apex, transformer-engine, and includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill streamlines the training process for large language models (LLMs), enabling users to efficiently train models with parameters exceeding 1 billion using Megatron-Core's advanced parallelism strategies.

Core Features & Use Cases

  • Large-scale Training: Trains LLMs with parameters from 2B to 462B using advanced parallelism.
  • Advanced Parallelism: Utilizes tensor, pipeline, sequence, and expert parallelism for efficient GPU utilization.
  • High Efficiency: Achieves 47% Model FLOP Utilization on H100 GPUs.
  • Use Case: Ideal for users who require high efficiency and GPU utilization for training LLMs >1B parameters.

Quick Start

Use the training-llms-megatron skill to train a large language model with the following command:

python train_llama3_8b_fp8.sh
  • Ensure that Megatron-Core is installed and properly configured.
  • Prepare the training data and vocabulary files.
  • Run the script with the desired parameters, adjusting as necessary for your specific use case.

Frequently Asked Questions about training-llms-megatron

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I train large language models with advanced parallelism for high GPU utilization?▼

You can efficiently train large language models using Megatron-Core's advanced parallelism strategies, applying tensor, pipeline, sequence, and expert parallelism to maximize GPU utilization across massive parameter scales.

What is the best way to train LLMs exceeding 1 billion parameters efficiently?▼

The best way to train LLMs exceeding 1 billion parameters is using Megatron-Core's advanced parallelism, which achieves 47% Model FLOP Utilization on H100 GPUs for high efficiency training from 2B to 462B parameters.

Do I need Megatron-Core and transformer-engine to run large-scale training scripts?▼

Yes, you need Megatron-Core, torch, apex, and transformer-engine installed and properly configured to run large-scale training scripts for efficient GPU utilization.

How to start training a large language model using tensor and pipeline parallelism?▼

Prepare your training data and vocabulary files, ensure Megatron-Core is configured, then execute the provided training script like `train_llama3_8b_fp8.sh` with adjusted parameters for your specific use case.

Can I use Megatron-Core parallelism strategies for models under 1 billion parameters?▼

Megatron-Core parallelism strategies apply to scenarios where high efficiency and GPU utilization are critical, such as training models with parameters exceeding 1 billion, making it unsuitable for smaller models.

Related Skills

Explore More Software Engineering Skills→
Ced3-hanCed3-han

training-llms-megatron

Train large language models with NVIDIA Megatron-Core parallelism strategies.

Community
Advanced
tianhao909tianhao909
1

training-llms-megatron

Train large language models with NVIDIA Megatron-Core parallelism strategies.

Community
Advanced
ovachieverovachiever
52

training-llms-megatron

Train large language models with Megatron-Core using configurable 3D and 4D parallelism.

Community
Advanced
NextSpark-jsNextSpark-js

registry-system

Explain the auto-generated registry system in Next.js applications.

Official
Intermediate