skills.rest
NVIDIA
NVIDIAOFFICIAL
Explore all skills by this author →
WebsiteGitHub

mcore-migrate-gpt-to-hybrid

Migrates Megatron Core GPTModel checkpoints and training scripts to HybridModel.

17.7k|4.4k|Updated Mar 21, 2019
Search Tags:#Bash#Megatron#checkpoint conversion#Migration#Distributed training#hybridmodel#gptmodel
One-click install
npx skills add https://github.com/NVIDIA/Megatron-LM --skill mcore-migrate-gpt-to-hybrid
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: mcore-migrate-gpt-to-hybrid
Source: https://github.com/NVIDIA/Megatron-LM/tree/main/skills/mcore-migrate-gpt-to-hybrid
Command: npx skills add https://github.com/NVIDIA/Megatron-LM --skill mcore-migrate-gpt-to-hybrid

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Moving an existing Megatron Core GPTModel training setup to HybridModel involves checkpoint conversion, layer-pattern generation, and launch-script rewrites where silent failures (stale --num-layers values, glob-expanded patterns, rejected pipeline arguments) are common. This Skill provides the mechanical procedure and verification checks to perform that migration without silent breakage.

Core Features & Use Cases

  • Launch Script Transfer: Step-by-step edits to convert a pretrain_gpt.py script or sbatch launcher into a pretrain_hybrid.py invocation, including entrypoint, stack spec, and argument cleanup.
  • Hybrid Layer Pattern Generation: Shell snippets to programmatically build --hybrid-layer-pattern strings (dense '*-' or MoE '*E' blocks, with pipeline segments) instead of error-prone hand-typing.
  • Bash Quoting Hazard Guidance: Explains how unquoted array expansion glob-corrupts layer patterns and how to quote expansions correctly.
  • Verification Checks: grep and pattern-shape commands that confirm no stale arguments survived and that attention/MLP counts match the source GPT layer count.
  • Use Case: You have a 32-layer GPTModel trained via pretrain_gpt.py and want to rerun it under pretrain_hybrid.py with an equivalent '*-' pattern, verifying identical parameter counts and throughput.

Quick Start

Ask the assistant to migrate your existing pretrain_gpt.py launch script to pretrain_hybrid.py with an equivalent hybrid layer pattern and verify the rewrite.

Frequently Asked Questions about mcore-migrate-gpt-to-hybrid

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I migrate a Megatron GPTModel training script to HybridModel?▼

Replace the pretrain_gpt.py entrypoint with pretrain_hybrid.py, generate a --hybrid-layer-pattern matching your layer count, delete --num-layers and rejected pipeline-layout arguments, add the hybrid stack spec, and repoint --save to a new directory. Then run the provided grep and pattern-shape checks to verify the rewrite.

How do I generate a hybrid layer pattern for a 96-layer model?▼

Generate it programmatically in bash instead of typing it: use printf in a loop over the layer count to repeat '*-' for dense layers or '*E' for every-layer MoE. For pipeline segments, repeat a per-segment block joined by '|', ensuring the segment count divides the layer count.

Why does my hybrid layer pattern get corrupted in a bash training script?▼

Unquoted array expansion like ${MODEL_ARGS[@]} re-runs word-splitting and pathname expansion, so a pattern like '*-*-' gets globbed against files in the launch directory. Store the pattern in a variable and expand arrays quoted as "${MODEL_ARGS[@]}" to prevent corruption.

What happens if I leave --num-layers in a HybridModel training command?▼

A stale --num-layers only produces a warning while being silently overridden by the layer count derived from --hybrid-layer-pattern. The run looks healthy but may not match your intent, so the migration procedure deletes --num-layers entirely.

How do I verify a GPT-to-Hybrid migration preserved the model architecture?▼

Compare parameter counts, allocator layer logs, steady-state throughput, and early-iteration loss between the GPT and Hybrid runs. An architecture-preserving '*-' transfer shows identical parameter counts, throughput within noise, and only zero-mean loss drift; systematic offsets indicate a migration bug.

Related Skills

Explore More Software Engineering Skills→
t2ancet2ance

huggingface-accelerate

Configure and launch distributed PyTorch training with HuggingFace Accelerate.

Community
Advanced
yo-stevenyo-steven

mlm-bridge-training

Correlate Megatron-LM and Megatron Bridge training runs by matching configs and diagnosing loss divergence.

Community
Advanced
faisalanjumfaisalanjum
3

test-arguments

Test substitution of $ARGUMENTS placeholders in skill markdown content.

Community
Basic
furqan90furqan90

spec-remix

Reuse validated Project Spec files to generate v1.1-conformant updated specs.

Community
Intermediate