skills.rest
yo-steven
yo-stevenCOMMUNITY
Explore all skills by this author →
GitHub

perf-parallelism-strategies

Map model size and hardware topology to Megatron Bridge TP/DP/PP/SP/CP/EP degrees.

Updated May 23, 2026
Search Tags:#parallelism#tensor parallel#pipeline parallel#megatron bridge#expert parallel#gpu topology#sizing rules
One-click install
npx skills add https://github.com/yo-steven/skills-exploration-20260522 --skill perf-parallelism-strategies-yo-steven
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: perf-parallelism-strategies
Source: https://github.com/yo-steven/skills-exploration-20260522/tree/main/skills/Megatron-Bridge/perf-parallelism-strategies
Command: npx skills add https://github.com/yo-steven/skills-exploration-20260522 --skill perf-parallelism-strategies-yo-steven

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill helps you choose and combine TP/DP/PP/SP/CP/EP degrees for Megatron Bridge so you can hit the right performance and memory shape without over-allocating GPUs.

Core Features & Use Cases

  • Model-size heuristics: recommends starting parallelism plans for dense and MoE model ranges (including active-parameter-aware MoE sizing).
  • Hardware topology mapping: guides how to keep TP within NVLink domains and use PP/DP across nodes for stable scaling.
  • Sequence-length driven add-ons: suggests enabling SP and CP based on sequence length constraints.
  • Combined parallelism configuration: provides concrete parameter combinations for 3D (TP+PP+SP+DP) and 4D (TP+PP+CP+DP) setups, plus MoE-specific EP+PP layouts.
  • Minimum GPU count sanity check: explains the correct min-GPU formula using shared mesh overlap across PP stages to avoid common over-splitting mistakes.

Quick Start

Run through the dense or MoE table for your model size, then translate the selected TP/PP/(CP|SP) guidance into the corresponding cfg.model.* settings and sanity-check min_gpus with PP * max(TPCP, EPETP).

Frequently Asked Questions about perf-parallelism-strategies

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I choose the right tensor parallel and pipeline parallel degrees for Megatron?▼

To choose tensor parallel (TP) and pipeline parallel (PP) degrees for Megatron, map your model size to recommended ranges, keeping TP within an NVLink domain and using PP across nodes for stable scaling.

When do I need to enable sequence parallelism and context parallelism in Megatron Bridge?▼

You need to enable sequence parallelism (SP) and context parallelism (CP) in Megatron Bridge when your sequence length constraints exceed standard memory capacity, with CP requiring specific divisibility constraints to function correctly.

What is the minimum GPU count formula for 3D and 4D parallelism configurations?▼

The minimum GPU count formula for 3D and 4D parallelism configurations is calculated as PP * max(TP*CP, EP*ETP), using shared mesh overlap across PP stages to avoid common over-splitting mistakes.

How do I configure expert parallel and pipeline parallel layouts for MoE models?▼

To configure expert parallel (EP) and pipeline parallel (PP) layouts for MoE models, use active-parameter-aware MoE sizing rules to determine EP degrees, combining them with PP across nodes to optimize memory and performance.

Why does my Megatron training run out of memory after changing parallelism degrees?▼

Your Megatron training runs out of memory after changing parallelism degrees because TP might have exceeded the NVLink domain, or CP divisibility constraints were violated, requiring a sanity check against the minimum-GPU overlap formula.

Related Skills

Explore More Software Engineering Skills→
mlc-aimlc-ai
327

estimate-memory

Estimate peak GPU memory usage for DualPipeV MoE training runs.

Official
Advanced
sayalinvidiasayalinvidia

nemo-mbridge-perf-parallelism-strategies

Select and tune Megatron Bridge parallelism configurations for model and hardware.

Community
Advanced
tadod12tadod12

training-llms-megatron

Train large language models with Megatron-Core parallelism on GPU clusters.

Community
Advanced
zeronx798zeronx798

huggingface-accelerate

Implement distributed training strategies for PyTorch models across GPUs and nodes.

Community
Advanced