skills.rest
yo-steven
yo-stevenCOMMUNITY
Explore all skills by this author →
GitHub

perf-moe-long-context

Configure MoE long-context training with CP sizing and selective recompute.

Updated May 23, 2026
Search Tags:#long-context#dispatcher#moe#megatron-bridge#recompute#context-parallelism#throughput-optimization
One-click install
npx skills add https://github.com/yo-steven/skills-exploration-20260522 --skill perf-moe-long-context-yo-steven
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: perf-moe-long-context
Source: https://github.com/yo-steven/skills-exploration-20260522/tree/main/skills/Megatron-Bridge/perf-moe-long-context
Command: npx skills add https://github.com/yo-steven/skills-exploration-20260522 --skill perf-moe-long-context-yo-steven

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

It solves the problem of unstable and memory-bound MoE training when sequence length moves far beyond the 4K regime, causing long-context OOMs or major throughput drops.

Core Features & Use Cases

  • CP sizing for long sequences: Establishes a practical starting point (CP ≈ seq_len / 4096) and explains how CP changes the memory/throughput trade-offs for MoE.
  • Selective recompute guidance: Recommends recomputing MoE/MLP-side modules (and avoiding SDPA-heavy recompute at very long context) to preserve memory headroom.
  • Dispatcher and parallelism patterns: Provides dispatcher choices (e.g., DeepEP vs HybridEP) and representative TP/CP/EP/PP/VPP layouts drawn from DSV3, Qwen3, and Qwen3-Next long-context runs.
  • Practical pitfalls and batching/graph constraints: Highlights why GPU-count feasibility becomes the real constraint, and why CUDA graphs require static shapes.

Quick Start

Use the perf-moe-long-context skill to pick a CP-first layout, apply selective recompute for MoE-side modules, and choose a dispatcher strategy when training an MoE model at 128K tokens with an NVIDIA Megatron-Bridge setup.

Frequently Asked Questions about perf-moe-long-context

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I prevent OOMs when training MoE models with long context lengths?▼

To prevent MoE long-context OOMs, establish context parallelism sizing at CP ≈ seq_len / 4096 and apply selective recompute for MoE-side modules to preserve memory headroom during Megatron-Bridge runs.

What is the best context parallelism layout for 128K token MoE training?▼

For 128K token MoE training, start with CP ≈ seq_len / 4096 and select representative TP/CP/EP/PP/VPP layouts from DSV3, Qwen3, or Qwen3-Next runs to determine feasible GPU-count configurations.

Does Megatron-Bridge support DeepEP and HybridEP dispatchers for long-context MoE?▼

Megatron-Bridge supports dispatcher choices including DeepEP and HybridEP, which combine with context parallelism and pipeline parallelism layouts to sustain throughput for DSV3, Qwen3, and Qwen3-Next long-context runs.

Why does my MoE long-context training fail with CUDA graphs at extreme sequence lengths?▼

MoE long-context training fails because CUDA graphs require static shapes, creating constraints at extreme sequence lengths. Additionally, avoiding SDPA-heavy recompute prevents memory bottlenecks when scaling to 256K tokens.

When should I avoid SDPA-heavy recompute during long-context MoE training?▼

Avoid SDPA-heavy recompute at extreme context lengths (16K–256K) to preserve memory headroom. Instead, recompute MoE and MLP-side modules to prevent OOMs and sustain throughput when sequence length increases.

Related Skills

Explore More Software Engineering Skills→
sayalinvidiasayalinvidia

nemo-mbridge-perf-parallelism-strategies

Select and tune Megatron Bridge parallelism configurations for model and hardware.

Community
Advanced
sayalinvidiasayalinvidia

nemo-mbridge-perf-moe-long-context

Guide long-context MoE training with CP heuristics and selective recompute for Megatron Bridge.

Community
Advanced
NVIDIANVIDIA
2.8k

memory-tuning

Mitigate GPU memory pressure and OOM during Megatron training with fragmentation fixes.

Official
Advanced
sayalinvidiasayalinvidia

nemo-mbridge-perf-megatron-fsdp

Enable Megatron FSDP overrides in Megatron-Bridge for distributed training.

Community
Advanced