skills.rest
sayalinvidia
sayalinvidiaCOMMUNITY
Explore all skills by this author →
GitHub

nemo-mbridge-perf-moe-long-context

Guide long-context MoE training with CP heuristics and selective recompute for Megatron Bridge.

Updated Apr 16, 2026
Search Tags:#training#throughput#long-context#dispatcher#moe#cp-sizing#selective-recompute
One-click install
npx skills add https://github.com/sayalinvidia/sayali-skills-test --skill nemo-mbridge-perf-moe-long-context
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: nemo-mbridge-perf-moe-long-context
Source: https://github.com/sayalinvidia/sayali-skills-test/tree/main/skills/nemo-mbridge-perf-moe-long-context
Command: npx skills add https://github.com/sayalinvidia/sayali-skills-test --skill nemo-mbridge-perf-moe-long-context

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Provides practical guidance for training MoE models with long-context sequences, including CP sizing, selective recompute, and dispatcher choices to avoid OOM and throughput degradation.

Core Features & Use Cases

  • Long-context MoE training guidance covering CP sizing rules, selective recompute recommendations (up_proj, norm, moe, mlp), and CUDA graph considerations.
  • Representative config templates and best practices for DSV3 and Qwen3-Next long-context experiments, enabling efficient planning and deployment.
  • Pitfalls and safety considerations to help engineers avoid memory and performance issues when scaling to 128K+ context lengths.

Quick Start

Configure your training using the CP ~= seq_len / 4096 heuristic with preserved DP and selective recompute for long-context MoE workloads.

Frequently Asked Questions about nemo-mbridge-perf-moe-long-context

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I size context parallelism for long-context MoE training?▼

To avoid out-of-memory errors during long-context MoE training, enable selective recompute for specific modules including up_proj, norm, moe, moe_act, and mlp to manage memory footprints efficiently.

Does Megatron Bridge support DeepEP and HybridEP dispatcher layouts for MoE?▼

Yes, Megatron Bridge supports DeepEP and HybridEP dispatcher layouts for MoE training, providing specific CUDA graph recommendations to optimize performance and mitigate memory issues at 128K+ context lengths.

What are common pitfalls when scaling MoE training to 128K or 256K context lengths?▼

Common pitfalls when scaling MoE training to 128K+ context lengths include ignoring CP sizing heuristics, improper selective recompute configuration, and suboptimal dispatcher layouts, which lead to memory and performance degradation.

Can I use this long-context MoE guidance for Qwen3 and DSV3 configurations?▼

Yes, this guidance applies to researchers and engineers designing MoE training runs across DSV3, Qwen3, and Qwen3-Next configurations at 4K to 256K context lengths, offering representative config templates and best practices.

Related Skills

Explore More Software Engineering Skills→
NVIDIANVIDIA
2.8k

moe-long-context

Identifies best practices for scaling MoE long-context training in Megatron Bridge.

Official
Advanced
yo-stevenyo-steven

perf-moe-long-context

Configure MoE long-context training with CP sizing and selective recompute.

Community
Intermediate
yo-stevenyo-steven

perf-expert-parallel-overlap

Overlap expert-parallel token dispatch and combine communication with expert feed-forward compute in Megatron-Bridge MoE models.

Community
Advanced
NVIDIA-NeMoNVIDIA-NeMo
852

perf-moe-long-context

Guide training of large MoE models with 16K to 256K token contexts.

Official
Advanced