skills.rest
NVIDIA
NVIDIAOFFICIAL
Explore all skills by this author →
WebsiteGitHub

moe-long-context

Identifies best practices for scaling MoE long-context training in Megatron Bridge.

2.8k|332|Updated Feb 25, 2026
Search Tags:#training#long-context#dispatcher#megatron#moe#cp-sizing#recompute
One-click install
npx skills add https://github.com/NVIDIA/skills --skill moe-long-context
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: moe-long-context
Source: https://github.com/NVIDIA/skills/tree/main/skills/Megatron-Bridge/perf-techniques/moe-long-context
Command: npx skills add https://github.com/NVIDIA/skills --skill moe-long-context

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Long-context MoE training presents memory and throughput challenges as sequence lengths grow; this guide helps engineers optimize CP sizing, dispatcher layouts, and recompute strategies to sustain performance.

Core Features & Use Cases

  • Provides CP scaling rules for MoE training on DSV3, Qwen3, and Qwen3-Next on NV hardware.
  • Recommends selective recompute, CUDA graphs advice, and offload strategies to manage memory and compute.
  • Includes representative config patterns and practical tips for success in long-context MoE workloads.

Quick Start

Configure your Megatron Bridge MoE training with the recommended CP, dispatcher, and selective recompute settings to evaluate throughput.

Frequently Asked Questions about moe-long-context

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I scale MoE long-context training in Megatron without running out of memory?▼

To scale MoE long-context training, apply best practices for CP sizing, selective recompute, and dispatcher selection to manage memory and sustain throughput as sequence lengths grow.

What is the best CP sizing strategy for large-scale MoE runs on DSV3 and Qwen3?▼

CP sizing strategies for DSV3, Qwen3, and Qwen3-Next experiments are determined by specific scaling rules that optimize memory and compute distribution across NV hardware for long-context workloads.

How does selective recompute help with MoE long-context throughput?▼

Selective recompute manages memory and compute during long-context MoE training by strategically re-evaluating activations, freeing up resources to sustain overall throughput.

Can I use CUDA graphs and offload strategies for Megatron Bridge MoE training?▼

Yes, CUDA graph guidance and offload strategies are recommended to manage memory and compute efficiently, providing practical patterns for success in large-scale MoE runs.

Are there representative config patterns available for long-context MoE workloads?▼

Representative configuration patterns and practical tips are provided to ensure reproducibility, offering clear config examples for evaluating throughput in Megatron Bridge MoE training.

Related Skills

Explore More Software Engineering Skills→
NVIDIANVIDIA
2.8k

moe-optimization-workflow

Diagnose and optimize MoE training bottlenecks across memory, compute, and communication.

Official
Advanced
pacphipacphi
14

PyTorch ML

Define and train PyTorch neural networks with CUDA acceleration.

Community
Advanced
NVIDIANVIDIA
2.8k

activation-recompute

Configure selective or full activation recompute to trade memory for compute in Megatron Bridge.

Official
Advanced
KappTech88KappTech88

deepspeed

Configure DeepSpeed ZeRO, pipeline parallelism, and MoE for large-scale training.

Community
Advanced