nemo-mbridge-perf-moe-long-context

What problem does it solve?

Provides practical guidance for training MoE models with long-context sequences, including CP sizing, selective recompute, and dispatcher choices to avoid OOM and throughput degradation.

Core Features & Use Cases

• Long-context MoE training guidance covering CP sizing rules, selective recompute recommendations (up_proj, norm, moe, mlp), and CUDA graph considerations.
• Representative config templates and best practices for DSV3 and Qwen3-Next long-context experiments, enabling efficient planning and deployment.
• Pitfalls and safety considerations to help engineers avoid memory and performance issues when scaling to 128K+ context lengths.

Quick Start

Configure your training using the CP ~= seq_len / 4096 heuristic with preserved DP and selective recompute for long-context MoE workloads.