moe-optimization-workflow

To optimize MoE training bottlenecks, apply the Three Walls framework to diagnose memory, compute, and communication limits, then adjust parallel folding and FP8 strategies within Megatron-Bridge runs to resolve the identified constraints.

The Three Walls framework for MoE training optimization is a diagnostic methodology that categorizes performance bottlenecks into memory, communication, and compute limits, guiding targeted tuning strategies like parallel folding and dispatcher selection to maximize throughput.

To decouple MoE and attention parallelism in Megatron-Bridge, apply the parallel folding and recompute strategy provided by the optimization workflow, which separates the parallelism dimensions to prevent resource contention during training.

Yes, tuning MoE dispatcher settings requires existing knowledge of Megatron, MoE architectures, and hardware topology, as the workflow relies on this understanding to guide configuration changes and profiling for scalable GPU setups.

You should use CUDA Graphs for MoE training throughput when seeking to maximize performance on scalable GPU setups, applying dispatcher tuning alongside it as needed to reduce overhead during Megatron-Bridge execution.

The best way to apply FP8 strategies to Megatron Core MoE runs is to follow the structured optimization workflow, which integrates FP8 configuration with parallel folding and dispatcher tuning to systematically address compute and memory walls.