nemo-mbridge-perf-expert-parallel-overlap

To overlap expert-parallel token dispatch, enable MoE expert-parallel communication overlap in Megatron-Bridge using either alltoall or flex dispatchers. This hides dispatch and all-to-all communication costs alongside FFN compute to reduce per-step latency.

Prerequisites for MoE expert-parallel overlap include applying it to models with expert_model_parallel_size > 1, enabling the overlap flag, and setting moe_shared_expert_overlap to False to ensure correct distributed training behavior.

Megatron-Bridge EP overlap does not work with FP32 or older PyTorch versions. Compatibility constraints require BF16 or FP16 precision and PyTorch version 2.6.0 or higher for correct MoE overlap functionality.

You can use delayed wgrad compute with expert-parallel overlap, but it introduces additional Transformer Engine and CUDA-graph considerations. You must account for these specific constraints when configuring large MoE models for throughput tuning.

The best way to validate performance is running the Megatron-Bridge performance setup for EP overlap testing on a representative MoE model. This provides correctness-first validation and performance evaluation for your specific deployment.