expert-parallel-overlap

To overlap expert-parallel communication with compute in MoE training, you can enable this Skill to reduce training latency by hiding token dispatch and gather operations behind expert compute using alltoall or flex dispatchers.

Overlapping MoE expert-parallel compute requires expert_model_parallel_size > 1, num_moe_experts > 1, moe_token_dispatcher_type set to alltoall or flex, bf16 or fp16 precision, and PyTorch >= 2.6.0.

Yes, MoE expert-parallel overlap supports optional delayed wgrad scheduling and CUDA graph interactions to further hide communication latency across Ampere, Hopper, and Blackwell GPU architectures.

Yes, you can use the flex dispatcher alongside the alltoall backend for MoE token dispatch overlap, which is specifically designed to unlock throughput gains in large-scale and high-EP MoE workloads.

Expert-parallel communication causes latency in distributed MoE workloads because dispatching tokens across experts creates synchronization bottlenecks that block compute pipelines unless explicitly overlapped.

The best way to reduce step time when dispatching tokens across MoE experts is to overlap the alltoall or flex communication operations with expert compute, optionally scheduling delayed wgrad to maximize throughput.