parallelism-strategies

Configuring 3D parallelism for Megatron Bridge involves identifying optimal starting values for TP, PP, DP, CP, and EP that satisfy specific sizing guidelines. It applies decision guidance to optimize throughput and memory across various hardware topologies.

The best way to choose parallelism strategies for a 236B MoE model on a 256-GPU cluster is applying recommended EP, PP, and TP settings. This approach maximizes training throughput while effectively controlling InfiniBand interconnect overhead.

Yes, Megatron Bridge parallelism supports both single-node NVLink setups and multi-node InfiniBand clusters. It provides hardware-topology mappings and combined parallelism configurations tailored to optimize resource usage across these distinct environments.

To optimize distributed training throughput, combine TP, PP, and DP settings using provided sizing rules and hardware-topology mappings. This combination balances computational load and memory efficiency to achieve scalable training performance.

Expert parallelism (EP) and context parallelism (CP) are needed when configuring dense and MoE models for scalable training. They are specified alongside TP, PP, and DP to satisfy guidelines and control memory usage across large GPU clusters.