nemo-mbridge-perf-tp-dp-comm-overlap

To enable communication overlap in Megatron-Bridge, apply the recommended CommOverlapConfig settings, ensuring tensor_model_parallel_size is greater than 1 and sequence_parallel is enabled. You also need to activate the distributed-optimizer path with overlap_grad_reduce and overlap_param_gather for proper wiring.

TP overlap requires tensor_model_parallel_size to be greater than 1 and sequence parallelism to be enabled. Activating the distributed optimizer with overlap_grad_reduce and overlap_param_gather is also required to properly support the overlap configuration in Megatron-Bridge.

Verify communication overlap behavior by running the provided unit tests to check the wiring. The Skill includes verification steps and outlines pitfalls to help you validate correct overlap behavior and detect performance regressions during large-scale transformer training.

Throughput may not improve if overlap configurations are incorrectly applied. Ensure the distributed-optimizer path is active with overlap_grad_reduce and overlap_param_gather, and check the provided launch-time environment tuning guidance to diagnose performance regressions.

Yes, Megatron-Bridge supports pipeline parallel communication overlap to boost training throughput. The Skill provides configuration knobs for TP, DP, and PP overlap, alongside code anchors and usage notes to ensure safe and correct overlap behavior.

Diagnose performance regressions by utilizing the Skill's verification steps and pitfalls. Validate proper overlap behavior by running unit tests and applying the recommended CommOverlapConfig settings along with launch-time environment tuning guidance.