What problem does it solve?
Communication overlap configuration is often the difference between steady throughput and a sudden regression, because TP/DP/PP pipelines depend on specific gating conditions and runtime choices inside Megatron-Bridge.
Core Features & Use Cases
- Enables TP communication overlap: Turns on tensor-parallel comm overlap when the required gating conditions are met, including sequence parallelism and TE availability.
- Enables DP comm overlap on distributed optimizer paths: Sets distributed optimizer modes and overlaps gradient reduce and parameter gather behavior for higher utilization.
- Selects PP overlap behavior correctly: Applies the PP overlap configuration logic that depends on pipeline parallel and virtual pipeline sizes, including p2p vs batch behavior.
- Provides code anchors and pitfalls: Points directly to the relevant Megatron-Bridge implementation sections and highlights common misconfigurations that silently disable overlap.
- Verification-focused: Uses the repository’s checked-in unit tests to confirm the expected overlap logic before running expensive benchmarks.
Quick Start
Use the Megatron-Bridge CommOverlapConfig snippet to set tp_comm_overlap=True plus ddp distributed-optimizer overlap flags, then validate with the provided unit test commands.
