nemo-mbridge-perf-tp-dp-comm-overlap

Configure TP, DP, and PP communication overlap in Megatron-Bridge training.

Updated Apr 16, 2026

Search Tags:#throughput #megatron-bridge #comm_overlap #tp_overlap #dp_overlap #pp_overlap

Ready to run

npx skills add https://github.com/sayalinvidia/sayali-skills-test --skill nemo-mbridge-perf-tp-dp-comm-overlap

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: nemo-mbridge-perf-tp-dp-comm-overlap
Source: https://github.com/sayalinvidia/sayali-skills-test/tree/main/skills/nemo-mbridge-perf-tp-dp-comm-overlap
Command: npx skills add https://github.com/sayalinvidia/sayali-skills-test --skill nemo-mbridge-perf-tp-dp-comm-overlap

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Enables and validates Tensor-Parallel, Data-Parallel, and Pipeline-Parallel communication overlap in Megatron-Bridge to boost training throughput and efficiency.

Core Features & Use Cases

Provides configuration knobs for TP, DP, and PP overlap in Megatron-Bridge training.
Includes code anchors and launch-time environment tuning guidance.
Offers verification steps and pitfalls to ensure safe, correct overlap behavior.
Use Case: Optimizing throughput for large-scale transformer training and diagnosing related performance regressions.

Quick Start

Enable TP/DP/PP communication overlap by applying the recommended CommOverlapConfig settings in Megatron-Bridge and verify the wiring with the provided unit tests.

Frequently Asked Questions about nemo-mbridge-perf-tp-dp-comm-overlap

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I enable communication overlap for tensor and data parallelism in Megatron-Bridge?▼

To enable communication overlap in Megatron-Bridge, apply the recommended CommOverlapConfig settings, ensuring tensor_model_parallel_size is greater than 1 and sequence_parallel is enabled. You also need to activate the distributed-optimizer path with overlap_grad_reduce and overlap_param_gather for proper wiring.

What prerequisites are needed for TP overlap in transformer training?▼

TP overlap requires tensor_model_parallel_size to be greater than 1 and sequence parallelism to be enabled. Activating the distributed optimizer with overlap_grad_reduce and overlap_param_gather is also required to properly support the overlap configuration in Megatron-Bridge.

How can I verify if TP, DP, and PP communication overlap is working correctly?▼

Verify communication overlap behavior by running the provided unit tests to check the wiring. The Skill includes verification steps and outlines pitfalls to help you validate correct overlap behavior and detect performance regressions during large-scale transformer training.

Why is my tensor parallel communication overlap not improving training throughput?▼

Throughput may not improve if overlap configurations are incorrectly applied. Ensure the distributed-optimizer path is active with overlap_grad_reduce and overlap_param_gather, and check the provided launch-time environment tuning guidance to diagnose performance regressions.

Does Megatron-Bridge support overlapping pipeline parallel communication?▼

Yes, Megatron-Bridge supports pipeline parallel communication overlap to boost training throughput. The Skill provides configuration knobs for TP, DP, and PP overlap, alongside code anchors and usage notes to ensure safe and correct overlap behavior.

What is the best way to diagnose performance regressions related to communication overlap?▼

Diagnose performance regressions by utilizing the Skill's verification steps and pitfalls. Validate proper overlap behavior by running unit tests and applying the recommended CommOverlapConfig settings along with launch-time environment tuning guidance.