tp-dp-comm-overlap
OfficialEnable TP/DP/PP overlap in Megatron-Bridge.
Software Engineering#performance-tuning#unit-tests#distributed-training#gpu-parallelism#megatron-bridge#tp-dp-pp-overlap#communication-overlap
AuthorNVIDIA
Version1.0.0
Installs0
System Documentation
What problem does it solve?
Overcoming communication bottlenecks in Megatron-Bridge by enabling overlap among tensor-parallel, data-parallel, and pipeline-parallel paths during distributed training.
Core Features & Use Cases
- Config-driven overlap: Tune tensor-model-parallel size, sequence parallel, and pipeline parallel size to achieve simultaneous data, tensor, and pipeline communications.
- Code anchors & guidance: Provides code anchors and practical pitfalls to ensure correct overlap behavior and safe deployment.
- Verification & troubleshooting: Includes verification steps, pitfalls to avoid, and launch-time environment tuning recommendations for different scheduling modes.
Quick Start
Enable TP/DP/PP overlap in Megatron-Bridge by applying the CommOverlapConfig knobs shown in the skill and verify the overlap behavior with the provided unit tests.
Dependency Matrix
Required Modules
None requiredComponents
Standard package💻 Claude Code Installation
Recommended: Let Claude install automatically. Simply copy and paste the text below to Claude Code.
Please help me install this Skill: Name: tp-dp-comm-overlap Download link: https://github.com/NVIDIA/skills/archive/main.zip#tp-dp-comm-overlap Please download this .zip file, extract it, and install it in the .claude/skills/ directory.
Agent Skills Search Helper
Install a tiny helper to your Agent, search and equip skill from 471,000+ vetted skills library on demand.