hybrid-context-parallel

What problem does it solve?

Hierarchical context parallelism enables scalable training across multiple GPU groups by structuring compute and communication in hierarchical levels within Megatron-Bridge.

Core Features & Use Cases

• Provides configuration knobs for enabling HCP, including context_parallel_size, cp_comm_type, and hierarchical_context_parallel_sizes.
• Documents code anchors and validation rules to ensure correct setup and verification of HCP groups.
• Describes pitfalls and verification steps to avoid misconfiguration and silent failures in distributed runs.

Quick Start

Configure context_parallel_size to 4, set cp_comm_type to a2a+p2p, and provide hierarchical_context_parallel_sizes as [2,2], then run a small multi-GPU test to verify hierarchical groups are created.