hybrid-context-parallel

To enable hierarchical context parallelism in Megatron-Bridge, set context_parallel_size, configure cp_comm_type to 'a2a+p2p', and define hierarchical_context_parallel_sizes. This structures compute and communication hierarchically to scale multi-GPU Transformer workloads.

Hierarchical context parallelism is used to enable scalable distributed training across multiple GPU groups. It structures compute and communication in hierarchical levels within Megatron-Bridge, preventing silent failures and ensuring correct setup for large-scale Transformer workloads.

Yes, Transformer Engine supports hierarchical context parallelism but requires version 1.12.0 or higher. You must also ensure the product of hierarchical_context_parallel_sizes equals context_parallel_size for proper group initialization and CP scaling.

Configure context_parallel_size to 4 and set hierarchical_context_parallel_sizes to [2,2] as a quick start. The product of hierarchical_context_parallel_sizes must equal context_parallel_size, and seq_length must be divisible by 2 times context_parallel_size.

Your multi-GPU context parallel run fails validation if the product of hierarchical_context_parallel_sizes does not equal context_parallel_size, or if seq_length is not divisible by 2 times context_parallel_size. Verify group initialization and CP scaling to avoid misconfiguration.

Using a2a+p2p for context parallelism requires strict configuration matching, specifically Transformer Engine 1.12.0 or higher and exact divisibility constraints for seq_length. Misconfiguration can lead to silent failures during distributed runs if group initialization is not validated.