perf-hierarchical-context-parallel

To enable hierarchical context parallelism in Megatron-Bridge, set cp_comm_type to a2a+p2p and configure hierarchical_context_parallel_sizes. You must also set dist.use_decentralized_pg to false and launch a multi-GPU run.

Context parallelism causes OOM or regressions when hierarchical settings are misconfigured. The product of hierarchical_context_parallel_sizes must equal context_parallel_size, and sequence length must be divisible, otherwise training breaks or fails silently.

Hierarchical context parallelism requires Transformer Engine version 1.12.0 or higher. You must verify this dependency before attempting to scale context parallelism beyond KV heads using a2a+p2p communication.

To validate hierarchical context parallel groups, launch a small multi-GPU run and check the training logs. You should confirm the presence of HIERARCHICAL_CONTEXT_PARALLEL_GROUPS entries to ensure correct process-group creation.

You need hierarchical context parallelism when scaling context parallelism beyond KV heads in distributed training. It prevents misconfiguration issues when using a2a+p2p communication types in Megatron-Bridge runs.

The prod of hierarchical_context_parallel_sizes must exactly equal context_parallel_size, and sequence length must be divisible. These constraints match Megatron-LM expectations and prevent OOM or broken training.