perf-hybrid-context-parallel

To configure hierarchical context parallelism in Megatron-Bridge, set the `context_parallel_size` and `hierarchical_context_parallel_sizes` parameters. This Skill provides configuration guidance and validation rules to ensure correct parallelism setup and prevent training failures.

You need hierarchical context parallelism when scaling Megatron models beyond KV heads or troubleshooting OOM issues related to context parallelism. It enables efficient scaling of large-scale transformer training workloads with layered parallel splits.

Context parallelism fails due to misconfigurations of hierarchical parallelism parameters and Transformer Engine version incompatibility. This Skill validates parameter constraints and provides code anchors to avoid these training regressions and failures.

No, using hierarchical context parallelism requires prior knowledge of Megatron's parallel setup and Transformer Engine version compatibility. This Skill targets model developers and system architects implementing large-scale transformer training.

Validation rules for hierarchical context parallelism parameters govern the correct configuration of `context_parallel_size` and `hierarchical_context_parallel_sizes`. This Skill details these constraints and provides code anchors to ensure valid parameter combinations.