hybrid-context-parallel

Enable hierarchical context parallelism in Megatron-Bridge for multi-GPU Transformer workloads.

2.8k|332|Updated Feb 25, 2026

Search Tags:#distributed-training #megatron-bridge #hierarchical-context-parallel #transformer-engine #context-parallelism #a2a+p2p #gpu-scaling

Ready to run

npx skills add https://github.com/NVIDIA/skills --skill hybrid-context-parallel

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: hybrid-context-parallel
Source: https://github.com/NVIDIA/skills/tree/main/skills/Megatron-Bridge/perf-techniques/hybrid-context-parallel
Command: npx skills add https://github.com/NVIDIA/skills --skill hybrid-context-parallel

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Hierarchical context parallelism enables scalable training across multiple GPU groups by structuring compute and communication in hierarchical levels within Megatron-Bridge.

Core Features & Use Cases

Provides configuration knobs for enabling HCP, including context_parallel_size, cp_comm_type, and hierarchical_context_parallel_sizes.
Documents code anchors and validation rules to ensure correct setup and verification of HCP groups.
Describes pitfalls and verification steps to avoid misconfiguration and silent failures in distributed runs.

Quick Start

Configure context_parallel_size to 4, set cp_comm_type to a2a+p2p, and provide hierarchical_context_parallel_sizes as [2,2], then run a small multi-GPU test to verify hierarchical groups are created.

Frequently Asked Questions about hybrid-context-parallel

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I enable hierarchical context parallelism in Megatron-Bridge?▼

To enable hierarchical context parallelism in Megatron-Bridge, set context_parallel_size, configure cp_comm_type to 'a2a+p2p', and define hierarchical_context_parallel_sizes. This structures compute and communication hierarchically to scale multi-GPU Transformer workloads.

What is hierarchical context parallelism used for in distributed training?▼

Hierarchical context parallelism is used to enable scalable distributed training across multiple GPU groups. It structures compute and communication in hierarchical levels within Megatron-Bridge, preventing silent failures and ensuring correct setup for large-scale Transformer workloads.

Does Transformer Engine support hierarchical context parallelism?▼

Yes, Transformer Engine supports hierarchical context parallelism but requires version 1.12.0 or higher. You must also ensure the product of hierarchical_context_parallel_sizes equals context_parallel_size for proper group initialization and CP scaling.

How do I configure context_parallel_size and hierarchical_context_parallel_sizes correctly?▼

Configure context_parallel_size to 4 and set hierarchical_context_parallel_sizes to [2,2] as a quick start. The product of hierarchical_context_parallel_sizes must equal context_parallel_size, and seq_length must be divisible by 2 times context_parallel_size.

Why does my multi-GPU context parallel run fail validation checks?▼

Your multi-GPU context parallel run fails validation if the product of hierarchical_context_parallel_sizes does not equal context_parallel_size, or if seq_length is not divisible by 2 times context_parallel_size. Verify group initialization and CP scaling to avoid misconfiguration.

What are the limitations of using a2a+p2p for context parallelism?▼

Using a2a+p2p for context parallelism requires strict configuration matching, specifically Transformer Engine 1.12.0 or higher and exact divisibility constraints for seq_length. Misconfiguration can lead to silent failures during distributed runs if group initialization is not validated.