skills.rest
sayalinvidia
sayalinvidiaCOMMUNITY
Explore all skills by this author →
GitHub

nemo-mbridge-perf-hierarchical-context-parallel

Configure hierarchical context parallelism for Megatron-Bridge training jobs.

Updated Apr 16, 2026
Search Tags:#distributed-training#Megatron-Bridge#hierarchical-context-parallel#transformer-engine#context-parallel
One-click install
npx skills add https://github.com/sayalinvidia/sayali-skills-test --skill nemo-mbridge-perf-hierarchical-context-parallel
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: nemo-mbridge-perf-hierarchical-context-parallel
Source: https://github.com/sayalinvidia/sayali-skills-test/tree/main/skills/nemo-mbridge-perf-hierarchical-context-parallel
Command: npx skills add https://github.com/sayalinvidia/sayali-skills-test --skill nemo-mbridge-perf-hierarchical-context-parallel

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Scaling context parallelism in Megatron-Bridge to support larger models and datasets by enabling hierarchical context parallelism (HCP) across groups.

Core Features & Use Cases

  • Enable per-level context-parallel grouping for a2a+p2p communication in Megatron-Bridge.
  • Enforce configuration constraints (product of hierarchical_context_parallel_sizes equals context_parallel_size; seq_length divisible by 2 * context_parallel_size; Transformer Engine version >= 1.12.0).
  • Verify implementation through logs showing HIERARCHICAL_CONTEXT_PARALLEL_GROUPS during initialization and by running manual smoke tests.

Quick Start

Set cfg.model.context_parallel_size=4, cfg.model.cp_comm_type="a2a+p2p", cfg.model.hierarchical_context_parallel_sizes=[2, 2], and verify logs show HIERARCHICAL_CONTEXT_PARALLEL_GROUPS.

Frequently Asked Questions about nemo-mbridge-perf-hierarchical-context-parallel

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I configure hierarchical context parallelism in Megatron-Bridge?▼

Verify hierarchical context parallelism initialization by checking logs for HIERARCHICAL_CONTEXT_PARALLEL_GROUPS during startup or by running manual smoke tests to confirm per-level grouping is active.

What constraints apply to seq_length when using hierarchical context parallelism?▼

Transformer Engine version 1.12.0 or higher is required for hierarchical context parallelism in Megatron-Bridge, ensuring a2a+p2p communication supports per-level grouping across multi-GPU deployments.

Why do I need hierarchical context parallelism for distributed training?▼

Set context_parallel_size=4, cp_comm_type="a2a+p2p", and hierarchical_context_parallel_sizes=[2, 2] to enable hierarchical CP, verifying logs show HIERARCHICAL_CONTEXT_PARALLEL_GROUPS during initialization.

Does hierarchical context parallelism work with older Transformer Engine versions?▼

Hierarchical context parallelism applies to multi-GPU Megatron-Bridge deployments using a2a+p2p communication, enforcing that hierarchical_context_parallel_sizes product equals context_parallel_size and seq_length divisibility constraints.

Related Skills

Explore More Software Engineering Skills→
NVIDIANVIDIA
2.8k

hybrid-context-parallel

Enable hierarchical context parallelism in Megatron-Bridge for multi-GPU Transformer workloads.

Official
Advanced
automatedigitalautomatedigital
1

pytorch-fsdp

Configure PyTorch FSDP1 and FSDP2 for distributed training with mixed precision and CPU offloading.

Official
Advanced
yo-stevenyo-steven

perf-hierarchical-context-parallel

Configure hierarchical context parallelism for Megatron-Bridge training runs.

Community
Intermediate
ZurybrZurybr

torch-pipeline-parallelism

Implement PyTorch pipeline parallelism for distributed LLM training across multiple GPUs.

Community
Advanced