skills.rest
yo-steven
yo-stevenCOMMUNITY
Explore all skills by this author →
GitHub

perf-sequence-packing

Validate and configure packed-sequence and long-context training in Megatron-Bridge.

Updated May 23, 2026
Search Tags:#megatron-bridge#sequence packing#context parallelism#training config validation#packed sequences#CUDA graphs#VLM fine-tuning
One-click install
npx skills add https://github.com/yo-steven/skills-exploration-20260522 --skill perf-sequence-packing-yo-steven
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: perf-sequence-packing
Source: https://github.com/yo-steven/skills-exploration-20260522/tree/main/skills/Megatron-Bridge/perf-sequence-packing
Command: npx skills add https://github.com/yo-steven/skills-exploration-20260522 --skill perf-sequence-packing-yo-steven

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill helps you validate and correctly apply packed-sequence and long-context training behaviors in Megatron-Bridge, preventing subtle misconfigurations that break packing, CP constraints, or runtime assumptions.

Core Features & Use Cases

  • Offline packed SFT for LLMs: Enables packed datasets using PackedSequenceSpecs with consistent padding and micro-batch rules to match THD-packed dataset expectations.
  • In-batch packing for VLM finetuning: Turns on pack_sequences_in_batch for supported VLM flows, with micro-batch sizing rules that differ from offline packed SFT.
  • Context Parallel (CP) correctness: Applies the required divisibility and parameter constraints (e.g., calculate_per_token_loss and ddp.average_in_collective) so packed sequences work under CP without runtime failures.

Quick Start

Enable offline packed SFT by setting cfg.train.micro_batch_size = 1, matching cfg.dataset.seq_length and cfg.model.seq_length to your target length (e.g., 4096), and configuring cfg.dataset.packed_sequence_specs = PackedSequenceSpecs(packed_sequence_size=4096, pad_seq_to_mult=1).

Frequently Asked Questions about perf-sequence-packing

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I configure offline packed sequence SFT in Megatron-Bridge?▼

To configure offline packed sequence SFT in Megatron-Bridge, set your micro-batch size to 1, match dataset and model sequence lengths, and define PackedSequenceSpecs with consistent padding to align with THD-packed dataset expectations.

What parameters are required for context parallelism with packed sequences?▼

Context parallelism with packed sequences requires enforcing divisibility constraints and enabling parameters like calculate_per_token_loss and ddp.average_in_collective to prevent runtime failures during distributed training.

Can I use in-batch packing for VLM fine-tuning in Megatron-Bridge?▼

Yes, VLM fine-tuning supports in-batch packing by enabling pack_sequences_in_batch, but you must apply specific micro-batch sizing rules that differ from standard offline packed SFT to ensure correct execution.

Why does packed sequence training break when changing long-context behavior?▼

Packed sequence training breaks because changing long-context behavior risks violating padding alignment rules, context parallelism divisibility constraints, and CUDA-graph packed path runtime assumptions without proper validation.

What padding settings need to be aligned for Megatron-Bridge sequence packing?▼

Megatron-Bridge sequence packing requires aligning padding settings such as pad_seq_to_mult and pad_cu_seqlens to maintain dataset consistency and prevent micro-batch validation errors during training.

Related Skills

Explore More Software Engineering Skills→
NVIDIANVIDIA
2.8k

sequence-packing

Manages PackedSequenceSpecs and context-parallel constraints for Megatron-Bridge long-context training.

Official
Advanced
sayalinvidiasayalinvidia

nemo-mbridge-perf-sequence-packing

Configure and validate packed sequences and long-context training in Megatron-Bridge.

Community
Advanced
gztcodegztcode

pytorch-fsdp

Configure PyTorch FSDP for distributed training with parameter tuning.

Community
Advanced
davpatel605-beepdavpatel605-beep

simpo-training

Configure and run reference-free SimPO preference alignment training for LLMs.

Community
Advanced