sequence-packing

To configure packed sequences for long-context LLM finetuning, use the offline packed sequence SFT path in Megatron-Bridge, ensuring proper padding and sequence length divisibility by 2 * context_parallel_size for stable training.

Offline packed SFT handles long-context LLM finetuning by pre-packing sequences, while in-batch packing dynamically packs multimodal inputs for VLM finetuning, specifically requiring aligned context-parallel constraints when CP is enabled.

Megatron-Bridge training fails with packed sequences if sequence lengths are not divisible by 2 * context_parallel_size or padding rules are ignored, as the PackedSequenceSpecs enforce strict divisibility constraints for CP stability.

In-batch packing for VLM finetuning is designed for scenarios requiring context-parallelism, but you must apply the correct CP constraints and configuration keys to ensure proper sequence management and avoid training instability.

Megatron-Bridge packed sequence workflows require sequence lengths to be divisible by 2 * context_parallel_size, enforce proper padding rules, and mandate specific configuration keys to validate PackedSequenceSpecs during training.

Use sequence-packing for Megatron-Bridge models when planning or debugging long-context training runs, specifically for LLMs with large sequence lengths or VLMs requiring in-batch packing with context-parallelism enabled.