length-pool-sort-dataset

Name: length-pool-sort-dataset
Availability: InStock
Author: LMMs-Lab

Explain cross-rank length synchronization in LengthPoolSortDataset for multi-GPU training.

1.2k|78|Updated Sep 16, 2025

Search Tags:#synchronization #data-pipeline #distributed-training #multigpu #length-sort #length-pool

Ready to run

npx skills add https://github.com/EvolvingLMMs-Lab/LLaVA-OneVision-2 --skill length-pool-sort-dataset

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: length-pool-sort-dataset
Source: https://github.com/EvolvingLMMs-Lab/LLaVA-OneVision-2/tree/main/.opencode/skills/length-pool-sort-dataset
Command: npx skills add https://github.com/EvolvingLMMs-Lab/LLaVA-OneVision-2 --skill length-pool-sort-dataset

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

LengthPoolSortDataset provides a cross-rank length synchronization mechanism to reduce idle time in multi-GPU training by aligning sample lengths across ranks, improving effective training throughput.

Core Features & Use Cases

Three-step pipeline: upstream dataset → accumulate pool_size samples → sort by sequence length → deterministic seed shuffle → yield samples, enabling cross-rank length consistency.
Activation and tuning: enabled via --length-sort-size N (N > 0); guidance on pool_size selection, impact of num_workers, and memory considerations for stronger synchronization.
Debugging and caveats: notes on pool internal state not being saved, behavior with multiple workers, and recommended checks for length distribution and step-time variance.

Quick Start

Enable LengthPoolSortDataset by setting --length-sort-pool-size to a positive value and monitor training step time variance for improvement.

Frequently Asked Questions about length-pool-sort-dataset

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I reduce idle time in multi-GPU training caused by uneven sequence lengths?▼

Cross-rank length synchronization reduces multi-GPU training idle time by aligning sample lengths across ranks. LengthPoolSortDataset accumulates a pool of samples, sorts them by sequence length, and applies a deterministic shuffle to yield batches, improving effective training throughput.

How do I enable length sorting for my distributed training data pipeline?▼

Enable length sorting by setting the --length-sort-pool-size argument to a positive value. This activates the LengthPoolSortDataset, which synchronizes cross-rank lengths to speed up multi-GPU training throughput.

What is the best way to tune pool size for sequence length sorting in distributed training?▼

Tune pool size by monitoring training step time variance for improvement after setting --length-sort-pool-size. Consider memory limitations and the impact of num_workers, as a larger pool size provides stronger length synchronization but requires more memory.

Does LengthPoolSortDataset work with multiple workers and checkpoint resume?▼

LengthPoolSortDataset works with multiple workers but the pool's internal state is not saved during checkpoint resume. You should perform debugging checks on length distribution and step-time variance to ensure consistent behavior across workers.

Why does multi-GPU training step time fluctuate when batching by sequence length?▼

Step time fluctuates because uneven sequence lengths cause some GPUs to wait for others. LengthPoolSortDataset addresses this by synchronizing cross-rank lengths, accumulating and sorting samples to align processing times and reduce idle variance.