length-pool-sort-dataset

What problem does it solve?

LengthPoolSortDataset provides a cross-rank length synchronization mechanism to reduce idle time in multi-GPU training by aligning sample lengths across ranks, improving effective training throughput.

Core Features & Use Cases

• Three-step pipeline: upstream dataset → accumulate pool_size samples → sort by sequence length → deterministic seed shuffle → yield samples, enabling cross-rank length consistency.
• Activation and tuning: enabled via --length-sort-size N (N > 0); guidance on pool_size selection, impact of num_workers, and memory considerations for stronger synchronization.
• Debugging and caveats: notes on pool internal state not being saved, behavior with multiple workers, and recommended checks for length distribution and step-time variance.

Quick Start

Enable LengthPoolSortDataset by setting --length-sort-pool-size to a positive value and monitor training step time variance for improvement.