length-pool-sort-dataset

Cross-rank length synchronization reduces multi-GPU training idle time by aligning sample lengths across ranks. LengthPoolSortDataset accumulates a pool of samples, sorts them by sequence length, and applies a deterministic shuffle to yield batches, improving effective training throughput.

Enable length sorting by setting the --length-sort-pool-size argument to a positive value. This activates the LengthPoolSortDataset, which synchronizes cross-rank lengths to speed up multi-GPU training throughput.

Tune pool size by monitoring training step time variance for improvement after setting --length-sort-pool-size. Consider memory limitations and the impact of num_workers, as a larger pool size provides stronger length synchronization but requires more memory.

LengthPoolSortDataset works with multiple workers but the pool's internal state is not saved during checkpoint resume. You should perform debugging checks on length distribution and step-time variance to ensure consistent behavior across workers.

Step time fluctuates because uneven sequence lengths cause some GPUs to wait for others. LengthPoolSortDataset addresses this by synchronizing cross-rank lengths, accumulating and sorting samples to align processing times and reduce idle variance.