Data Loader Throughput + Sequence Packing

To optimize dataloader throughput and eliminate padding waste, apply sequence packing and bucketing strategies. This Skill audits data loading and GPU compute phases to maximize effective tokens per second in distributed training.

Sequence packing combines multiple training examples into fixed-length blocks to avoid wasted compute from padding. You need it for pretraining blocks and SFT boundary-aware packing to maximize dataloader throughput at scale.

You benchmark dataloader throughput by configuring a synthetic dataset and running the six-phase pipeline. This instruments data loading and compute phases to measure throughput gains across HF streaming, WebDataset tar shards, and memmap backends.

Yes, it supports deterministic per-rank sharding for distributed training environments. It applies deterministic sharding alongside caching and bucketing strategies to satisfy production-grade requirements for large-scale training workflows.

Yes, numpy is required to run the dataloader throughput and sequence packing pipeline. It serves as the sole dependency for executing the scripts, processing assets, and utilizing the provided references.

Data stalls occur when the dataloader cannot supply data fast enough to keep the GPU compute phase saturated. Instrumenting the data loading and compute phases identifies these bottlenecks and guides configuration changes to improve throughput.