Data Loader Throughput + Sequence Packing
CommunityMaximize dataloader throughput at scale
Software Engineering#benchmarking#packaging#sharding#throughput#dataloader#bucketing#sequence-packing
Authorsovr610
Version1.0.0
Installs0
System Documentation
What problem does it solve?
DataLoader Throughput + Sequence Packing provides a structured approach to measuring and optimizing the end-to-end data pipeline for large-scale training, identifying data stalls, and eliminating wasted compute from padding.
Core Features & Use Cases
- Audits and improves input data throughput by instrumenting the data loading and GPU compute phases.
- Supports deterministic per-rank sharding, streaming and memmap backends, and bucketing/padding strategies to maximize effective tokens per second.
- Offers utilities for sequence packing (pretraining blocks and SFT boundary-aware packing) and integrated metrics reporting to guide configuration.
Quick Start
Configure a synthetic dataset and run the six-phase pipeline to observe throughput gains and iterate on packing and sharding settings.
Dependency Matrix
Required Modules
numpy
Components
scriptsreferencesassets
💻 Claude Code Installation
Recommended: Let Claude install automatically. Simply copy and paste the text below to Claude Code.
Please help me install this Skill: Name: Data Loader Throughput + Sequence Packing Download link: https://github.com/sovr610/refffiy/archive/main.zip#data-loader-throughput-sequence-packing Please download this .zip file, extract it, and install it in the .claude/skills/ directory.
Agent Skills Search Helper
Install a tiny helper to your Agent, search and equip skill from 471,000+ vetted skills library on demand.