distributed-offline-packing

You can pack large SFT JSONL datasets into Energon WebDataset shards across multiple nodes by partitioning the data, computing token lengths, binning samples, and writing tar shards with a final aggregated metadataset.

To avoid filename collisions when generating WebDataset shards in a distributed pipeline, configure shard-prefixing for each node so that per-node outputs remain unique before aggregation into the final metadataset.

Yes, distributing SFT data packing across nodes requires a shared NFS to mount the repository and ensure all nodes can access the necessary scripts and data slices consistently.

The tokenization and binning process for SFT data packing involves computing token lengths for each JSONL sample and grouping them into bins to create padding-free WebDataset shards.

The end-to-end distributed packing of SFT data uses script stages s1 through s4, which handle partitioning, tokenizing, binning, and generating WebDataset shards with deterministic reproducibility.

Multi-node SFT data packing requires a consistent Docker image across all nodes to ensure deterministic reproducibility when running the offline packing pipeline and generating Energon WebDataset shards.