distributed-offline-packing
OfficialDistribute and pack large SFT data across nodes.
AuthorEvolvingLMMs-Lab
Version1.0.0
Installs0
System Documentation
What problem does it solve?
Use this skill to orchestrate the multi-node packing of enormous SFT JSONL datasets into Energon WebDataset shards, enabling scalable SFT data preparation with padding-free shards and a consolidated metadataset.
Core Features & Use Cases
- End-to-end distributed packing across multiple nodes using offline_packing/auto_pipe.sh to partition work, tokenize/prompts, bin samples, and generate webdataset shards.
- Per-node Metadataset assembly that aggregates node outputs into a single logical dataset, suitable for large-scale SFT pipelines.
- Prerequisites and guardrails including shared NFS, consistent Docker image, and script stages s1–s4 to ensure deterministic reproducibility.
Quick Start
Prepare an N-node environment with a shared NFS, mount the repository, and run offline_packing/auto_pipe.sh on your Part JSONL to produce Energon WebDataset shards.
Dependency Matrix
Required Modules
None requiredComponents
Standard package💻 Claude Code Installation
Recommended: Let Claude install automatically. Simply copy and paste the text below to Claude Code.
Please help me install this Skill: Name: distributed-offline-packing Download link: https://github.com/EvolvingLMMs-Lab/LLaVA-OneVision-2/archive/main.zip#distributed-offline-packing Please download this .zip file, extract it, and install it in the .claude/skills/ directory.
Agent Skills Search Helper
Install a tiny helper to your Agent, search and equip skill from 471,000+ vetted skills library on demand.