hf-dataset-management

What problem does it solve?

Ensures datasets are available, consistent, and verified for ML training by providing an offline-first caching workflow, preflight checks to catch missing or malformed data before expensive jobs, and guidance for uploading and validating datasets on the HuggingFace Hub.

Core Features & Use Cases

• Offline-first caching: Pre-cache datasets on login nodes and configure offline environment variables to avoid runtime downloads on compute nodes.
• Preflight verification: Validate cache directories, expected file formats, and counts to prevent failed Slurm jobs and wasted GPU time.
• Reliable uploads and round-trip checks: Push datasets to HF Hub and immediately download to confirm integrity and completeness.
• Scalable alternatives: Convert very large corpora to WebDataset shards and tune num_workers and shard counts for distributed training.
• Use case: Pre-cache a 10 TB dataset on the cluster login node, run a preflight script to ensure expected parquet shards are present, then push and verify a smaller evaluation split to the HF Hub.

Quick Start

Pre-cache the target HuggingFace dataset to data/my_dataset, run a preflight integrity check, and then push and verify the dataset on the HF Hub before submitting the Slurm job.