skills.rest
dongzhuoyao
dongzhuoyaoCOMMUNITY
Explore all skills by this author →
Website𝕏@vtaohuGitHub

hf-dataset-management

Manage HuggingFace dataset caching, uploads, and preflight verification for ML training.

10|1|Updated Feb 21, 2026
Search Tags:#caching#preflight#datasets#huggingface#hf-hub#webdataset
One-click install
npx skills add https://github.com/dongzhuoyao/tao-research-skills --skill hf-dataset-management
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: hf-dataset-management
Source: https://github.com/dongzhuoyao/tao-research-skills/tree/main/hf-dataset-management
Command: npx skills add https://github.com/dongzhuoyao/tao-research-skills --skill hf-dataset-management

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Ensures datasets are available, consistent, and verified for ML training by providing an offline-first caching workflow, preflight checks to catch missing or malformed data before expensive jobs, and guidance for uploading and validating datasets on the HuggingFace Hub.

Core Features & Use Cases

  • Offline-first caching: Pre-cache datasets on login nodes and configure offline environment variables to avoid runtime downloads on compute nodes.
  • Preflight verification: Validate cache directories, expected file formats, and counts to prevent failed Slurm jobs and wasted GPU time.
  • Reliable uploads and round-trip checks: Push datasets to HF Hub and immediately download to confirm integrity and completeness.
  • Scalable alternatives: Convert very large corpora to WebDataset shards and tune num_workers and shard counts for distributed training.
  • Use case: Pre-cache a 10 TB dataset on the cluster login node, run a preflight script to ensure expected parquet shards are present, then push and verify a smaller evaluation split to the HF Hub.

Quick Start

Pre-cache the target HuggingFace dataset to data/my_dataset, run a preflight integrity check, and then push and verify the dataset on the HF Hub before submitting the Slurm job.

Frequently Asked Questions about hf-dataset-management

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I pre-cache HuggingFace datasets for offline Slurm jobs?▼

Pre-cache HuggingFace datasets by downloading them on cluster login nodes and setting HF_DATASETS_OFFLINE and HF_HUB_OFFLINE environment variables. This ensures compute nodes access data directly from the local cache_dir without runtime downloads.

What is a HuggingFace dataset preflight check before distributed training?▼

A HuggingFace dataset preflight check validates cache directories, expected file formats, and shard counts before expensive Slurm jobs. This verification catches missing or malformed data early, preventing failed runs and wasted GPU time.

How do I verify HuggingFace dataset uploads with a round-trip check?▼

Verify HuggingFace dataset uploads by pushing the data to the HF Hub and immediately downloading it back. This round-trip check confirms the uploaded dataset's integrity and completeness, ensuring the remote repository matches local files.

When should I convert HuggingFace datasets to WebDataset shards?▼

Convert HuggingFace datasets to WebDataset shards when scaling up to very large corpora for distributed training. This approach improves performance by allowing you to tune num_workers and shard counts, optimizing data loading across multiple compute nodes.

Why are my HuggingFace offline training jobs failing to find cached data?▼

HuggingFace offline training jobs fail when cache_dir conventions are misconfigured or preflight checks are skipped. Ensure the dataset is fully pre-cached on the login node and that expected file formats are present before submitting jobs.

Related Skills

Explore More Education & Research Skills→
plantcadplantcad
38

tacc

Automate experiment execution on the TACC Vista HPC cluster.

Official
Advanced
ovachieverovachiever
52

pytorch-fsdp

Apply PyTorch FSDP for sharded, mixed-precision distributed training of large models.

Community
Advanced
qdrantqdrant
220

qdrant-scaling-query-volume

Optimize Qdrant query volume by applying Poisson distribution to shard limits.

Official
Intermediate
jayll1303jayll1303
17

hf-hub-datasets

Download and upload HuggingFace Hub models and datasets with authentication.

Community
Intermediate