hf-hub-datasets

What problem does it solve?

Eliminates friction when transferring, accessing, and managing models and datasets on the HuggingFace Hub by providing clear patterns for authentication, partial downloads, uploads, and large-dataset workflows so teams can move artifacts reliably between local, CI, and Hub storage.

Core Features & Use Cases

• Authentication & Access Management: Login patterns, HF_TOKEN usage, gated model access, and org/private repo handling.
• Download Patterns: Full snapshots, partial downloads using allow_patterns/ignore_patterns, revision selection, and single-file retrieval.
• Upload & Repo Management: Create repos, upload_folder/upload_file/push_to_hub, model card generation, and visibility settings.
• Large Dataset Handling: Streaming, memory-mapped loading, iterable datasets, filtering/mapping at scale, caching, and Git LFS guidance.
• Use Case: Download a specific model revision with snapshot_download, stream a multi-gigabyte dataset for single-pass preprocessing, then push a processed dataset back to the Hub with push_to_hub.

Quick Start

Use the hf-hub-datasets skill to download a model snapshot and stream a large dataset for on-the-fly processing.