hugging-face-datasets

Manage Hugging Face datasets with SQL via DuckDB and the hf:// protocol.

Updated Mar 12, 2026

Search Tags:#data management #sql #data transformation #duckdb #datasets #hugging face

Ready to run

npx skills add https://github.com/3kpro/aiarsenal --skill hugging-face-datasets-3kpro

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: hugging-face-datasets
Source: https://github.com/3kpro/aiarsenal/tree/main/hugging-face-skills/skills/hugging-face-datasets
Command: npx skills add https://github.com/3kpro/aiarsenal --skill hugging-face-datasets-3kpro

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires huggingface_hub, duckdb, datasets, pandas, and includes scripts (resource) and references (resource) and assets (resource) components.

What problem does it solve?

This Skill simplifies the creation, management, and querying of datasets on the Hugging Face Hub, enabling efficient data manipulation and transformation using SQL.

Core Features & Use Cases

SQL-Based Querying: Directly query any Hugging Face dataset using familiar SQL syntax via DuckDB.
Dataset Lifecycle Management: Initialize new datasets, configure them with system prompts, and stream row updates efficiently.
Data Transformation & Export: Filter, join, and transform datasets using SQL, then export results to local files or push to new Hub repositories.
Use Case: Analyze a large dataset on Hugging Face by writing SQL queries to filter specific data points, aggregate statistics, and then push a curated subset to a new, private repository for further analysis.

Quick Start

Query the 'cais/mmlu' dataset for nutrition-related entries using SQL.

Frequently Asked Questions about hugging-face-datasets

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I query Hugging Face datasets using SQL?▼

You can query Hugging Face datasets using SQL by leveraging DuckDB and the hf:// protocol. This allows you to directly filter, join, and transform dataset rows with familiar SQL syntax without downloading the entire dataset locally.

Can I manage the lifecycle of a Hugging Face dataset and push updates to a new repository?▼

Yes, you can manage the full lifecycle of a Hugging Face dataset by initializing new repositories, configuring system prompts, and streaming row updates. You can also export transformed data and push curated subsets to new Hub repositories for further analysis.

Does DuckDB support filtering and transforming large datasets hosted on Hugging Face?▼

Yes, DuckDB supports filtering and transforming large Hugging Face datasets via the hf:// protocol. It enables efficient data manipulation by allowing you to write SQL queries to aggregate statistics and extract specific data points directly from the Hub.

What's the best way to analyze and export a subset of a Hugging Face dataset?▼

The best way to analyze and export a Hugging Face dataset subset is by using SQL queries via DuckDB to filter specific data points. You can then export the transformed results to local files or push them to a new private repository on the Hub.

Do I need pandas to transform data extracted from Hugging Face datasets?▼

While pandas is supported as a dependency for data manipulation, this approach primarily relies on DuckDB and SQL syntax to perform data transformation. You can filter, join, and aggregate datasets directly using SQL before exporting.

Are there limitations when using SQL for data transformation on Hugging Face datasets?▼

Data transformation is constrained by DuckDB's SQL capabilities and the hf:// protocol's streaming efficiency. Complex transformations are supported, but performance depends on the dataset size and the specific SQL operations applied during querying.