hugging-face-datasets

Manage Hugging Face Hub datasets for LLM training by automating repo initialization, configuring system prompts, streaming rows, and pushing data subsets to the Hub. It handles the end-to-end dataset lifecycle across SFT, RLHF, and DPO pipelines.

Query and transform Hugging Face datasets using DuckDB-backed SQL. Apply SQL operations to filter and manipulate dataset rows directly, then export the transformed subsets back to the Hugging Face Hub.

A valid HF_TOKEN with write access is required to create, manage, and push datasets to the Hugging Face Hub. The environment also requires uv installed and optionally supports the HF MCP server.

Stream new rows into a Hugging Face dataset by executing the provided scripts within the workflow. This enables iterative data improvements and row streaming directly into the configured repository.

Dataset configuration supports SFT, RLHF, and DPO pipelines by initializing repos, setting system prompts, and managing metadata. Transformations apply SQL to prepare and export subsets for specific training workflows.

DuckDB provides SQL-based querying and transformation capabilities over Hugging Face datasets. It allows efficient filtering and manipulation of data before pushing the resulting subsets to the Hub.