add-dataset

Inspect Hugging Face dataset schemas and register them as Marin ExecutorSteps.

1.2k|146|Updated Mar 22, 2024

Search Tags:#data pipeline #foundation models #dataset #hugging face #schema inspection #data registration

Ready to run

npx skills add https://github.com/marin-community/marin --skill add-dataset-marin-community

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: add-dataset
Source: https://github.com/marin-community/marin/tree/main/.agents/skills/add-dataset
Command: npx skills add https://github.com/marin-community/marin --skill add-dataset-marin-community

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill streamlines the process of integrating new Hugging Face datasets into Marin's data pipelines, enabling efficient training of foundation models.

Core Features & Use Cases

Schema Inspection: Automatically inspects Hugging Face dataset schemas to identify text fields and available splits.
Dataset Registration: Facilitates the registration of datasets as ExecutorSteps within Marin, making them available for download and processing.
Use Case: When starting a new pre-training experiment, use this skill to quickly inspect a dataset like 'wikitext' and register it for subsequent tokenization and training.

Quick Start

Use the add-dataset skill to inspect the Hugging Face dataset named 'roneneldan/TinyStories'.

Frequently Asked Questions about add-dataset

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I add a Hugging Face dataset to Marin for foundation model training?▼

To add a Hugging Face dataset to Marin, inspect its schema to identify text fields and splits, then register it as an ExecutorStep for downstream pipeline processing and training.

Can I register Hugging Face datasets with multiple configurations in Marin?▼

Yes, you can register Hugging Face datasets with multiple configurations in Marin. The skill handles these complex schemas during inspection to ensure proper text field identification for data pipelines.

How do I inspect a Hugging Face dataset schema to find text fields for pre-training?▼

Inspecting a Hugging Face dataset schema involves automatically scanning the dataset structure to identify available data splits and the specific text fields needed for foundation model pre-training.

Does adding Hugging Face datasets to Marin work with datasets requiring remote code execution?▼

Yes, adding Hugging Face datasets to Marin works with datasets requiring remote code execution. The skill specifically handles these datasets during schema inspection and registration.

What is the process for integrating new Hugging Face datasets into data pipelines?▼

Integrating new Hugging Face datasets into data pipelines requires schema inspection to find text fields, followed by dataset registration as an ExecutorStep to make the data available for download and processing.