add-dataset

To add a Hugging Face dataset to Marin, inspect its schema to identify text fields and splits, then register it as an ExecutorStep for downstream pipeline processing and training.

Yes, you can register Hugging Face datasets with multiple configurations in Marin. The skill handles these complex schemas during inspection to ensure proper text field identification for data pipelines.

Inspecting a Hugging Face dataset schema involves automatically scanning the dataset structure to identify available data splits and the specific text fields needed for foundation model pre-training.

Yes, adding Hugging Face datasets to Marin works with datasets requiring remote code execution. The skill specifically handles these datasets during schema inspection and registration.

Integrating new Hugging Face datasets into data pipelines requires schema inspection to find text fields, followed by dataset registration as an ExecutorStep to make the data available for download and processing.