Scaffold and register AReaL dataset loaders for SFT and RL workflows.
npx skills add https://github.com/inclusionAI/AReaL --skill add-dataset
Please help me install this Agent Skill. Skill: add-dataset Source: https://github.com/inclusionAI/AReaL/tree/main/.claude/skills/add-dataset Command: npx skills add https://github.com/inclusionAI/AReaL --skill add-dataset
This Skill helps data engineers and ML engineers add new dataset loaders to AReaL, standardizing the process of integrating diverse data sources for SFT and RL workflows.
Use the add-dataset skill to scaffold a new dataset loader and register it in the areal project.
High-intent search queries and answers about installing and using this skill.
To add a new dataset loader to AReaL, you create a loader file using provided code templates, define the SFT or RL dataset structures, and register it in areal/dataset/__init__.py. This standardizes integrating diverse data sources.
SFT and RL dataset structures require specific fields and processing steps to ensure the loader outputs HuggingFace Datasets with the expected schema. The skill provides map and filter column expectations to validate these required fields.
Yes, AReaL supports integrating HuggingFace datasets by guiding the creation of custom loaders. These loaders define map and filter operations to ensure data conforms to the expected column schema for SFT and RL.
You test a custom dataset loader using provided testing scaffolds. These scaffolds validate that the loader correctly processes data and produces HuggingFace Datasets matching the expected SFT or RL schema.
The best way to register a Python dataset loader is to add it to the project's dataset module initialization file. This skill provides registration snippets to update areal/dataset/__init__.py, ensuring the loader is recognized.
Updating areal/dataset/__init__.py is required to formally register the new dataset loader within the AReaL framework, making it accessible for SFT and RL workflows and ensuring proper module integration.
zli5460Develop and test Hermes RL environments with custom dataset loading and evaluation.
afel6Fine-tune LLMs with TRL for SFT, DPO, PPO/GRPO, and reward-model training.
thinking-machines-labConstruct and normalize ML datasets across JSONL, HuggingFace, and conversation formats.
cmatKhanInspect and audit HuggingFace dataset repositories via the labretriever DataCard API.
