nemo-curator

Curate large-scale LLM training datasets with GPU-accelerated deduplication, filtering, and PII redaction.

Updated May 4, 2026

Search Tags:#deduplication #data curation #pii redaction #gpu acceleration #llm training #nemo-curator

Ready to run

npx skills add https://github.com/InverterNetwork/hermes-agent --skill nemo-curator-inverternetwork

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: nemo-curator
Source: https://github.com/InverterNetwork/hermes-agent/tree/main/optional-skills/mlops/nemo-curator
Command: npx skills add https://github.com/InverterNetwork/hermes-agent --skill nemo-curator-inverternetwork

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires nemo-curator, cudf, dask, rapids, and includes references (resource) components.

What problem does it solve?

This Skill addresses the high computational cost and complexity of preparing massive datasets for LLM training by providing a GPU-accelerated pipeline for data cleaning and refinement.

Core Features & Use Cases

High-Performance Curation: Execute deduplication, PII redaction, and quality filtering up to 16 times faster than CPU-based alternatives.
Multi-modal Support: Process text, image, video, and audio datasets within a unified Ray-based pipeline architecture.
Use Case: Use this Skill to process terabytes of web-scraped data from Common Crawl, ensuring high-quality training inputs by removing toxic content, duplicates, and PII before model fine-tuning.

Quick Start

Use the nemo-curator skill to initialize a GPU-accelerated pipeline that filters and deduplicates your raw dataset parquet files.

Frequently Asked Questions about nemo-curator

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I deduplicate and clean large-scale web-scraped datasets for LLM training?▼

This approach uses a Ray-based pipeline architecture with RAPIDS libraries to execute deduplication, quality filtering, and PII redaction on raw dataset parquet files, achieving near-linear scaling across GPU clusters.

What is GPU-accelerated data curation and when do I need it for machine learning models?▼

GPU-accelerated data curation uses RAPIDS libraries to clean and refine massive datasets for machine learning models. It is needed when preparing terabytes of multi-modal data like text, images, video, and audio to ensure high-quality training inputs.

Does the nemo-curator pipeline support multi-modal data formats like images, video, and audio?▼

Yes, the nemo-curator pipeline supports multi-modal data formats including text, images, video, and audio. It processes these diverse data modalities within a unified Ray-based pipeline architecture to prepare high-quality inputs for machine learning models.

Can I use dask and RAPIDS to achieve near-linear scaling for PII redaction across GPU clusters?▼

Yes, the pipeline utilizes dask and RAPIDS libraries within a Ray-based architecture to achieve near-linear scaling across GPU clusters for PII redaction and other data curation tasks.

What is the best way to remove toxic content and duplicates from Common Crawl data?▼

The best way is to use a GPU-accelerated curation pipeline that executes quality filtering and deduplication on raw dataset parquet files, removing toxic content and duplicates before model fine-tuning.

Do I need a GPU cluster to run data curation pipelines for terabyte-scale datasets?▼

You do not strictly need a GPU cluster, but utilizing one with RAPIDS libraries provides near-linear scaling and processes terabyte-scale datasets up to 16 times faster than CPU-based alternatives, significantly reducing the high computational cost of data preparation.