nemo-curator

Curate multi-modal LLM training data with GPU-accelerated deduplication and filtering.

2|Updated Apr 25, 2026

Search Tags:#multimodal #deduplication #gpu-acceleration #pii-redaction #data-curation #nsfw-detection #nemo-curator

Ready to run

npx skills add https://github.com/AlexiosBluffMara/mercury --skill nemo-curator-alexiosbluffmara

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: nemo-curator
Source: https://github.com/AlexiosBluffMara/mercury/tree/main/optional-skills/mlops/nemo-curator
Command: npx skills add https://github.com/AlexiosBluffMara/mercury --skill nemo-curator-alexiosbluffmara

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires nemo-curator, cudf, dask, rapids, and includes references (resource) components.

What problem does it solve?

Curates high-quality training data for LLMs using GPU-accelerated data curation workflows. Applies to multi-modal datasets (text, images, video, audio) and large web corpora, enabling deduplication, quality filtering, PII redaction, and NSFW detection at scale.

Core Features & Use Cases

GPU-accelerated multi-modal data curation (text, image, video, audio) with deduplication, quality filtering, PII redaction, and NSFW detection.
Scales across RAPIDS-enabled GPUs for large corpora and web-scraped data, enabling end-to-end data preparation, cleaning, and deduplication.
Suitable for building high-quality training datasets for Nemotron/NVIDIA-style models and other large language models.

Quick Start

Install NeMo Curator and run a sample end-to-end data-curation workflow on your dataset.

Frequently Asked Questions about nemo-curator

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I deduplicate and clean large web corpora for LLM training data?▼

Deduplicating and cleaning large web corpora for LLM training data is achieved through GPU-accelerated data curation workflows. This approach applies quality filtering, PII redaction, and NSFW detection at scale across multi-modal datasets.

What is GPU-accelerated data curation for multi-modal datasets?▼

GPU-accelerated data curation for multi-modal datasets is an end-to-end data preparation process that scales across RAPIDS-enabled GPUs. It enables deduplication, quality filtering, PII redaction, and NSFW detection for text, images, video, and audio.

Do I need RAPIDS and Dask to run GPU-accelerated data curation workflows?▼

Yes, you need RAPIDS and Dask to run these GPU-accelerated data curation workflows. The pipelines require RAPIDS-enabled GPUs and Nemo Curator modules to deliver end-to-end data preparation, cleaning, and deduplication at scale.

Can I use Nemo Curator for PII redaction and NSFW detection on text data?▼

Yes, you can use Nemo Curator for PII redaction and NSFW detection on text data. It applies these quality filtering steps natively within its GPU-accelerated multi-modal data curation pipelines for large web corpora.

What's the best way to scale data preparation across large web-scraped datasets?▼

The best way to scale data preparation across large web-scraped datasets is using RAPIDS-enabled GPUs. This distributes the end-to-end data cleaning, deduplication, and quality filtering workloads required for high-quality LLM training data.

When should I not use GPU-accelerated pipelines for multi-modal data curation?▼

You should not use GPU-accelerated pipelines for multi-modal data curation if your environment lacks RAPIDS-enabled GPUs. The end-to-end data preparation, cleaning, and deduplication workflows specifically require this hardware to scale across large corpora.