nemo-curator

Filter, deduplicate, and redact web-scale datasets for LLM training.

Updated May 4, 2026

Search Tags:#deduplication #data curation #pii redaction #gpu acceleration #nemo curator #llm training data

Ready to run

npx skills add https://github.com/JamesFincher/gengar --skill nemo-curator-jamesfincher

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: nemo-curator
Source: https://github.com/JamesFincher/gengar/tree/main/optional-skills/mlops/nemo-curator
Command: npx skills add https://github.com/JamesFincher/gengar --skill nemo-curator-jamesfincher

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires nemo-curator, cudf, dask, rapids, and includes references (resource) components.

What problem does it solve?

NeMo Curator removes low-quality, duplicate, and sensitive content from large web-derived datasets so your LLM training data is cleaner and more useful.

Core Features & Use Cases

Quality filtering with heuristics and classifiers: Uses 30+ quality heuristics and optional ML classifiers to filter out short, noisy, repetitive, or suspicious documents.
GPU-accelerated deduplication (exact, fuzzy, semantic): Eliminates identical, near-duplicate, and semantically similar examples using GPU-friendly methods like MinHash+LSH and embedding similarity.
Multimodal support and safety hygiene: Handles text plus images/video/audio pipelines, including PII redaction and NSFW detection for safer training corpora.

Quick Start

Use the nemo-curator skill to curate a raw dataset and write a deduplicated, filtered Parquet output you can feed into your next LLM training run.

Frequently Asked Questions about nemo-curator

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I clean and deduplicate web-scale datasets for LLM training?▼

To clean and deduplicate LLM training data, you can apply heuristic quality filters, GPU-accelerated exact and fuzzy matching, and PII redaction to raw web corpora, outputting a filtered Parquet dataset ready for model training.

How does GPU-accelerated deduplication work for large text corpora?▼

GPU-accelerated deduplication processes large text corpora using RAPIDS, cuDF, and Dask to distribute MinHash+LSH and embedding similarity computations across GPUs, rapidly eliminating exact, near-duplicate, and semantically similar documents.

Do I need a GPU to run NeMo Curator for dataset preparation?▼

Yes, NeMo Curator requires a GPU-capable execution environment to leverage NVIDIA RAPIDS and Dask modules for distributed processing, ensuring sufficient throughput for web-scale dataset filtering and deduplication tasks.

What is the best way to remove PII and NSFW content from multimodal training data?▼

The best way to remove PII and NSFW content from multimodal training data is to apply automated redaction and detection pipelines that process text, images, video, and audio records to ensure safety hygiene before model training.

Can I use Dask and RAPIDS for distributed data curation on Common Crawl?▼

Yes, you can use Dask and RAPIDS to distribute data curation workflows on Common Crawl, enabling GPU-accelerated filtering, deduplication, and sensitive content redaction across large-scale multimodal records.

What are the limitations of using heuristic classifiers for data quality filtering?▼

Heuristic classifiers filter short, noisy, and repetitive documents using rule-based scores, but they may miss nuanced semantic quality issues that ML-based classifiers catch, requiring combined approaches for optimal corpus preparation.