Verified:GitHub

nemo-curator

Curate and deduplicate LLM training datasets on GPUs with NeMo Curator.

Updated Apr 18, 2026

Search Tags:#deduplication #quality filtering #data curation #pii redaction #gpu acceleration #nemo curator #llm training data

Ready to run

npx skills add https://github.com/azaanaliraza/operarius --skill nemo-curator-azaanaliraza

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: nemo-curator
Source: https://github.com/azaanaliraza/operarius/tree/main/src-tauri/bin/hermes/optional-skills/mlops/nemo-curator
Command: npx skills add https://github.com/azaanaliraza/operarius --skill nemo-curator-azaanaliraza

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires nemo-curator, cudf, dask, rapids, and includes references (resource) components.

What problem does it solve? Preparing high-quality training data from massive raw corpora like Common Crawl is slow and expensive on CPUs, and low-quality, duplicated, or unsafe content degrades LLM training outcomes. ## Core Features & Use Cases - GPU-Accelerated Deduplication: Exact, fuzzy (MinHash + LSH), and semantic deduplication running up to 16× faster than CPU pipelines. - Quality Filtering & Safety: 30+ heuristic filters, quality classifiers, NSFW detection, and PII redaction for text, image, video, and audio datasets. - Multi-GPU Scaling: Distributed processing across GPU clusters using RAPIDS, cuDF, and Dask for terabyte-scale corpora. - Use Case: Curate an 8TB Common Crawl scrape by filtering low-quality documents, removing near-duplicates, redacting PII, and exporting clean Parquet files for LLM pretraining. ## Quick Start Ask the AI to build a NeMo Curator pipeline that filters, deduplicates, and redacts PII from a Parquet dataset of web-scraped text.

Frequently Asked Questions about nemo-curator

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I deduplicate a large text dataset for LLM training?▼

Use NeMo Curator's FuzzyDuplicates module, which applies MinHash and LSH to remove near-duplicates. On GPUs it processes 8TB in about 7.5 hours, roughly 16× faster than CPU-based deduplication.

What is the difference between exact, fuzzy, and semantic deduplication?▼

Exact deduplication removes identical documents via hashing, fuzzy deduplication catches near-duplicates using MinHash and LSH, and semantic deduplication uses embedding similarity to detect paraphrases. Fuzzy dedup offers the best speed-to-recall trade-off for large corpora.

Does NeMo Curator work without a GPU?▼

Yes, a CPU-only installation is available via pip install nemo-curator[cpu], but it runs significantly slower. GPU acceleration with RAPIDS delivers 10-16× speedups on deduplication and filtering workloads.

Can NeMo Curator process image and video datasets?▼

Yes, it supports multimodal curation including aesthetic scoring and NSFW filtering for images, scene detection and clip extraction for video, and ASR transcription with WER filtering for audio.

NeMo Curator vs datatrove for data curation?▼

NeMo Curator is GPU-accelerated and optimized for large-scale deduplication and multimodal curation, while datatrove is CPU-based and lighter weight. Choose NeMo Curator when processing terabyte-scale corpora with available GPU infrastructure.