skills.rest
Chensihakniroth
ChensihaknirothCOMMUNITY
Explore all skills by this author →
GitHub

nemo-curator

Curate LLM training datasets with GPU-accelerated deduplication, filtering, and PII redaction.

Updated Jun 7, 2026
Search Tags:#GPU acceleration#deduplication#quality filtering#PII redaction#data curation#LLM training data#Nemo Curator
One-click install
npx skills add https://github.com/Chensihakniroth/ANAKOT-AGENT --skill nemo-curator-chensihakniroth
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: nemo-curator
Source: https://github.com/Chensihakniroth/ANAKOT-AGENT/tree/main/optional-skills/mlops/nemo-curator
Command: npx skills add https://github.com/Chensihakniroth/ANAKOT-AGENT --skill nemo-curator-chensihakniroth

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires nemo-curator, cudf, dask, rapids, and includes references (resource) components.

What problem does it solve? Preparing high-quality training data for LLMs from raw web scrapes or large corpora is slow and expensive on CPUs, and low-quality, duplicated, or privacy-sensitive content degrades model performance. ## Core Features & Use Cases - GPU-Accelerated Deduplication: Exact, fuzzy (MinHash + LSH), and semantic deduplication running up to 16× faster than CPU alternatives using RAPIDS and Dask. - Quality Filtering & Classifiers: 30+ heuristic filters (word count, repeated lines, URL ratio) plus GPU classifiers for quality scoring and NSFW detection. - PII Redaction & Multimodal Support: Redact emails, phone numbers, and names, and curate text, image, video, and audio datasets. - Use Case: You scraped terabytes of Common Crawl data for pretraining. Use this Skill to filter low-quality documents, remove near-duplicates with fuzzy deduplication, redact PII, and export a clean Parquet dataset across a multi-GPU cluster. ## Quick Start Use the nemo-curator skill to deduplicate and quality-filter my Common Crawl parquet files into a curated training dataset.

Frequently Asked Questions about nemo-curator

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I deduplicate LLM training data with NeMo Curator?▼

NeMo Curator deduplicates training data using ExactDuplicates for identical documents, FuzzyDuplicates with MinHash and LSH for near-duplicates, and SemanticDuplicates with embedding models for paraphrases. Fuzzy deduplication on GPUs processes 8TB in about 7.5 hours.

What is the difference between fuzzy and semantic deduplication?▼

Fuzzy deduplication uses MinHash and LSH to detect near-identical text quickly with about 95% recall, while semantic deduplication uses sentence embeddings to catch paraphrases and rewrites. Fuzzy is recommended for large datasets; semantic is slower but more thorough.

NeMo Curator vs datatrove for data curation?▼

NeMo Curator is GPU-accelerated with RAPIDS, delivering up to 16× faster fuzzy deduplication and multimodal support for images, video, and audio. Datatrove is CPU-based and open-source, making it a better fit when GPUs are unavailable.

Does NeMo Curator support image and video datasets?▼

Yes, NeMo Curator supports multimodal curation including image aesthetic scoring, NSFW detection, and CLIP embeddings, plus video scene detection, clip extraction, and InternVideo2 embeddings. Audio curation includes ASR transcription and WER filtering.

Can NeMo Curator run without a GPU?▼

Yes, install the CPU-only extra with uv pip install "nemo-curator[cpu]", but processing is significantly slower. GPU acceleration with CUDA 12 provides 10-16× speedups for deduplication and filtering workloads.

What data formats does NeMo Curator support?▼

NeMo Curator reads Parquet, JSONL, and CSV inputs, and outputs Parquet (recommended) or JSONL. It also supports WebDataset TAR archives for multimodal data such as images and video.

Related Skills

Explore More Data & Analytics Skills→
xu1713xu1713

nemo-curator

Curate and deduplicate LLM training datasets on GPUs with NeMo Curator.

Community
Intermediate
atrawogatrawog

finetuning

Fine-tune domain-specific LLMs with PyTorch and HuggingFace Trainer.

Community
Advanced
davpatel605-beepdavpatel605-beep

nemo-curator

Clean and deduplicate large-scale datasets for LLM training.

Community
Advanced
Orchestra-ResearchOrchestra-Research
13.0k

ray-data

Process large ML datasets with distributed streaming execution across CPU and GPU clusters.

Official
Intermediate