What problem does it solve? Preparing high-quality training data for LLMs from raw web scrapes requires deduplication, quality filtering, and PII redaction at terabyte scale, which is prohibitively slow and expensive on CPU-only pipelines. ## Core Features & Use Cases - GPU-Accelerated Deduplication: Exact, fuzzy (MinHash + LSH), and semantic deduplication running up to 16× faster than CPU, cutting an 8TB RedPajama v2 job from 120 hours to 7.5 hours. - Quality Filtering & PII Redaction: 30+ heuristic filters, classifier-based quality and NSFW scoring, and redaction of emails, phone numbers, names, and locations. - Multimodal Curation: Text, image (aesthetic/NSFW/CLIP), video (scene detection, clip extraction), and audio (ASR, WER filtering) pipelines with near-linear scaling across GPU clusters. - Use Case: Curate a Common Crawl dump by chaining word-count and URL-ratio filters, language identification, exact and fuzzy deduplication, and PII redaction, then export the result as Parquet for LLM pretraining. ## Quick Start Ask the agent to build a NeMo Curator pipeline that loads your Parquet dataset, applies quality filters, removes fuzzy duplicates, redacts PII, and writes the curated output to disk.
