What problem does it solve? Preparing high-quality training data for LLMs from raw web scrapes and multi-modal sources is slow and expensive on CPU, and low-quality, duplicated, or privacy-sensitive content degrades model training. ## Core Features & Use Cases - Deduplication at scale: Exact, fuzzy (MinHash + LSH), and semantic deduplication running up to 16× faster on GPU, e.g., 8TB RedPajama v2 in 7.5 hours instead of 120. - Quality filtering and classification: 30+ heuristic filters (word count, repeated lines, URL ratio) plus GPU classifiers for quality and NSFW content. - PII redaction and multi-modal curation: Redact emails, phone numbers, and names from text; curate image, video, and audio datasets with aesthetic scoring, scene detection, and ASR-based filtering. - Use Case: Curate a Common Crawl dump by chaining language identification, heuristic filters, exact and fuzzy deduplication, and PII redaction, then write the result to Parquet. ## Quick Start Ask the agent to build a NeMo Curator pipeline that loads your Parquet dataset, applies quality filters, removes fuzzy duplicates, redacts PII, and saves the curated output.
