What problem does it solve? Preparing high-quality training data from large web scrapes or raw corpora is slow and expensive on CPUs, and low-quality, duplicated, or sensitive content degrades LLM training results. ## Core Features & Use Cases - GPU-Accelerated Deduplication: Exact, fuzzy (MinHash + LSH), and semantic deduplication running up to 16× faster than CPU on multi-terabyte datasets. - Quality Filtering & Classifiers: 30+ heuristic filters plus GPU classifiers for quality scoring, NSFW detection, and language identification. - PII Redaction & Multimodal Support: Redact emails, phone numbers, and names, and curate text, image, video, and audio datasets with RAPIDS scaling across GPU clusters. - Use Case: Curate a Common Crawl dump by filtering short or repetitive documents, removing near-duplicates with fuzzy dedup, redacting PII, and exporting clean Parquet files for LLM pretraining. ## Quick Start Use the nemo-curator skill to deduplicate and quality-filter my training corpus in the attached Parquet files and save the curated output.
