nemo-curator

To clean a Common Crawl dataset for LLM training, you apply GPU-accelerated fuzzy MinHash/LSH and semantic deduplication alongside heuristic quality filters. This removes near-duplicates and low-quality text, outputting a curated Parquet/JSONL corpus ready for fine-tuning.

GPU-accelerated dataset curation uses RAPIDS, cuDF, and Dask to process large multimodal corpora at scale. You need it when cleaning web-scale text, images, video, or audio data where CPU-based processing becomes a bottleneck for deduplication and filtering.

Yes, GPU acceleration with RAPIDS, cuDF, and Dask support is required to run the fuzzy and semantic deduplication pipelines. These frameworks provide the necessary parallel processing power to handle exact, near-duplicate, and embedding-based deduplication at scale.

You redact PII and filter NSFW content by applying built-in safety controls during the curation pipeline. The system automatically detects and redacts sensitive identifying information while removing disallowed NSFW data using 30+ heuristic quality filters.

Yes, you can curate multimodal datasets containing images, video, and audio alongside text. The pipeline applies the same GPU-accelerated deduplication, quality filtering, and safety controls to clean assets and produce a unified training corpus.

The best way to remove semantically similar samples is using GPU-accelerated embedding-based deduplication methods. This approach identifies and filters semantic duplicates alongside exact and fuzzy matches, ensuring high-quality and diverse data for pretraining.