nemo-curator

You can deduplicate large-scale Common Crawl datasets by applying GPU-accelerated exact, fuzzy (MinHash+LSH), and semantic deduplication to remove identical and near-duplicate content at multi-terabyte scale. This process outputs a cleaned Parquet corpus ready for model training.

GPU-accelerated data curation cleans, filters, and deduplicates massive text, image, video, and audio corpora. It uses RAPIDS and Dask for distributed execution, applying modality-specific filters and embedders to process high-quality multimodal datasets efficiently.

Yes, you need a GPU-capable RAPIDS stack and Dask for distributed execution to run NeMo Curator pipelines. These dependencies provide the near-linear scaling across GPU clusters required to process multi-terabyte web-scraped datasets.

Yes, you can redact PII and detect NSFW content during data quality filtering. The pipeline includes PII anonymization and GPU classifiers for quality, language, and NSFW detection to remove low-value or harmful data from your corpora.

The best way to remove near-duplicate text from web scrapes is using fuzzy (MinHash+LSH) and semantic deduplication accelerated on GPUs. This approach identifies and eliminates redundant content across multi-terabyte datasets to prevent model performance degradation.

There are over 30 heuristic filters available for quality filtering web-scraped text. These filters work alongside GPU classifiers to evaluate document quality and language, ensuring low-value data is removed from your training datasets.