nemo-curator

Deduplicating and cleaning large web corpora for LLM training data is achieved through GPU-accelerated data curation workflows. This approach applies quality filtering, PII redaction, and NSFW detection at scale across multi-modal datasets.

GPU-accelerated data curation for multi-modal datasets is an end-to-end data preparation process that scales across RAPIDS-enabled GPUs. It enables deduplication, quality filtering, PII redaction, and NSFW detection for text, images, video, and audio.

Yes, you need RAPIDS and Dask to run these GPU-accelerated data curation workflows. The pipelines require RAPIDS-enabled GPUs and Nemo Curator modules to deliver end-to-end data preparation, cleaning, and deduplication at scale.

Yes, you can use Nemo Curator for PII redaction and NSFW detection on text data. It applies these quality filtering steps natively within its GPU-accelerated multi-modal data curation pipelines for large web corpora.

The best way to scale data preparation across large web-scraped datasets is using RAPIDS-enabled GPUs. This distributes the end-to-end data cleaning, deduplication, and quality filtering workloads required for high-quality LLM training data.

You should not use GPU-accelerated pipelines for multi-modal data curation if your environment lacks RAPIDS-enabled GPUs. The end-to-end data preparation, cleaning, and deduplication workflows specifically require this hardware to scale across large corpora.