nemo-curator

GPU-accelerated data curation speeds up dataset deduplication and cleaning for LLM training using RAPIDS and CUDA, performing tasks up to 16 times faster than CPU-based alternatives.

Yes, NVIDIA CUDA-enabled hardware is strictly required. GPU-accelerated data curation relies on the NeMo Curator toolkit and RAPIDS to execute high-performance data transformation tasks across distributed GPU clusters.

The best way to remove PII and NSFW content from large-scale web corpora is using a comprehensive curation pipeline, which includes over 30 heuristic filters alongside dedicated PII redaction and NSFW detection modules.

Yes, GPU-accelerated data curation supports multi-modal datasets. It can process and clean text, image, video, and audio data to ensure high-quality, safe training data for foundation models.

Fuzzy deduplication on parquet files works by applying GPU-based heuristics across distributed clusters to identify and remove near-duplicate records, accelerating the curation of multi-terabyte web-scraped corpora like Common Crawl.

Yes, Dask and RAPIDS are used together for distributed semantic filtering in data curation. This combination enables complex pipelines to process multi-terabyte datasets efficiently across distributed GPU clusters.