nemo-curator

GPU-accelerated data curation pipelines use exact, fuzzy, and semantic deduplication to clean web-scraped datasets. By scaling across RAPIDS GPU clusters, this approach processes text, images, video, and audio to output high-quality Parquet or JSONL data for LLM training.

Specialized GPU-accelerated curation pipelines provide built-in PII redaction and NSFW detection modules. These features run alongside over 30 quality filters to safely sanitize multimodal training data before exporting it for model training.

Yes, the NVIDIA RAPIDS stack (including cudf and dask) and Nemo-curator dependencies are required. These provide the necessary GPU-acceleration to run fuzzy and semantic deduplication efficiently across large-scale multimodal datasets.

Fuzzy deduplication on GPU clusters identifies and removes near-duplicate records across large multimodal datasets. This GPU-accelerated approach operates up to 16 times faster than CPU-based methods, scaling efficiently across RAPIDS environments.

Yes, GPU-accelerated curation supports multimodal data sources including text, image, video, and audio. It applies quality filtering, deduplication, and PII redaction across these formats to prepare web-scraped data for LLM training pipelines.

You can export curated training data to Parquet or JSONL formats after applying GPU-accelerated deduplication and quality filtering. The curation pipeline processes multimodal sources and outputs these structured files directly for LLM training.