nemo-curator

To curate multimodal LLM training data, you can apply GPU-accelerated deduplication, quality filtering, and PII redaction to large web-scraped corpora. This scales from single-node workstations to GPU clusters to ensure high-quality, safe datasets.

Performing fuzzy, exact, and semantic deduplication on large datasets is best handled using GPU-accelerated workflows. This approach integrates with RAPIDS and Dask to efficiently process and clean web-scraped corpora at scale.

Yes, GPU-accelerated data curation supports PII redaction and NSFW detection. It applies over 30 quality filters alongside these safety measures to ensure multimodal datasets containing text, image, video, and audio are safe for LLM training.

Yes, you can use Dask and RAPIDS for dataset cleaning on a single-node workstation. GPU-accelerated data curation workflows scale seamlessly from single-node environments up to full CUDA-enabled GPU clusters.

GPU acceleration is used for data curation to handle the massive computational demands of fuzzy, exact, and semantic deduplication across large datasets. CUDA-enabled workflows process multimodal data significantly faster than CPU-based filtering methods.

NeMo Curator supports multimodal data including text, image, video, and audio for quality filtering. It applies scalable GPU workloads and over 30 quality filters to ensure high-quality training data across all supported formats.