nemo-curator

GPU-accelerated data curation uses GPU clusters to automate deduplication, PII redaction, and quality filtering across large-scale multimodal datasets. It processes text, image, video, and audio data to improve dataset quality for LLM training pipelines.

You can deduplicate multimodal datasets by applying fuzzy, semantic, and exact deduplication across text, image, video, and audio data. This process scales across GPU clusters to clean large-scale corpora effectively.

Yes, multimodal data curation supports cross-format I/O for both Parquet and JSONL files. It processes these formats across text, image, video, and audio data while managing dependencies within modular GPU pipelines.

Yes, you can apply PII redaction and NSFW detection during data curation. The process includes over 30 quality filters and classifiers to ensure dataset safety and improve overall quality for ML training.

The best way to scale data curation pipelines is using GPU-accelerated execution across clusters. This approach streamlines end-to-end fuzzy and semantic deduplication, quality filtering, and multimodal processing for massive ML training datasets.

You should use GPU-accelerated deduplication when managing large-scale multimodal datasets where manual cleaning is impractical. It automates fuzzy, semantic, and exact deduplication alongside PII redaction to drastically reduce manual effort.