nemo-curator

GPU-accelerated data curation cleans large training datasets for LLMs by handling text, image, video, and audio formats. It performs fuzzy and semantic deduplication, quality filtering, PII redaction, and NSFW detection to prepare high-quality training data.

Yes, GPU-accelerated data curation works with multimodal datasets. It supports text, image, video, and audio data, allowing you to organize and clean diverse training corpora using modular pipelines for future extensions.

You can deduplicate large corpora across GPUs using NVIDIA RAPIDS. GPU-accelerated data curation scales across GPUs to perform both fuzzy and semantic deduplication on large-scale datasets, ensuring corpus quality.

The best way to prepare high-quality training data from web data is using GPU-accelerated data curation. It cleans web data by applying quality filtering, PII redaction, and NSFW detection to ensure the resulting corpus is safe and effective for LLM training.

Yes, you can redact PII and detect NSFW content during data curation. GPU-accelerated data curation integrates PII redaction and NSFW detection directly into the pipeline to ensure corpus quality and safety.