nemo-curator

Clean and deduplicate web data for LLM training by applying GPU-accelerated quality filters, fuzzy and semantic deduplication, PII redaction, and NSFW detection using RAPIDS. This prepares high-quality, compliant multimodal corpora for scalable model training.

Yes, you can curate multimodal datasets across text, image, video, and audio on a GPU cluster. The data curation process uses RAPIDS acceleration to scale quality filtering and deduplication across large multimodal training corpora.

Redact PII and detect NSFW content in large training corpora by running GPU-accelerated data curation with RAPIDS. This enforces privacy and safety compliance while scaling efficiently across large datasets intended for LLM training.

Fuzzy deduplication works for large text corpora without slowing processing by utilizing GPU acceleration via RAPIDS. This approach operates 16 times faster than standard methods to ensure unique, high-quality training data.

Apply quality filters to web-scraped data before model training by running GPU-accelerated data curation. The process enforces over 30 quality heuristics alongside semantic deduplication to ensure clean, production-grade corpora.

You need semantic deduplication for LLM training datasets when handling large-scale multimodal corpora where exact matching is insufficient. It identifies and removes contextually similar data points, ensuring high-quality, unique training samples.