nemo-curator

You can deduplicate multimodal datasets for LLM training using GPU-accelerated fuzzy, exact, and semantic deduplication. This process scales across text, image, video, and audio sources using RAPIDS-backed tools to identify and remove duplicate records efficiently.

GPU-accelerated data curation for LLM training automates the preparation of high-quality datasets using RAPIDS-backed tools. It applies 30+ quality filters, deduplication, PII redaction, and NSFW detection across multimodal data sources to ensure safer model training.

Yes, you can use RAPIDS-backed tools to perform PII redaction and NSFW detection on text corpora. This workflow applies quality filtering and redaction during the data curation process to produce safer training data for LLMs.

GPU-accelerated data curation supports web-scraped text and image sources, alongside video and audio data. It cleans these multimodal inputs by applying quality filters and deduplicating copies across different sources for model training preparation.

The best way to scale quality filtering across large multimodal datasets is using GPU-accelerated workflows with RAPIDS. This approach applies 30+ quality filters and semantic deduplication efficiently, preparing high-quality training data across text, image, video, and audio formats.

Semantic deduplication is used when preparing LLM training data to identify and remove contextually similar records, not just exact matches. This GPU-accelerated process improves dataset quality and reduces redundancy across large multimodal sources before model training.