nemo-curator

Deduplicate large-scale LLM training data using GPU-accelerated exact, fuzzy, and semantic deduplication to remove redundancy from web-scraped and internal corpora. This reduces multi-terabyte dataset sizes and improves model training efficiency.

Redact PII from multimodal datasets using automated GPU-accelerated PII redaction to protect privacy and maintain compliance across text, images, video, and audio. This ensures sensitive data is removed before LLM training.

GPU-accelerated data curation supports Parquet, JSONL, and CSV formats for processing large-scale multimodal datasets. This allows seamless integration with existing data pipelines using RAPIDS for fast, cost-effective processing.

Filter NSFW content from text and image corpora automatically using built-in NSFW detection during the data curation pipeline. This removes inappropriate content to maintain safety and compliance across multimodal training datasets.

Over 30 quality filters are available for cleaning noisy web-scraped training data, including automated checks for language, content quality, and formatting. These filters cleanse multi-terabyte corpora to improve LLM performance.

Use GPU-acceleration for multimodal data curation pipelines when processing multi-terabyte corpora across text, images, video, and audio. RAPIDS integration provides fast, cost-effective processing required for large-scale LLM training data preparation.