nemo-curator

Deduplicate large multimodal datasets for LLM training using GPU-accelerated exact, fuzzy, and semantic matching. This eliminates duplicate records across text, image, video, and audio pipelines to produce a clean training corpus.

GPU-accelerated data curation is the process of using parallel processing to filter noisy, low-quality data. It applies stage-based quality heuristics and classifiers to accelerate reliable multimodal model training at scale.

Automate PII redaction on training corpora to remove sensitive information before model training. This privacy feature safely processes large text datasets, including RedPajama-like corpora, ensuring compliant data preparation.

GPU-accelerated data curation requires RAPIDS, cuDF, and Dask to process large datasets. These dependencies enable distributed GPU processing, ensuring cross-modal compatibility and scalable data preparation.

Filter low-quality text from common crawl datasets by applying over 30 quality heuristics and classifiers. This stage-based quality filtering eliminates noisy data, accelerating reliable model training and reproducible curation.

Data curation is necessary for multimodal LLM pipelines because large-scale training data is noisy, duplicated, and costly. Curation eliminates low-quality data across text, image, video, and audio formats to accelerate model development.