nemo-curator

GPU-accelerated fuzzy deduplication processes large LLM training corpora 16× faster by scaling across RAPIDS-enabled GPU clusters. It identifies and removes near-duplicate entries to improve dataset quality and training efficiency.

Yes, PII redaction and quality filtering support multi-modal datasets including text, image, video, and audio. The pipeline applies over 30 heuristics to filter data quality and redact sensitive information across all modalities.

GPU-accelerated data curation supports flexible I/O using Parquet, JSONL, and CSV formats. This allows seamless integration with existing data pipelines for reading raw data and writing curated datasets.

Yes, RAPIDS-backed GPU acceleration is required to run these data curation workflows efficiently. It enables modular stages like quality filtering, deduplication, and redaction to scale across large web-scraped corpora.

The best way to clean web-scraped data for LLM training is using a modular pipeline with semantic deduplication, NSFW detection, and quality filtering. This approach scales across GPU clusters to produce high-quality training datasets.

Semantic deduplication removes conceptually similar entries across multi-modal data like text, images, and video. It works alongside exact and fuzzy deduplication within the GPU-accelerated pipeline to ensure diverse training data.