nemo-curator

You can deduplicate large datasets for LLM training using GPU-accelerated MinHash and LSH for fuzzy deduplication, alongside exact and semantic deduplication, to remove near-duplicates and redundant data across huge corpora.

Yes, PII redaction and NSFW detection can be applied across multimodal datasets including text, image, audio, and video. This protects privacy and ensures compliance by filtering unsafe content during the GPU-accelerated curation pipeline.

The best way to filter low-quality text is applying extensive quality filtering with over 30 heuristics. This prunes toxic and low-quality content while preserving valuable signals in your web-scraped or multimodal training data.

GPU-accelerated data curation supports multi-format inputs including text, image, audio, and video. It uses RAPIDS-based acceleration and scalable orchestration to process multimodal corpora through pluggable Python workflow stages.

Semantic deduplication cleans multimodal datasets by identifying and removing contextually similar data points, ensuring redundancy-free data. It operates within modular, GPU-accelerated pipelines alongside exact and fuzzy deduplication stages.

Yes, RAPIDS-based acceleration is required to run these GPU-accelerated data curation pipelines. It provides the scalable orchestration and processing power needed for exact, fuzzy, and semantic deduplication across large multimodal corpora.