nemo-curator

You can accelerate LLM data curation by using GPU-intensive operations with NVIDIA RAPIDS, achieving up to 16x faster deduplication and efficient scaling across multiple GPUs for massive text and multimodal datasets.

Yes, GPU-accelerated data processing supports multimodal datasets, handling text, image, video, and audio data curation to prepare comprehensive training data for multimodal LLMs.

The best way to perform fuzzy and semantic deduplication on web scraped text is using GPU-accelerated operations that scale efficiently across GPUs, removing exact and near-duplicate content rapidly.

Yes, you can redact PII and filter NSFW content during LLM training data preparation by applying advanced quality filtering and sensitive information redaction features directly within the curation pipeline.

Yes, you need Dask and cuDF dependencies to scale data curation across multiple GPUs, as these frameworks enable the distributed computing and GPU acceleration required for high-throughput dataset preparation.

You can perform fuzzy deduplication on raw text data by executing GPU-accelerated operations that process the dataset at '/data/raw_text_data', removing near-duplicates efficiently using NVIDIA RAPIDS.