nemo-curator

Semantic deduplication identifies and removes near-duplicate and paraphrased data points within training datasets. This process reduces redundancy and ensures the LLM receives a diverse, high-quality range of information during training.

Yes, CUDA-enabled GPUs and RAPIDS are required to enable distributed processing and GPU acceleration for data curation. This setup allows the pipelines to scale efficiently across multi-GPU clusters when handling large multimodal datasets.

The best way to remove PII and NSFW content from training data is to apply specialized quality filters during the curation pipeline. This detects and redacts sensitive information to maintain privacy, safety, and regulatory compliance across datasets.

Fuzzy and exact deduplication works by identifying and removing identical or near-identical entries across multimodal data formats. By applying these techniques within a GPU-accelerated pipeline, you significantly reduce near-duplicates and paraphrases to improve dataset quality.

Yes, you can apply over 30 quality filters across text, image, video, and audio data formats. This multi-modal support enables end-to-end data curation, ensuring comprehensive dataset quality and safety improvements before LLM training.