nemo-curator

You can deduplicate large multimodal datasets for LLM training using GPU-accelerated data curation. It supports exact, fuzzy, and semantic deduplication strategies at scale across text, image, video, and audio.

GPU-accelerated data curation cleans and deduplicates large training datasets using GPU clusters. It targets multi-modal data workflows across text, image, video, and audio to improve data quality and reduce processing time.

Yes, you can redact PII and filter unsafe content during dataset preparation. The pipeline includes PII redaction to remove sensitive information and NSFW detection to flag unsafe content for review.

You can apply quality filtering to web-scraped text and image data using over 30 built-in heuristics. This filters low-quality content to prepare cleaned, RedPajama-like training data.

Yes, you need a GPU cluster to run multi-stage data curation pipelines. The workflow is scalable to GPU clusters and uses Nemo Curator with RAPIDS to satisfy end-to-end pipeline requirements.

The best way to curate web-scraped training data at scale is running a multi-stage pipeline across a GPU cluster. This process handles quality filtering, various deduplication strategies, PII redaction, and safety checks simultaneously.