nemo-curator

GPU-accelerated data curation streamlines preparation of high-quality training datasets for LLMs by performing fuzzy deduplication and applying over 30 quality filtering heuristics across text, image, video, and audio formats.

Fuzzy deduplication using GPU acceleration achieves 16x faster processing than CPU-based methods, efficiently cleaning and deduplicating large text corpora to ensure high-quality inputs for LLM training tasks.

nemo-curator supports data curation for text, image, video, and audio data, applying GPU-accelerated quality filtering and deduplication across multiple modalities for LLM training dataset preparation.

Yes, GPU-accelerated data curation requires a GPU environment with dependencies including cudf, dask, and rapids to achieve 16x faster fuzzy deduplication and quality filtering for LLM training datasets.

GPU-accelerated fuzzy deduplication is 16x faster than CPU-based methods, significantly reducing processing time while applying over 30 quality filtering heuristics to clean large corpora for LLM training.

You can apply quality filtering by running the curation process, which automatically evaluates datasets using over 30 heuristics alongside fuzzy deduplication to clean data in your directory for LLM training.