nemo-curator

Name: nemo-curator
Availability: InStock
Author: Nitish

Deduplicate and filter LLM training datasets with GPU acceleration.

Updated Jun 26, 2026

Search Tags:#GPU acceleration #deduplication #LLM training #quality filtering #data curation

Ready to run

npx skills add https://github.com/NITISH-gitbit/hermes-custom --skill nemo-curator-nitish-gitbit

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: nemo-curator
Source: https://github.com/NITISH-gitbit/hermes-custom/tree/main/optional-skills/mlops/nemo-curator
Command: npx skills add https://github.com/NITISH-gitbit/hermes-custom --skill nemo-curator-nitish-gitbit

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires nemo-curator, cudf, dask, rapids, and includes scripts (resource) and references (resource) and assets (resource) components.

What problem does it solve?

This Skill streamlines the preparation of high-quality training datasets for Large Language Models (LLMs), reducing the manual labor involved in data curation and ensuring efficient processing.

Core Features & Use Cases

Data Curation: Supports text, image, video, and audio data curation.
Fuzzy Deduplication: Achieves 16x faster deduplication than CPU-based methods.
Quality Filtering: Applies over 30 heuristics for quality filtering.
Use Case: Use this Skill to clean and deduplicate a large corpus of text data for LLM training.

Quick Start

Run the 'nemo-curator' skill to deduplicate and filter the dataset in the 'data' directory.

Frequently Asked Questions about nemo-curator

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I prepare high-quality training datasets for LLMs with GPU acceleration?▼

GPU-accelerated data curation streamlines preparation of high-quality training datasets for LLMs by performing fuzzy deduplication and applying over 30 quality filtering heuristics across text, image, video, and audio formats.

What is the best way to deduplicate a large text corpus for LLM training?▼

Fuzzy deduplication using GPU acceleration achieves 16x faster processing than CPU-based methods, efficiently cleaning and deduplicating large text corpora to ensure high-quality inputs for LLM training tasks.

Does nemo-curator support image, video, and audio data curation or just text?▼

nemo-curator supports data curation for text, image, video, and audio data, applying GPU-accelerated quality filtering and deduplication across multiple modalities for LLM training dataset preparation.

Do I need a GPU and RAPIDS environment to run GPU-accelerated data curation?▼

Yes, GPU-accelerated data curation requires a GPU environment with dependencies including cudf, dask, and rapids to achieve 16x faster fuzzy deduplication and quality filtering for LLM training datasets.

How does fuzzy deduplication compare to CPU-based methods for cleaning LLM training data?▼

GPU-accelerated fuzzy deduplication is 16x faster than CPU-based methods, significantly reducing processing time while applying over 30 quality filtering heuristics to clean large corpora for LLM training.

How do I apply quality filtering heuristics to clean training data?▼

You can apply quality filtering by running the curation process, which automatically evaluates datasets using over 30 heuristics alongside fuzzy deduplication to clean data in your directory for LLM training.