nemo-curator

Curate LLM training data with GPU-accelerated quality filtering and deduplication.

1.2k|116|Updated Mar 19, 2026

Search Tags:#multimodal #deduplication #gpu-acceleration #llm-training #data-curation #quality-filtering #nemo-curator

Ready to run

npx skills add https://github.com/math-inc/OpenGauss --skill nemo-curator-math-inc

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: nemo-curator
Source: https://github.com/math-inc/OpenGauss/tree/main/skills/mlops/evaluation/nemo-curator
Command: npx skills add https://github.com/math-inc/OpenGauss --skill nemo-curator-math-inc

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Nemo Curator provides GPU-accelerated tools to curate high-quality training data for LLMs, enabling scalable data preprocessing and cleaning.

Core Features & Use Cases

GPU-accelerated data curation for LLM training, including multi-modal support and fast deduplication
Quality filtering with 30+ heuristics, PII redaction, and NSFW safeguards
Use cases include preparing RedPajama/The Pile style datasets, web-scraped data, and large-scale curation workflows

Quick Start

Install Nemo Curator and run a small dataset through a sample pipeline to see quality filtering and deduplication in action.

Frequently Asked Questions about nemo-curator

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I curate high-quality training data for large language models using GPU acceleration?▼

GPU-accelerated data curation for LLM training uses Nemo Curator's pipeline to perform quality filtering, deduplication, and PII redaction on web-scraped datasets at scale.

What is the best way to prepare web-scraped datasets for LLM training?▼

The best way to prepare web-scraped datasets for LLM training is applying a modular data curation pipeline that executes quality filtering, deduplication, and PII redaction.

Does Nemo Curator support multi-modal data sources for quality filtering and deduplication?▼

Yes, Nemo Curator supports multi-modal sources alongside web-scraped datasets, enabling GPU-accelerated deduplication and quality filtering across diverse training data formats.

How do I redact PII and filter NSFW content during large-scale data curation?▼

Large-scale data curation pipelines handle PII redaction and NSFW safeguards by applying over 30 quality-filtering heuristics during the GPU-accelerated preprocessing workflow.

When do I need GPU-accelerated deduplication for my LLM training datasets?▼

GPU-accelerated deduplication is needed when curating large-scale, RedPajama or The Pile style datasets where CPU-based processing creates bottlenecks in the data cleaning workflow.

Can I run a sample data curation pipeline on a small dataset before full-scale LLM training?▼

Yes, you can install Nemo Curator and run a small dataset through a sample pipeline to verify quality filtering and deduplication before scaling up to production-ready workflows.