nemo-curator

Name: nemo-curator
Availability: InStock
Author: icyzh

Curate large-scale LLM training datasets with GPU-accelerated deduplication, filtering, and PII redaction.

1|Updated Jul 31, 2026

Search Tags:#deduplication #data curation #pii redaction #gpu acceleration #llm training #nemo-curator

Ready to run

npx skills add https://github.com/icyzh/hermes-web --skill nemo-curator-icyzh

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: nemo-curator
Source: https://github.com/icyzh/hermes-web/tree/main/optional-skills/mlops/nemo-curator
Command: npx skills add https://github.com/icyzh/hermes-web --skill nemo-curator-icyzh

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires nemo-curator, cudf, dask, rapids, and includes references (resource) components.

What problem does it solve?

This Skill addresses the massive computational bottleneck of preparing high-quality training data for Large Language Models by leveraging GPU acceleration to perform deduplication, filtering, and PII redaction at scale.

Core Features & Use Cases

High-Speed Deduplication: Perform exact, fuzzy, and semantic deduplication up to 16 times faster than CPU-based alternatives.
Multi-Modal Processing: Curate diverse datasets including text, images, video, and audio with specialized heuristic and classifier-based filters.
Use Case: When preparing a multi-terabyte web-scraped dataset like Common Crawl, use this Skill to remove toxic content, redact sensitive PII, and eliminate near-duplicate documents efficiently on a GPU cluster.

Quick Start

Use the nemo-curator skill to run a deduplication pipeline on the dataset located in the current directory.

Frequently Asked Questions about nemo-curator

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I accelerate deduplication for large-scale LLM training datasets?▼

GPU-accelerated deduplication uses RAPIDS libraries to process large-scale LLM training datasets up to 16 times faster than CPU-based alternatives, performing exact, fuzzy, and semantic matching efficiently.

Can I curate multi-modal data like images and audio for LLM training?▼

Multi-modal data curation supports text, images, video, and audio modalities. It applies specialized heuristic and classifier-based filters to prepare diverse production-grade training datasets.

What's the best way to redact PII from web-scraped datasets like Common Crawl?▼

PII redaction removes sensitive personal information from multi-terabyte web-scraped datasets like Common Crawl. It leverages GPU clusters to efficiently redact sensitive data during high-throughput preparation pipelines.

Does GPU-accelerated data curation require Ray and Dask architectures?▼

High-throughput GPU-accelerated data curation relies on Ray-based architectures and RAPIDS libraries like cudf and dask to distribute workloads across GPU clusters for production-grade dataset preparation.

Why use GPU-accelerated filtering instead of CPU-based data curation tools?▼

GPU-accelerated filtering overcomes massive computational bottlenecks in data curation. It removes toxic content and near-duplicate documents significantly faster than CPU-based alternatives, satisfying high-throughput processing requirements.

When should I not use GPU-accelerated data curation for dataset preparation?▼

GPU-accelerated data curation is designed for massive computational bottlenecks and multi-terabyte datasets. It requires a GPU cluster and dependencies like cudf and dask, making it unsuitable for smaller scale or single-CPU processing environments.