skills.rest
Tnemo65
Tnemo65COMMUNITY
Explore all skills by this author →
GitHub

nemo-curator

Process and curate large ML training datasets with GPU-accelerated deduplication, filtering, and redaction.

Updated Apr 27, 2026
Search Tags:#multimodal#deduplication#gpu-acceleration#pii-redaction#data-curation#nsfw-detection#nemo-curator
One-click install
npx skills add https://github.com/Tnemo65/template --skill nemo-curator-tnemo65
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: nemo-curator
Source: https://github.com/Tnemo65/template/tree/main/.cursor/skills/03-data-processing/nemo-curator
Command: npx skills add https://github.com/Tnemo65/template --skill nemo-curator-tnemo65

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

NeMo Curator automates the end-to-end data curation workflow for ML datasets, combining fast GPU-accelerated deduplication, quality filtering, PII redaction, and NSFW detection to produce clean, production-ready training data.

Core Features & Use Cases

  • Fuzzy and exact deduplication across text and multimodal data to dramatically reduce data redundancy.
  • Quality filtering with 30+ heuristics to remove low-quality or toxic content before training.
  • PII redaction & NSFW detection to protect privacy and ensure safety in training corpora.
  • Multimodal support enabling curation of text, image, video, and audio datasets at scale on RAPIDS-powered GPUs.
  • Use cases include preparing high-quality training datasets, cleaning web data, and deduplicating large corpora for reproducible experiments.

Quick Start

Run Nemo Curator on your dataset to curate multimodal data with deduplication, filtering, and redaction.

Frequently Asked Questions about nemo-curator

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I deduplicate large multimodal datasets for LLM training?▼

Redact PII from training corpora by applying GPU-accelerated PII redaction to your datasets. This process scans text data to identify and remove sensitive information, ensuring privacy and safety before model training.

Can I clean web data and remove toxic content before model training?▼

Yes, you can clean web data by applying quality filtering with 30+ heuristics alongside NSFW detection. This removes low-quality or toxic content to produce clean, reproducible training data.

Does GPU-accelerated data curation work for both text and image data?▼

Deduplicate large corpora by running fuzzy and exact deduplication across your datasets. This dramatically reduces data redundancy, ensuring cleaner and more reproducible experiments for machine learning models.

What is the best way to scale data curation for large ML corpora?▼

Remove NSFW content from datasets by applying NSFW detection during the data curation workflow. This identifies and filters explicit material, ensuring safety in your training corpora before model training.

Do I need GPU clusters to run fuzzy deduplication on training data?▼

You do not need GPU clusters to run fuzzy deduplication; it scales from a single machine to GPU clusters. However, using RAPIDS-powered GPUs significantly accelerates the deduplication process for large corpora.

Related Skills

Explore More Data & Analytics Skills→
Chris-Chai-MinjaeChris-Chai-Minjae

nemo-curator

Automate GPU-accelerated curation of multimodal datasets with deduplication and filtering.

Community
Advanced
PlaidmustachePlaidmustache

nemo-curator

GPU-accelerate multimodal data curation for LLM training with RAPIDS.

Community
Intermediate
ovachieverovachiever
52

nemo-curator

Curate multimodal LLM training data with GPU-accelerated filtering and deduplication.

Community
Advanced
overviewlabsoverviewlabs

nemo-curator

Curate multi-modal datasets for LLM training with GPU-accelerated filters and deduplication.

Community
Advanced