skills.rest
devMoez
devMoezCOMMUNITY
Explore all skills by this author →
GitHub

nemo-curator

Filter, deduplicate, and sanitize LLM training datasets with GPU acceleration and privacy controls.

1|Updated May 16, 2026
Search Tags:#deduplication#data curation#rapids#pii redaction#gpu acceleration#llm training data
One-click install
npx skills add https://github.com/devMoez/titan --skill nemo-curator-devmoez
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: nemo-curator
Source: https://github.com/devMoez/titan/tree/main/optional-skills/mlops/nemo-curator
Command: npx skills add https://github.com/devMoez/titan --skill nemo-curator-devmoez

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires nemo-curator, cudf, dask, rapids, and includes references (resource) components.

What problem does it solve?

NeMo Curator helps you turn messy, duplicate-heavy, and low-quality web data into high-quality, deduplicated training datasets for LLMs, while reducing privacy risks through PII redaction and filtering unsafe content.

Core Features & Use Cases

  • GPU-accelerated curation: Performs fast fuzzy deduplication and quality filtering using RAPIDS for large-scale datasets.
  • Multimodal support: Handles text plus image/video/audio curation workflows for modern LLM training pipelines.
  • Safety and privacy controls: Applies PII redaction and NSFW detection to reduce harmful or sensitive content.
  • Use case example: You have a multi-terabyte web scrape (e.g., Common Crawl) and need to deduplicate near-matches, remove low-quality/toxic items, redact PII, and export a curated Parquet dataset for training.

Quick Start

Use the nemo-curator tool to fuzzy-deduplicate and quality-filter a Parquet dataset, then redact PII and write the curated output back to Parquet.

Frequently Asked Questions about nemo-curator

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I deduplicate and clean large LLM training datasets on GPUs?▼

You can clean LLM training datasets on GPUs by applying fuzzy, semantic, and exact deduplication alongside quality classifiers using RAPIDS for accelerated, distributed processing. This workflow filters low-quality text and exports curated Parquet files.

What is the best way to redact PII from web scrape data for LLM training?▼

PII redaction for LLM training data is handled by applying safety controls that detect and sanitize sensitive information within large collections of text. This reduces privacy risks before exporting the curated dataset for model training.

Does nemo-curator support multimodal data curation for image and video datasets?▼

Multimodal data curation is supported for handling text, image, video, and audio workflows. This allows you to apply quality filtering and deduplication across diverse data types required for modern LLM training pipelines.

Can I use Dask and RAPIDS for distributed dataset cleaning on multi-terabyte Common Crawl data?▼

Distributed dataset cleaning on multi-terabyte Common Crawl data is supported using Dask and RAPIDS. This combination enables scalable, GPU-accelerated execution for fuzzy deduplication and NSFW detection across massive web scrapes.

How do I prepare high-quality training data with NSFW detection and quality heuristics?▼

High-quality training data preparation involves applying quality heuristics, classifiers, and NSFW detection to filter unsafe content. This sanitizes large text collections by removing toxic items before generating production-ready datasets.

Do I need a GPU to run fuzzy deduplication on large Parquet datasets?▼

A GPU is required to achieve GPU-accelerated fuzzy deduplication and quality filtering using RAPIDS. While the workflow processes Parquet datasets for deduplication, the accelerated performance relies specifically on GPU hardware.

Related Skills

Explore More Data & Analytics Skills→
vikrant-projectvikrant-project
2

nemo-curator

Curate multimodal data for LLM training with GPU-accelerated filtering and deduplication.

Community
Advanced
nelohenriqnelohenriq
1

nemo-curator

Curate large multimodal datasets with GPU-accelerated filtering, deduplication, and PII redaction.

Community
Advanced
blueskies1818blueskies1818
1

nemo-curator

Accelerates data curation for LLM training with GPU-based deduplication, filtering, and PII redaction.

Community
Advanced
overviewlabsoverviewlabs

nemo-curator

Curate multi-modal datasets for LLM training with GPU-accelerated filters and deduplication.

Community
Advanced