skills.rest
t2ance
t2anceCOMMUNITY
Explore all skills by this author →
GitHub

nemo-curator

Clean and curate multi-modal LLM training data with GPU-accelerated filtering and deduplication.

Updated Aug 23, 2026
Search Tags:#multimodal#deduplication#gpu-acceleration#llm-training#data-curation#quality-filtering#nemo-curator
One-click install
npx skills add https://github.com/t2ance/dr-claw-plugin --skill nemo-curator-t2ance
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: nemo-curator
Source: https://github.com/t2ance/dr-claw-plugin/tree/main/plugins/ml-training-stack/skills/data-processing/nemo-curator
Command: npx skills add https://github.com/t2ance/dr-claw-plugin --skill nemo-curator-t2ance

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Curating high-quality training data for LLMs is resource-intensive and error-prone; Nemo Curator provides GPU-accelerated tools to filter, deduplicate, redact PII, and manage multi-modal datasets at scale.

Core Features & Use Cases

  • Quality filtering with 30+ heuristics to remove low-quality data
  • Exact, fuzzy, and semantic deduplication across large corpora
  • PII redaction and NSFW detection to protect privacy and safety
  • Multimodal support for text, image, video, and audio data
  • Use cases include preparing RedPajama-like datasets, cleaning Common Crawl, and building high-signal training corpora

Quick Start

Install Nemo Curator and run a complete curation workflow on your dataset path to generate a clean, deduplicated dataset.

Frequently Asked Questions about nemo-curator

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I clean and deduplicate large-scale text corpora for LLM training?▼

You can clean and deduplicate large-scale text corpora for LLM training by applying GPU-accelerated quality filtering, exact and fuzzy deduplication, and PII redaction to remove low-quality and duplicate content.

What is semantic deduplication and how does it work for multimodal datasets?▼

Semantic deduplication removes contextually similar data points across multimodal datasets. It works alongside exact and fuzzy matching to filter text, image, video, and audio corpora for high-signal LLM training.

Do I need CUDA-enabled GPUs to run GPU-accelerated data curation?▼

Yes, GPU-accelerated data curation requires CUDA-enabled GPUs. The processing leverages RAPIDS to perform high-speed quality filtering and deduplication on large multimodal datasets.

Can I use Parquet and JSONL input formats for quality filtering?▼

Yes, you can use Parquet and JSONL input formats for quality filtering. The system applies over 30 heuristics to redact PII, detect NSFW content, and remove low-quality data from these formats.

What is the best way to remove PII and NSFW content from Common Crawl data?▼

The best way to remove PII and NSFW content from Common Crawl data is to apply automated quality filtering and redaction heuristics. This ensures privacy and safety while building high-signal training corpora.

Related Skills

Explore More Data & Analytics Skills→
KarlinskySKarlinskyS

nemo-curator

Deduplicate, filter, redact PII, and remove NSFW content from large multimodal corpora.

Community
Advanced
sheawinklersheawinkler
78

nemo-curator

Deduplicate, filter, and redact multi-modal training data with GPU acceleration.

Community
Advanced
thisismynewfmail-uithisismynewfmail-ui

nemo-curator

Curate multimodal datasets with quality filtering, deduplication, and PII redaction.

Community
Advanced
OpenLAIROpenLAIR
1.0k

nemo-curator

Automate GPU-accelerated data curation for LLM training datasets.

Official
Advanced