skills.rest
SethyPagna
SethyPagnaCOMMUNITY
Explore all skills by this author →
GitHub

nemo-curator

Filter, deduplicate, and redact large-scale training datasets with GPU-accelerated RAPIDS processing.

Updated May 14, 2026
Search Tags:#multimodal#deduplication#quality filtering#data curation#pii redaction#gpu acceleration#llm training
One-click install
npx skills add https://github.com/SethyPagna/Secretary-Jarvis --skill nemo-curator-sethypagna
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: nemo-curator
Source: https://github.com/SethyPagna/Secretary-Jarvis/tree/main/src/capabilities/optional-skills/mlops/nemo-curator
Command: npx skills add https://github.com/SethyPagna/Secretary-Jarvis --skill nemo-curator-sethypagna

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

NeMo Curator removes the manual, error-prone work of cleaning massive training corpora by filtering low-quality content, removing duplicate records, and redacting sensitive information at scale.

Core Features & Use Cases

  • High-speed deduplication: Use exact, fuzzy, and semantic methods to clean web-scale text collections and reduce repeated examples.
  • Quality and safety filtering: Apply 30+ heuristics, language checks, NSFW detection, and classifier-based scoring to keep only useful data.
  • Multimodal curation: Prepare text, image, video, and audio datasets for LLM and multimodal model training, including PII redaction and distributed GPU processing.

Quick Start

Use NeMo Curator to curate your dataset by filtering low-quality rows, deduplicating near matches, and redacting sensitive information before saving the cleaned output.

Frequently Asked Questions about nemo-curator

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I clean and deduplicate large-scale web scrape datasets for LLM training?▼

Large-scale dataset cleaning for LLM training is achieved by applying exact, fuzzy, and semantic deduplication methods to web scrapes and Common Crawl pipelines, removing duplicate records to reduce repeated examples.

What is the best way to redact sensitive information in multimodal training corpora?▼

Redacting sensitive information in multimodal corpora involves applying PII redaction techniques across text, image, video, and audio datasets, ensuring privacy and safety before using the data for model training.

Can I use GPU acceleration for quality filtering on text and image collections?▼

GPU acceleration is supported for quality filtering through distributed RAPIDS processing and GPU-accelerated heuristics, enabling high-speed application of classifier-based scoring and safety checks across large corpora.

How do I filter low-quality records from Common Crawl pipelines?▼

Filtering low-quality records from Common Crawl pipelines utilizes over 30 heuristics, language checks, and NSFW detection to evaluate content, keeping only useful data for subsequent training tasks.

Does distributed RAPIDS processing work for exact and fuzzy deduplication?▼

Distributed RAPIDS processing handles both exact and fuzzy deduplication at scale, cleaning web-scale text collections by identifying and removing near-match records efficiently.

Related Skills

Explore More Data & Analytics Skills→
Clay-HHKClay-HHK
2

nemo-curator

Curate and deduplicate large-scale multimodal corpora with GPU acceleration.

Community
Advanced
JeffallanJeffallan
10.9k

fine-tuning-expert

Automate fine-tuning workflows for large language models using LoRA/QLoRA and PEFT methods.

Community
Advanced
founderphantomfounderphantom

nemo-curator

Automates GPU-accelerated cleaning and deduplication of multimodal LLM training data with fuzzy/semantic dedup, PII redaction, NSFW detection via Nemo Curator and optional RAPIDS stack.

Community
Advanced
ricablericable

nemo-curator

Curate large-scale multimodal datasets with quality filtering, deduplication, and PII redaction.

Community
Advanced