skills.rest
Z43L
Z43LCOMMUNITY
Explore all skills by this author →
GitHub

nemo-curator

Clean and deduplicate large-scale LLM training datasets with GPU-accelerated filters and deduplication methods.

Updated May 5, 2026
Search Tags:#multimodal#deduplication#pii redaction#gpu acceleration#llm training#dataset curation#nsfw filtering
One-click install
npx skills add https://github.com/Z43L/zeus-agent --skill nemo-curator-z43l
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: nemo-curator
Source: https://github.com/Z43L/zeus-agent/tree/main/optional-skills/mlops/nemo-curator
Command: npx skills add https://github.com/Z43L/zeus-agent --skill nemo-curator-z43l

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

NeMo Curator removes low-quality, duplicated, and unsafe content from large web-scale datasets so you can train LLMs on cleaner material.

Core Features & Use Cases

  • GPU-accelerated quality filtering: Applies 30+ heuristic filters to drop short, repetitive, noisy, or suspicious text.
  • Exact, fuzzy, and semantic deduplication: Removes identical, near-duplicate, and paraphrase-like repeats using MinHash/LSH and embedding similarity.
  • PII and safety handling: Redacts personally identifiable information and supports NSFW classification for safer training corpora.
  • Multimodal support: Works across text, images, video, and audio curation pipelines for multimodal model training.

Use it to clean Common Crawl-derived datasets, curate high-quality training sets for production runs, or deduplicate massive corpora before fine-tuning.

Quick Start

Run NeMo Curator to curate a dataset by installing nemo-curator with the appropriate CUDA or CPU extras and then applying quality filters, deduplication, and PII/NSFW stages in a single pipeline.

Frequently Asked Questions about nemo-curator

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I deduplicate large-scale LLM training datasets?▼

Large-scale LLM training datasets are deduplicated using exact, fuzzy, and semantic deduplication methods. MinHash/LSH and embedding similarity remove identical, near-duplicate, and paraphrase-like repeats to ensure a cleaner corpus.

Does GPU acceleration support dataset curation for multimodal model training?▼

GPU acceleration does support dataset curation for multimodal model training. It applies heuristic quality filters and deduplication across text, images, video, and audio curation pipelines using distributed processing.

How do I redact personally identifiable information from Common Crawl preprocessing pipelines?▼

Personally identifiable information is redacted from Common Crawl preprocessing pipelines using dedicated PII handling stages. It detects and removes sensitive content to build safer training corpora for LLM runs.

What is the best way to clean low-quality text from web-scale corpora?▼

The best way to clean low-quality text from web-scale corpora is applying 30+ heuristic quality filters. This GPU-accelerated process drops short, repetitive, noisy, or suspicious text before LLM training.

Can I filter NSFW content during dataset curation?▼

You can filter NSFW content during dataset curation using supported NSFW classification. It identifies and removes unsafe material from training sets alongside PII redaction and quality filtering.

Do I need GPU-capable execution for fuzzy and semantic deduplication?▼

GPU-capable execution is required for fuzzy and semantic deduplication. The pipeline relies on NeMo Curator components and distributed processing to handle massive corpora efficiently during training data preparation.

Related Skills

Explore More Data & Analytics Skills→
KappTech88KappTech88

nemo-curator

Curate multimodal datasets for LLM training with GPU-accelerated pipelines.

Community
Advanced
ricablericable

unsloth

Accelerate LLM fine-tuning with LoRA and QLoRA on local hardware.

Community
Advanced
sovr610sovr610

Training Data Dedup + Leakage Guard

Deduplicate training corpora and detect holdout leakage with MinHash+LSH.

Community
Advanced
qcmuuqcmuu

nemo-curator

Filter, deduplicate, and redact large Parquet corpora on GPUs.

Community
Advanced