skills.rest
ruiyangruiyi
ruiyangruiyiCOMMUNITY
Explore all skills by this author →
GitHub

nemo-curator

Curate large text and multimodal datasets for LLM training.

Updated May 26, 2026
Search Tags:#deduplication#rapids#pii redaction#gpu acceleration#dataset curation#llm training data#nsfw filtering
One-click install
npx skills add https://github.com/ruiyangruiyi/hermes-agent --skill nemo-curator-ruiyangruiyi
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: nemo-curator
Source: https://github.com/ruiyangruiyi/hermes-agent/tree/main/optional-skills/mlops/nemo-curator
Command: npx skills add https://github.com/ruiyangruiyi/hermes-agent --skill nemo-curator-ruiyangruiyi

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires nemo-curator, cudf, dask, rapids, and includes references (resource) components.

What problem does it solve?

NeMo Curator helps you turn raw web-scraped or user-generated data into cleaner, higher-quality training datasets by removing duplicates, filtering low-quality content, and redacting sensitive information at scale.

Core Features & Use Cases

  • GPU-accelerated curation: Perform fast, scalable cleaning pipelines across GPUs using RAPIDS/Dask for large corpora.
  • Multimodal quality gates: Apply quality filtering and classifier-based filtering to text and other modalities (where supported).
  • Deduplication at multiple levels: Use exact, fuzzy (MinHash/LSH), and semantic (embedding-based) deduplication to reduce wasted training on redundant data.
  • PII redaction and NSFW filtering: Redact personally identifiable information and detect/filter NSFW content to reduce compliance and safety risk.
  • Use case: Clean Common Crawl or collected web data before training so your model learns from diverse, non-redundant, and safer examples.

Quick Start

Use the nemo-curator skill to deduplicate, quality-filter, redact PII, and export your cleaned dataset for LLM training from a large Parquet or JSONL corpus.

Frequently Asked Questions about nemo-curator

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I clean and deduplicate Common Crawl data for LLM training?▼

To clean and deduplicate Common Crawl data for LLM training, you can curate raw corpora by removing exact, fuzzy, and semantic duplicates while applying quality heuristics to filter low-quality text.

What is the best way to remove fuzzy and semantic duplicates from large text corpora?▼

Removing fuzzy and semantic duplicates from large text corpora involves using MinHash/LSH algorithms and embedding-based techniques to identify and drop redundant data, reducing wasted training compute on repetitive examples.

Do I need a RAPIDS-compatible GPU to run dataset curation pipelines?▼

Yes, you need a RAPIDS-compatible GPU because the dataset curation pipelines rely on RAPIDS, cuDF, and Dask to execute fast, distributed data processing and scaling across GPU clusters.

How does PII redaction and NSFW filtering work for web-scraped datasets?▼

PII redaction and NSFW filtering work by automatically detecting and masking personally identifiable information and unsafe content within web-scraped datasets, reducing compliance and safety risks before LLM training.

Can I use Dask to scale distributed dataset cleaning across multiple GPUs?▼

Yes, you can use Dask to scale distributed dataset cleaning across multiple GPUs, enabling fast and scalable curation pipelines that process large multimodal and text corpora efficiently.

What are the limitations of GPU-accelerated text deduplication for LLM training data?▼

A limitation of GPU-accelerated text deduplication is the strict hardware requirement for RAPIDS-compatible GPUs, which is necessary to execute the distributed processing and scale the curation workflows.

Related Skills

Explore More Data & Analytics Skills→
thisismynewfmail-uithisismynewfmail-ui

nemo-curator

Curate multimodal datasets with quality filtering, deduplication, and PII redaction.

Community
Advanced
ricablericable

unsloth

Accelerate LLM fine-tuning with LoRA and QLoRA on local hardware.

Community
Advanced
KarlinskySKarlinskyS

nemo-curator

Deduplicate, filter, redact PII, and remove NSFW content from large multimodal corpora.

Community
Advanced
carterwayneskhizeinecarterwayneskhizeine
19

nemo-curator

Curate noisy training data with GPU-accelerated deduplication and PII redaction.

Community
Advanced