skills.rest
dawsonblock
dawsonblockCOMMUNITY
Explore all skills by this author →
GitHub

nemo-curator

Automates GPU-accelerated data curation for LLM training datasets.

Updated Apr 26, 2026
Search Tags:#multimodal#deduplication#gpu-accelerated#pii-redaction#data-curation#nemo-curator
One-click install
npx skills add https://github.com/dawsonblock/HERMY --skill nemo-curator-dawsonblock
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: nemo-curator
Source: https://github.com/dawsonblock/HERMY/tree/main/hermes-agent-2026.4.23/optional-skills/mlops/nemo-curator
Command: npx skills add https://github.com/dawsonblock/HERMY --skill nemo-curator-dawsonblock

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Nemo Curator addresses the challenge of building high-quality training datasets by providing GPU-accelerated data curation tools that perform deduplication, quality filtering, PII redaction, and NSFW detection across text, image, video, and audio data.

Core Features & Use Cases

  • GPU-accelerated deduplication (exact, fuzzy, semantic) across large datasets.
  • 30+ quality filters for data cleaning.
  • PII redaction and NSFW detection for compliant datasets.
  • Multimodal support for text, image, video, and audio data.
  • Scales across RAPIDS-enabled GPUs with distributed processing.
  • Use cases include preparing large-scale training data, cleaning web-scraped corpora, and deduplicating substantial multimodal datasets.

Quick Start

Run the Nemo Curator pipeline on a multimodal dataset to deduplicate, filter quality, redact PII, and detect NSFW.

Frequently Asked Questions about nemo-curator

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I deduplicate large web-scraped corpora for LLM training?▼

GPU-accelerated data curation automates deduplication of large web-scraped corpora for LLM training by applying exact, fuzzy, and semantic matching across RAPIDS-enabled clusters to ensure high dataset quality.

Can I redact PII and detect NSFW content in multimodal datasets?▼

Yes, you can redact PII and detect NSFW content in multimodal datasets. The data curation pipeline supports text, image, video, and audio data, applying PII redaction and NSFW detection to ensure your training data is compliant.

What quality filters are available for cleaning LLM training data?▼

Over 30 quality filters are available for cleaning LLM training data. These GPU-accelerated filters automate the removal of low-quality content during the data curation pipeline.

Does GPU-accelerated data curation scale across multiple GPUs?▼

Yes, GPU-accelerated data curation scales across multiple GPUs. It uses RAPIDS-based distributed processing to handle large-scale multimodal datasets and deduplication across substantial GPU clusters.

What is semantic deduplication and when do I need it for LLM datasets?▼

Semantic deduplication removes near-duplicate data based on meaning rather than exact text matches. You need it for LLM datasets to improve training efficiency and prevent the model from overfitting to redundant concepts.

Related Skills

Explore More Data & Analytics Skills→
ovachieverovachiever
52

nemo-curator

Curate multimodal LLM training data with GPU-accelerated filtering and deduplication.

Community
Advanced
sheawinklersheawinkler
78

nemo-curator

Deduplicate, filter, and redact multi-modal training data with GPU acceleration.

Community
Advanced
projectedanxprojectedanx
1

nemo-curator

Deduplicate and filter large-scale LLM training data with GPU-optimized pipelines.

Community
Advanced
Quill-AgentQuill-Agent
3

nemo-curator

Accelerate LLM training data curation with GPU-based deduplication and quality filtering.

Community
Advanced