skills.rest
zerwiz
zerwizCOMMUNITY
Explore all skills by this author →
GitHub

nemo-curator

Filter low-quality records, remove duplicates, and redact sensitive information from large datasets.

1|Updated Apr 10, 2026
Search Tags:#multimodal#deduplication#quality filtering#data curation#pii redaction#gpu acceleration#llm training
One-click install
npx skills add https://github.com/zerwiz/wayofpi --skill nemo-curator-zerwiz
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: nemo-curator
Source: https://github.com/zerwiz/wayofpi/tree/main/.hermes/hermes-agent/optional-skills/mlops/nemo-curator
Command: npx skills add https://github.com/zerwiz/wayofpi --skill nemo-curator-zerwiz

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill helps you turn messy raw data into high-quality training datasets by removing duplicates, filtering low-quality content, and redacting sensitive information.

Core Features & Use Cases

  • Quality filtering: Apply heuristic and classifier-based filters to remove weak, repetitive, or low-value records.
  • Deduplication: Use exact, fuzzy, or semantic deduplication to eliminate repeated or near-duplicate documents at scale.
  • Multimodal curation: Prepare text, image, video, and audio datasets for model training and evaluation.
  • Use case: Clean large web scrapes or curated corpora before LLM training so downstream models learn from higher-signal data.

Quick Start

Use the skill to curate a large text, image, video, or audio dataset by filtering low-quality records, removing duplicates, and redacting sensitive content.

Frequently Asked Questions about nemo-curator

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I clean large web scrapes for LLM training by removing duplicates and low-quality text?▼

You can curate multimodal datasets for model training by filtering text, image, video, and audio records. The process applies quality filters and deduplication to prepare large-scale multimodal corpora for downstream evaluation.

How do I redact PII and detect NSFW content in large-scale training data?▼

Redacting PII and detecting NSFW content in large-scale training data is handled during the curation workflow. The system processes high-volume datasets to identify and redact sensitive information while filtering inappropriate material.

Can I use GPU acceleration for distributed data curation and preprocessing on multi-GPU systems?▼

Yes, you can use GPU acceleration for distributed data curation on single or multi-GPU systems. This approach accelerates large-scale preprocessing tasks like quality filtering and deduplication for massive training corpora.

What is the best way to perform semantic deduplication on high-volume text corpora?▼

The best way to perform semantic deduplication on high-volume text corpora is using GPU-accelerated curation tools. This removes near-duplicate documents efficiently, ensuring downstream models learn from higher-signal data without repetitive content.

Does near-duplicate document removal work for both text and multimodal datasets?▼

Yes, near-duplicate document removal works for both text and multimodal datasets. The curation process supports exact, fuzzy, and semantic deduplication across text, image, video, and audio data to eliminate repeated records.

Related Skills

Explore More Data & Analytics Skills→
KarlinskySKarlinskyS

nemo-curator

Deduplicate, filter, redact PII, and remove NSFW content from large multimodal corpora.

Community
Advanced
ever-oliever-oli
3

huggingface-tokenizers

Tokenize text with fast, Rust-backed NLP preprocessing for model training and inference.

Community
Intermediate
zeronx798zeronx798

nemo-curator

Cure and deduplicate large-scale multimodal datasets for LLM training on GPU clusters.

Community
Advanced
NITISH-gitbitNITISH-gitbit

nemo-curator

Deduplicate and filter LLM training datasets with GPU acceleration.

Community
Advanced