nemo-curator

Name: nemo-curator
Availability: InStock
Author: zerwiz

Filter low-quality records, remove duplicates, and redact sensitive information from large datasets.

1|Updated Apr 10, 2026

Search Tags:#multimodal #deduplication #quality filtering #data curation #pii redaction #gpu acceleration #llm training

Ready to run

npx skills add https://github.com/zerwiz/wayofpi --skill nemo-curator-zerwiz

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: nemo-curator
Source: https://github.com/zerwiz/wayofpi/tree/main/.hermes/hermes-agent/optional-skills/mlops/nemo-curator
Command: npx skills add https://github.com/zerwiz/wayofpi --skill nemo-curator-zerwiz

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill helps you turn messy raw data into high-quality training datasets by removing duplicates, filtering low-quality content, and redacting sensitive information.

Core Features & Use Cases

Quality filtering: Apply heuristic and classifier-based filters to remove weak, repetitive, or low-value records.
Deduplication: Use exact, fuzzy, or semantic deduplication to eliminate repeated or near-duplicate documents at scale.
Multimodal curation: Prepare text, image, video, and audio datasets for model training and evaluation.
Use case: Clean large web scrapes or curated corpora before LLM training so downstream models learn from higher-signal data.

Quick Start

Use the skill to curate a large text, image, video, or audio dataset by filtering low-quality records, removing duplicates, and redacting sensitive content.

Frequently Asked Questions about nemo-curator

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I clean large web scrapes for LLM training by removing duplicates and low-quality text?▼

You can curate multimodal datasets for model training by filtering text, image, video, and audio records. The process applies quality filters and deduplication to prepare large-scale multimodal corpora for downstream evaluation.

How do I redact PII and detect NSFW content in large-scale training data?▼

Redacting PII and detecting NSFW content in large-scale training data is handled during the curation workflow. The system processes high-volume datasets to identify and redact sensitive information while filtering inappropriate material.

Can I use GPU acceleration for distributed data curation and preprocessing on multi-GPU systems?▼

Yes, you can use GPU acceleration for distributed data curation on single or multi-GPU systems. This approach accelerates large-scale preprocessing tasks like quality filtering and deduplication for massive training corpora.

What is the best way to perform semantic deduplication on high-volume text corpora?▼

The best way to perform semantic deduplication on high-volume text corpora is using GPU-accelerated curation tools. This removes near-duplicate documents efficiently, ensuring downstream models learn from higher-signal data without repetitive content.

Does near-duplicate document removal work for both text and multimodal datasets?▼

Yes, near-duplicate document removal works for both text and multimodal datasets. The curation process supports exact, fuzzy, and semantic deduplication across text, image, video, and audio data to eliminate repeated records.