skills.rest
carterwayneskhizeine
carterwayneskhizeineCOMMUNITY
Explore all skills by this author →
GitHub

nemo-curator

Curate noisy training data with GPU-accelerated deduplication and PII redaction.

19|4|Updated Apr 22, 2026
Search Tags:#multimodal#deduplication#quality filtering#data curation#pii redaction#gpu acceleration#llm training
One-click install
npx skills add https://github.com/carterwayneskhizeine/hermes-agent-windows-R --skill nemo-curator-carterwayneskhizeine
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: nemo-curator
Source: https://github.com/carterwayneskhizeine/hermes-agent-windows-R/tree/main/optional-skills/mlops/nemo-curator
Command: npx skills add https://github.com/carterwayneskhizeine/hermes-agent-windows-R --skill nemo-curator-carterwayneskhizeine

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Needing clean, high-quality training data but facing slow CPU deduplication, noisy web content, and the risk of leaking sensitive information.

Core Features & Use Cases

  • GPU-accelerated deduplication (fuzzy, semantic, exact) to remove duplicates and near-duplicates across massive corpora.
  • Quality filtering with 30+ heuristics to drop low-value, repetitive, or malformed documents before training.
  • PII redaction and NSFW detection to reduce data leakage and unsafe content in downstream LLM training.
  • Multimodal support (text/image/video/audio) for building higher-quality multimodal datasets from web-scale sources.

Example use case: clean an 8TB web-crawl dataset by applying heuristic quality filters, running GPU fuzzy deduplication, then redacting PII and filtering NSFW items before exporting to Parquet for model training.

Quick Start

Ask the AI to help you curate your dataset by running a GPU-enabled NeMo Curator pipeline that performs quality filtering, fuzzy deduplication, and PII redaction, then saves the cleaned output as Parquet.

Frequently Asked Questions about nemo-curator

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run GPU-accelerated deduplication on large text datasets?▼

You can clean web-crawl datasets by applying 30+ heuristic quality filters to drop low-value, repetitive, or malformed documents before exporting the remaining data to Parquet.

How do I clean Common Crawl data for LLM training?▼

You can clean web-crawl datasets by applying heuristic quality filters to drop low-value, repetitive, or malformed documents before exporting the remaining data to Parquet for LLM training.

Can I redact PII and detect NSFW content in multimodal corpora?▼

You can load common structured inputs like Parquet, JSONL, and CSV files to apply quality filtering and deduplication, producing cleaned Parquet outputs for downstream model training.

Do I need RAPIDS to prepare datasets for multimodal LLM training?▼

GPU acceleration is necessary for large-scale corpus deduplication and web-crawl cleaning, as CPU-based methods face significant throughput bottlenecks when processing massive datasets.

What is the best way to remove duplicates from an 8TB web-crawl dataset?▼

The pipeline accepts Parquet, JSONL, and CSV inputs, applying configurable threshold and heuristic parameters to filter content, and exports the final cleaned corpus as Parquet files.

Related Skills

Explore More Education & Research Skills→
thisismynewfmail-uithisismynewfmail-ui

nemo-curator

Curate multimodal datasets with quality filtering, deduplication, and PII redaction.

Community
Advanced
twjarviszyra-webtwjarviszyra-web
3

axolotl

Configure Axolotl YAML for LLM fine-tuning and evaluation workflows.

Community
Advanced
ihatesea69ihatesea69
3

huggingface-tokenizers

Train custom BPE, WordPiece, and Unigram tokenizers with Rust-based implementations.

Community
Intermediate
zeronx798zeronx798

nemo-curator

Cure and deduplicate large-scale multimodal datasets for LLM training on GPU clusters.

Community
Advanced