skills.rest
KarlinskyS
KarlinskySCOMMUNITY
Explore all skills by this author →
GitHub

nemo-curator

Deduplicate, filter, redact PII, and remove NSFW content from large multimodal corpora.

Updated Apr 10, 2026
Search Tags:#multimodal#gpu#deduplication#quality filtering#data curation#rapids#pii redaction
One-click install
npx skills add https://github.com/KarlinskyS/hermesSkills --skill nemo-curator-karlinskys
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: nemo-curator
Source: https://github.com/KarlinskyS/hermesSkills/tree/main/mlops/evaluation/nemo-curator
Command: npx skills add https://github.com/KarlinskyS/hermesSkills --skill nemo-curator-karlinskys

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

NeMo Curator streamlines preparing high-quality training datasets for large language and multimodal models by removing duplicates, filtering low-quality content, redacting PII, and detecting unsafe material at GPU scale.

Core Features & Use Cases

  • High-performance deduplication: Exact, fuzzy (MinHash+LSH), and semantic deduplication to remove duplicates from large corpora.
  • Quality and safety filtering: 30+ heuristic filters plus classifier-based quality and NSFW filtering to enforce dataset standards.
  • Multimodal and scalable: Supports text, image, audio, and video with RAPIDS/Dask multi-GPU scaling and Parquet/JSONL inputs for production pipelines.
  • PII redaction and compliance: Built-in PII redactors and options to anonymize or redact sensitive entities.
  • Use Case: Clean and deduplicate Common Crawl or web-scraped corpora, then output curated Parquet for LLM training.

Quick Start

Run a GPU-accelerated curation pipeline to deduplicate, quality-filter, and redact PII from a Parquet Common Crawl dataset and write the curated Parquet output.

Frequently Asked Questions about nemo-curator

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I deduplicate large-scale multimodal datasets for LLM training?▼

Deduplicate large-scale multimodal datasets by applying exact, fuzzy MinHash+LSH, and semantic deduplication techniques. This GPU-accelerated curation removes exact and near-duplicate content from large corpora using RAPIDS and Dask for multi-GPU scaling.

What is the best way to redact PII from Common Crawl data for LLM training?▼

Redact PII from Common Crawl data using built-in PII redactors that identify and anonymize sensitive entities. This ensures compliance during GPU-accelerated preprocessing of web scrapes before outputting curated Parquet files for LLM training.

Does this GPU-accelerated data curation pipeline support Parquet and JSONL inputs?▼

Yes, the GPU-accelerated data curation pipeline supports both Parquet and JSONL inputs. It processes these formats natively using RAPIDS and Dask to scale across multiple GPUs for production-level LLM training pipelines.

Can I apply heuristic and classifier-based quality filtering to web-scraped corpora?▼

Yes, you can apply heuristic and classifier-based quality filtering to web-scraped corpora. The curation pipeline includes over 30 heuristic filters and classifier-based quality enforcement to remove low-quality content from large datasets.

When do I need semantic deduplication for multimodal model preparation?▼

You need semantic deduplication for multimodal model preparation when removing near-duplicate records based on meaning rather than exact string matching. This is required alongside exact and fuzzy deduplication to ensure high-quality training corpora.

How do I remove NSFW content from large-scale training datasets?▼

Remove NSFW content from large-scale training datasets using classifier-based safety filtering. This detects and removes unsafe material during GPU-accelerated preprocessing, ensuring the final curated corpora meets safety standards.

Related Skills

Explore More Data & Analytics Skills→
Clay-HHKClay-HHK
2

nemo-curator

Curate and deduplicate large-scale multimodal corpora with GPU acceleration.

Community
Advanced
thisismynewfmail-uithisismynewfmail-ui

nemo-curator

Curate multimodal datasets with quality filtering, deduplication, and PII redaction.

Community
Advanced
blueskies1818blueskies1818
1

nemo-curator

Accelerates data curation for LLM training with GPU-based deduplication, filtering, and PII redaction.

Community
Advanced
OpenLAIROpenLAIR
1.0k

nemo-curator

Automate GPU-accelerated data curation for LLM training datasets.

Official
Advanced