skills.rest
eddielueng
eddieluengCOMMUNITY
Explore all skills by this author →
GitHub

nemo-curator

Filter, deduplicate, and redact large-scale multimodal datasets with GPU acceleration.

1|Updated Apr 14, 2026
Search Tags:#data curation#rapids#pii redaction#llm training data#nsfw detection#gpu deduplication#multimodal filtering
One-click install
npx skills add https://github.com/eddielueng/hermes-agent-zh --skill nemo-curator-eddielueng
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: nemo-curator
Source: https://github.com/eddielueng/hermes-agent-zh/tree/main/optional-skills/mlops/nemo-curator
Command: npx skills add https://github.com/eddielueng/hermes-agent-zh --skill nemo-curator-eddielueng

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires nemo-curator, cudf, dask, rapids, and includes references (resource) components.

What problem does it solve?

Preparing high-quality, deduplicated, and safe training datasets for LLMs from noisy large-scale sources is slow and error-prone, especially when the corpus is multimodal and needs privacy protection.

Core Features & Use Cases

  • GPU-accelerated curation for large text/image/video/audio datasets using NVIDIA tooling and distributed GPU execution.
  • Fuzzy, exact, and semantic deduplication to remove duplicates and near-duplicates (including embedding-based similarity removal).
  • Quality filtering, PII redaction, and NSFW detection to improve dataset trustworthiness before model training.
  • Use case: Curate Common Crawl-derived datasets into a cleaner multimodal corpus by filtering low-quality samples, deduplicating aggressively, redacting sensitive data, and dropping unsafe content.

Quick Start

Use the nemo-curator skill to deduplicate and filter your raw dataset with GPU acceleration, including PII redaction and NSFW removal, then save the curated output to Parquet.

Frequently Asked Questions about nemo-curator

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I curate large-scale multimodal LLM training data on GPUs?▼

PII redaction for LLM training datasets is handled by integrated pipelines that identify and mask sensitive information within your corpus before model training, ensuring dataset safety and trustworthiness.

Does GPU deduplication support fuzzy and semantic matching for web-crawled corpora?▼

GPU deduplication supports fuzzy, exact, and semantic matching by using embedding-based similarity removal to aggressively deduplicate web-crawled corpora and prepare cleaner multimodal datasets.

Can I use this to run NSFW detection and quality filtering on a Common Crawl dataset?▼

You can run NSFW detection and quality filtering on a Common Crawl dataset to drop unsafe content and low-quality samples, ultimately saving the curated output to Parquet files for downstream use.

Do I need RAPIDS and Dask to perform GPU-accelerated data curation?▼

You need RAPIDS and Dask to perform GPU-accelerated data curation because the Skill requires a GPU-capable execution environment with distributed GPU support to process large text, image, video, and audio datasets.

What is the best way to prepare safe multimodal datasets for foundation-model training?▼

The best way to prepare safe multimodal datasets for foundation-model training is to apply safety-oriented preprocessing that combines quality filtering, fuzzy deduplication, PII redaction, and NSFW removal pipelines.

Related Skills

Explore More Data & Analytics Skills→
Clay-HHKClay-HHK
2

nemo-curator

Curate and deduplicate large-scale multimodal corpora with GPU acceleration.

Community
Advanced
thisismynewfmail-uithisismynewfmail-ui

nemo-curator

Curate multimodal datasets with quality filtering, deduplication, and PII redaction.

Community
Advanced
ovachieverovachiever
52

nemo-curator

Curate multimodal LLM training data with GPU-accelerated filtering and deduplication.

Community
Advanced
twjarviszyra-webtwjarviszyra-web
3

nemo-curator

Clean and deduplicate large-scale training datasets with GPU acceleration.

Community
Advanced