nemo-curator

Name: nemo-curator
Availability: InStock
Author: adm-humanerd

Curate multimodal training data with GPU-accelerated deduplication and filtering.

Updated Aug 27, 2026

Search Tags:#multimodal #deduplication #gpu-accelerated #pii-redaction #data-curation #nemo-curator #nsfw-filtering

Ready to run

npx skills add https://github.com/adm-humanerd/drewgent --skill nemo-curator-adm-humanerd

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: nemo-curator
Source: https://github.com/adm-humanerd/drewgent/tree/main/optional-skills/mlops/nemo-curator
Command: npx skills add https://github.com/adm-humanerd/drewgent --skill nemo-curator-adm-humanerd

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

GPU-accelerated data curation for LLM training enables scalable deduplication, quality filtering, and multimodal data preparation across GPU clusters.

Core Features & Use Cases

Fuzzy deduplication (GPU-accelerated, 16× faster on large datasets)
Semantic deduplication
PII redaction
NSFW detection
30+ quality filters
Multimodal curation (text, image, video, audio)
Scales across RAPIDS GPUs
Use case: prepare training data from web scrapes, deduplicate, filter, and export to Parquet for model training.

Quick Start

Install Nemo Curator and run a GPU-accelerated data-curation pipeline on your dataset to produce a curated Parquet dataset.

Frequently Asked Questions about nemo-curator

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I deduplicate large-scale web-scraped datasets for LLM training?▼

GPU-accelerated data curation pipelines use exact, fuzzy, and semantic deduplication to clean web-scraped datasets. By scaling across RAPIDS GPU clusters, this approach processes text, images, video, and audio to output high-quality Parquet or JSONL data for LLM training.

What is the best way to filter NSFW content and redact PII from training data?▼

Specialized GPU-accelerated curation pipelines provide built-in PII redaction and NSFW detection modules. These features run alongside over 30 quality filters to safely sanitize multimodal training data before exporting it for model training.

Do I need the NVIDIA RAPIDS stack to run GPU-accelerated data curation?▼

Yes, the NVIDIA RAPIDS stack (including cudf and dask) and Nemo-curator dependencies are required. These provide the necessary GPU-acceleration to run fuzzy and semantic deduplication efficiently across large-scale multimodal datasets.

How does fuzzy deduplication work on GPU clusters for multimodal data?▼

Fuzzy deduplication on GPU clusters identifies and removes near-duplicate records across large multimodal datasets. This GPU-accelerated approach operates up to 16 times faster than CPU-based methods, scaling efficiently across RAPIDS environments.

Can I use GPU-accelerated curation for text, image, video, and audio sources?▼

Yes, GPU-accelerated curation supports multimodal data sources including text, image, video, and audio. It applies quality filtering, deduplication, and PII redaction across these formats to prepare web-scraped data for LLM training pipelines.

How do I export curated training data to Parquet for model training pipelines?▼

You can export curated training data to Parquet or JSONL formats after applying GPU-accelerated deduplication and quality filtering. The curation pipeline processes multimodal sources and outputs these structured files directly for LLM training.