skills.rest
box755
box755COMMUNITY
Explore all skills by this author →
GitHub

nemo-curator

Curate multimodal web-scale datasets with quality filtering, deduplication, and PII redaction.

Updated Aug 23, 2026
Search Tags:#deduplication#quality filtering#pii redaction#gpu data curation#nvidia nem o curator#multimodal dataset#rapids dask
One-click install
npx skills add https://github.com/box755/simlens-research --skill nemo-curator-box755
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: nemo-curator
Source: https://github.com/box755/simlens-research/tree/main/skill-packs/AI-Research-SKILLs/05-data-processing/nemo-curator
Command: npx skills add https://github.com/box755/simlens-research --skill nemo-curator-box755

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires nemo-curator, cudf, dask, rapids, and includes references (resource) components.

What problem does it solve?

NeMo Curator helps you clean and deduplicate large-scale web and multimodal datasets so LLM training data is higher quality, less redundant, and safer.

Core Features & Use Cases

  • GPU-accelerated curation pipeline for text, images, video, and audio, enabling fast preprocessing at scale.
  • Fuzzy and semantic deduplication to remove exact, near-duplicate, and meaning-level duplicates using GPU-friendly approaches.
  • Quality filtering, PII redaction, and NSFW detection to reduce low-quality, unsafe, and sensitive-content contamination before training.
  • Common use case: preparing curated corpora from web scrapes (e.g., Common Crawl) before training or fine-tuning a multimodal LLM.

Quick Start

Use the NeMo Curator Skill to run GPU-accelerated quality filtering, fuzzy deduplication, and PII redaction over your Parquet dataset and write the curated output to a new Parquet directory.

Frequently Asked Questions about nemo-curator

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I deduplicate large-scale multimodal datasets for LLM training?▼

Deduplicate large-scale multimodal datasets by applying GPU-accelerated fuzzy and semantic deduplication to remove exact, near-duplicate, and meaning-level matches from text, image, video, and audio data.

What is the best way to redact PII and filter unsafe content from Common Crawl data?▼

Redact PII and filter unsafe content from Common Crawl data by running GPU-accelerated quality filtering, PII redaction, and NSFW detection to reduce low-quality and sensitive-content contamination.

Can I use RAPIDS and Dask for GPU-accelerated data curation?▼

Yes, you can use RAPIDS and Dask for GPU-accelerated data curation to meet distributed processing and performance needs when curating web-scale multimodal corpora.

Do I need Parquet datasets to run quality filtering and deduplication?▼

You need Parquet datasets to run GPU-accelerated quality filtering and fuzzy deduplication, writing the curated output to a new Parquet directory for downstream LLM training.

Does GPU data curation support preprocessing pipelines for text, image, video, and audio?▼

GPU data curation supports preprocessing pipelines for text, image, video, and audio, enabling fast multimodal quality screening and large-scale deduplication before LLM fine-tuning.

When should I use fuzzy deduplication instead of exact deduplication for LLM training data?▼

Use fuzzy deduplication instead of exact deduplication to remove near-duplicates and meaning-level matches, ensuring LLM training data is less redundant and higher quality.

Related Skills

Explore More Data & Analytics Skills→
GregsGreyCodeGregsGreyCode
6

nemo-curator

Curate multimodal training data with GPU-accelerated deduplication, quality filtering, and redaction.

Community
Advanced
shuff57shuff57

axolotl

Configure Axolotl YAML files for LoRA, DPO, GRPO, and multimodal fine-tuning.

Community
Advanced
LynxLabVNLynxLabVN

nemo-curator

Accelerate LLM training dataset curation with GPU computing and RAPIDS.

Official
Advanced
NITISH-gitbitNITISH-gitbit

nemo-curator

Deduplicate and filter LLM training datasets with GPU acceleration.

Community
Advanced