665,000+ skills indexed

tadod12COMMUNITY

Explore all skills by this author →

Verified:GitHub

nemo-curator

Automate multimodal dataset curation with deduplication, quality filtering, PII redaction, and NSFW detection.

Updated Mar 18, 2026

Search Tags:#multimodal #deduplication #gpu-acceleration #data-curation #nemo-curator #nsfw-filtering #PII-redaction

One-click install

npx skills add https://github.com/tadod12/fraud-detection-research --skill nemo-curator-tadod12

Or copy as Structured Prompt for Agent▼

Please help me install this Agent Skill.
Skill: nemo-curator
Source: https://github.com/tadod12/fraud-detection-research/tree/main/.agent/skills/05-data-processing/nemo-curator
Command: npx skills add https://github.com/tadod12/fraud-detection-research --skill nemo-curator-tadod12

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Curating high-quality training data for multimodal LLMs is slow and error-prone. Nemo Curator accelerates this workflow by providing GPU-accelerated deduplication, quality filtering, PII redaction, and NSFW detection across text, image, video, and audio data.

Core Features & Use Cases

Fuzzy and semantic deduplication on large datasets to reduce data noise.
Multi-modal quality filtering and PII redaction to improve data safety.
GPU-accelerated scaling across clusters for faster data curation.
Use cases: cleaning web crawls for open-source model training, preparing RedPajama/The Pile-like corpora, and validating data quality.

Quick Start

Install Nemo Curator and run a multi-modal data curation pipeline on your dataset to produce a clean, deduplicated training set.

Frequently Asked Questions about nemo-curator

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I clean and deduplicate large web-scraped datasets for LLM training?▼

You can clean web-scraped datasets for LLM training by automating multimodal data curation, which removes duplicates, filters low-quality content, redacts PII, and detects NSFW material across text, image, audio, and video.

Does GPU acceleration help with multimodal data curation and PII redaction?▼

GPU acceleration scales multimodal data curation across clusters using RAPIDS, enabling fast fuzzy and semantic deduplication, quality filtering, and PII redaction on large web-scraped corpora.

Can I use this data curation pipeline for both text and image modalities?▼

Yes, the data curation pipeline supports multimodal datasets across text, image, audio, and video modalities, applying multi-stage filtering and deduplication to prepare standardized outputs for model training.

What output formats are produced after GPU-accelerated dataset filtering and deduplication?▼

After GPU-accelerated dataset filtering, deduplication, and NSFW detection, the data curation pipeline produces standardized Parquet and JSONL outputs that are ready for multimodal LLM training.

What is the best way to prepare a RedPajama-like corpus for open-source model training?▼

The best way to prepare a RedPajama-like corpus is to run a multimodal data curation pipeline that performs fuzzy deduplication, multi-stage quality filtering, and PII redaction across GPU clusters.

Do I need GPU clusters to run multi-stage data filtering and NSFW detection?▼

GPU clusters are required to achieve fast, end-to-end data cleaning and scaling. The pipeline uses RAPIDS for GPU-accelerated processing to handle large web-scraped corpora efficiently during multi-stage filtering and NSFW detection.

Related Skills

Explore More Data & Analytics Skills→

nemo-curator

Curate and deduplicate multimodal datasets with GPU-accelerated pipelines.

carterwayneskhizeine

nemo-curator

Curate noisy training data with GPU-accelerated deduplication and PII redaction.

nemo-curator

Curate multimodal LLM training data with GPU-accelerated filtering and deduplication.

nemo-curator

Deduplicate and filter LLM training datasets with GPU acceleration.