skills.rest
Chris-Chai-Minjae
Chris-Chai-MinjaeCOMMUNITY
Explore all skills by this author →
GitHub

nemo-curator

Automate GPU-accelerated curation of multimodal datasets with deduplication and filtering.

Updated Apr 23, 2026
Search Tags:#multimodal#gpu#deduplication#pii-redaction#llm-training#data-curation#nsfw-detection
One-click install
npx skills add https://github.com/Chris-Chai-Minjae/hermes-agent-r1-bridge --skill nemo-curator-chris-chai-minjae
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: nemo-curator
Source: https://github.com/Chris-Chai-Minjae/hermes-agent-r1-bridge/tree/main/optional-skills/mlops/nemo-curator
Command: npx skills add https://github.com/Chris-Chai-Minjae/hermes-agent-r1-bridge --skill nemo-curator-chris-chai-minjae

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

GPU-accelerated data curation pipelines for high-quality multimodal training data, reducing noise, duplicates, and unsafe content at scale.

Core Features & Use Cases

  • GPU-accelerated data curation: scalable processing across RAPIDS-enabled GPUs for large corpora.
  • Deduplication options: exact, fuzzy, and semantic to remove near-duplicate content efficiently.
  • Quality & safety: 30+ heuristics for quality filtering, PII redaction, and NSFW detection.
  • Multimodal support: text, image, video, and audio data curated for high-quality training datasets.
  • Use case: prepare RedPajama/The Pile-style training data, clean web crawls, or deduplicate massive archives.

Quick Start

Run Nemo Curator on a GPU-enabled cluster to ingest your dataset, perform deduplication, quality filtering, PII redaction, and NSFW detection, and export cleaned Parquet/JSONL outputs.

Frequently Asked Questions about nemo-curator

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I deduplicate large text and image datasets for LLM training?▼

PII redaction for training datasets is handled using GPU-accelerated processing to automatically detect and remove sensitive information from massive text corpora prior to model training.

Does GPU-accelerated data curation support multimodal video and audio corpora?▼

GPU-accelerated data curation supports multimodal datasets, processing text, image, video, and audio data to prepare high-quality training inputs using RAPIDS-enabled GPUs.

What's the best way to clean web crawls and prepare RedPajama-style training data?▼

The best way to clean web crawls is applying 30+ heuristics for quality filtering, NSFW detection, and PII redaction, then exporting the curated results as Parquet or JSONL files.

Can I use RAPIDS for NSFW detection and quality filtering on GPU clusters?▼

You can use RAPIDS-enabled GPU clusters to perform NSFW detection and quality filtering, applying scalable processing heuristics to reduce noise and unsafe content at scale.

Do I need a GPU-enabled cluster to run multimodal data curation pipelines?▼

A GPU-enabled cluster is required to run these multimodal data curation pipelines, as the processing leverages RAPIDS-enabled GPUs to handle large-scale text, image, video, and audio corpora efficiently.

Related Skills

Explore More Data & Analytics Skills→
GregsGreyCodeGregsGreyCode
6

nemo-curator

Curate multimodal training data with GPU-accelerated deduplication, quality filtering, and redaction.

Community
Advanced
jolequesjoleques

dataset-synthesizer

Merge interaction logs and documentation into Vertex AI Gemini JSONL fine-tuning datasets.

Community
Advanced
founderphantomfounderphantom

nemo-curator

Automates GPU-accelerated cleaning and deduplication of multimodal LLM training data with fuzzy/semantic dedup, PII redaction, NSFW detection via Nemo Curator and optional RAPIDS stack.

Community
Advanced
Yangel-hideYangel-hide

nemo-curator

Automate GPU-accelerated deduplication and quality filtering for multimodal LLM training data.

Community
Advanced