skills.rest
Yangel-hide
Yangel-hideCOMMUNITY
Explore all skills by this author →
GitHub

nemo-curator

Automate GPU-accelerated deduplication and quality filtering for multimodal LLM training data.

Updated May 3, 2026
Search Tags:#multimodal#deduplication#gpu-acceleration#rapids#pii-redaction#data-curation#nsfw-detection
One-click install
npx skills add https://github.com/Yangel-hide/video-production-planner-agent --skill nemo-curator-yangel-hide
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: nemo-curator
Source: https://github.com/Yangel-hide/video-production-planner-agent/tree/main/optional-skills/mlops/nemo-curator
Command: npx skills add https://github.com/Yangel-hide/video-production-planner-agent --skill nemo-curator-yangel-hide

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Efficiently curates high-quality training data for multimodal LLMs, accelerating GPU-accelerated workflows and improving dataset quality.

Core Features & Use Cases

  • Multi-modal curation: supports text, image, video, and audio data.
  • GPU-accelerated deduplication: exact, fuzzy, and semantic deduplication scale across large datasets.
  • Quality filtering: 30+ heuristics to remove low-quality or toxic content.
  • Safety tools: PII redaction and NSFW detection to protect privacy and compliance.
  • Scales across clusters: RAPIDS-enabled GPU acceleration for multi-node pipelines.

Quick Start

Install Nemo Curator and run a sample pipeline on your dataset to deduplicate, filter quality, and redact PII.

Frequently Asked Questions about nemo-curator

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I perform GPU-accelerated deduplication on large multimodal datasets for LLM training?▼

GPU-accelerated deduplication for LLM training datasets is automated using RAPIDS multi-GPU scaling, supporting exact, fuzzy, and semantic deduplication across text, image, video, and audio data.

What is the best way to redact PII and detect NSFW content in training data?▼

PII redaction and NSFW detection in training data are handled through integrated safety tools, protecting privacy and ensuring compliance during multimodal dataset preparation.

Can I use RAPIDS for multi-node scaling during data curation?▼

Yes, RAPIDS enables multi-node GPU acceleration for data curation pipelines, allowing rapid processing and quality filtering of large-scale datasets across clusters.

How many heuristics are available for quality filtering in multimodal data curation?▼

Multimodal data curation provides over 30 heuristics for quality filtering, designed to remove low-quality or toxic content from large-scale training datasets.

Does data curation for LLM training support both text and video modalities?▼

Data curation for LLM training supports text, image, video, and audio modalities, enabling comprehensive multimodal dataset preparation through GPU-accelerated workflows.

Related Skills

Explore More Data & Analytics Skills→
jleechanorgjleechanorg

nemo-curator

Automate GPU-accelerated filtering, deduplication, PII redaction, and NSFW detection on multimodal datasets.

Official
Advanced
HomericIntelligenceHomericIntelligence
18

train-model

Execute model training pipelines with forward and backward passes, gradient updates, and checkpoint management.

Official
Advanced
tassiovaletassiovale
13

optimize-for-gpu

Accelerates Python numerical and ML code on NVIDIA GPUs using CUDA and RAPIDS libraries.

Community
Advanced
SignmanalSignmanal
1

nemo-curator

Curate large-scale LLM training datasets with GPU-accelerated filtering and deduplication.

Community
Advanced