skills.rest
jason660519
jason660519COMMUNITY
Explore all skills by this author →
GitHub

nemo-curator

Curate multimodal training datasets with GPU-accelerated deduplication, filtering, and redaction.

Updated May 11, 2026
Search Tags:#multimodal#deduplication#gpu-acceleration#llm-training#data-curation#nemo-curator
One-click install
npx skills add https://github.com/jason660519/Project-Manager --skill nemo-curator-jason660519
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: nemo-curator
Source: https://github.com/jason660519/Project-Manager/tree/main/hermes-agent/optional-skills/mlops/nemo-curator
Command: npx skills add https://github.com/jason660519/Project-Manager --skill nemo-curator-jason660519

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires nemo-curator, cudf, dask, rapids, and includes references (resource) components.

What problem does it solve?

GPU-accelerated data curation for large multimodal datasets used to train LLMs, reducing manual data wrangling and enabling scalable, reproducible data preparation.

Core Features & Use Cases

  • Multimodal data curation: efficiently process text, image, video, and audio data for training pipelines.
  • Deduplication & quality filtering: fast fuzzy deduplication (16× faster on GPU), semantic deduplication, and 30+ quality filters to improve dataset quality.
  • PII/NSFW redaction and safety checks: automatically redact sensitive information and filter inappropriate content.

Quick Start

Run NeMo Curator on your dataset to filter, deduplicate, and redact data to prepare high-quality training material.

Frequently Asked Questions about nemo-curator

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I deduplicate large multimodal datasets for LLM training?▼

GPU-accelerated data curation enables fast deduplication of large multimodal datasets for LLM training. It uses NVIDIA RAPIDS to perform exact, fuzzy, and semantic deduplication on text, image, video, and audio sources.

Can I use GPU acceleration for PII redaction and data filtering in Python?▼

Yes, GPU acceleration supports PII redaction and data filtering in Python. It automatically redacts sensitive information and applies 30+ quality filters to clean large-scale training data.

Do I need NVIDIA GPUs with RAPIDS support to run data curation pipelines?▼

Yes, NVIDIA GPUs with RAPIDS support are required to run these GPU-accelerated data curation pipelines. The execution relies on Python-based environments using RAPIDS, cuDF, and Dask to scale data processing.

What is the best way to clean web crawled data for machine learning?▼

GPU-accelerated data curation is a highly efficient way to clean web crawled data for machine learning. It applies quality filters, PII and NSFW handling, and fuzzy deduplication to create reproducible, high-quality datasets.

Does fuzzy deduplication on GPU perform faster than CPU-based processing?▼

Fuzzy deduplication on GPU performs up to 16 times faster than standard CPU-based processing. This acceleration enables scalable and reproducible data preparation across massive multimodal corpora.

When should I use semantic deduplication in my data preparation workflow?▼

Semantic deduplication should be used in data preparation workflows when removing contextually similar data points is necessary to improve dataset quality. It complements exact and fuzzy deduplication for multimodal LLM training data.

Related Skills

Explore More Data & Analytics Skills→
galeepgaleep

optimize-for-gpu

Accelerate Python code execution on NVIDIA GPUs using CuPy, Numba, and cuDF.

Community
Advanced
nelohenriqnelohenriq
1

nemo-curator

Curate large multimodal datasets with GPU-accelerated filtering, deduplication, and PII redaction.

Community
Advanced
SignmanalSignmanal
1

nemo-curator

Curate large-scale LLM training datasets with GPU-accelerated filtering and deduplication.

Community
Advanced
heysuhasheysuhas
2

nemo-curator

Clean and deduplicate large-scale multimodal datasets for LLM training on GPU clusters.

Community
Advanced