665,000+ skills indexed

founderphantomCOMMUNITY

Explore all skills by this author →

Verified:GitHub

nemo-curator

Automates GPU-accelerated cleaning and deduplication of multimodal LLM training data with fuzzy/semantic dedup, PII redaction, NSFW detection via Nemo Curator and optional RAPIDS stack.

Updated Apr 1, 2026

Search Tags:#multimodal #deduplication #data-cleaning #gpu-acceleration #llm-training #data-curation #nemo-curator

One-click install

npx skills add https://github.com/founderphantom/zola-agent --skill nemo-curator-founderphantom

Or copy as Structured Prompt for Agent▼

Please help me install this Agent Skill.
Skill: nemo-curator
Source: https://github.com/founderphantom/zola-agent/tree/main/optional-skills/mlops/nemo-curator
Command: npx skills add https://github.com/founderphantom/zola-agent --skill nemo-curator-founderphantom

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

GPU-accelerated data curation streamlines the preparation of high-quality training data for LLMs by automating filtering, deduplication, and multi-modal content handling.

Core Features & Use Cases

Fuzzy deduplication across large datasets with GPU acceleration.
Quality filtering using 30+ heuristics for text, image, video, and audio data.
Semantic deduplication and PII redaction to improve data safety.
NSFW detection and scalable deployment across RAPIDS-enabled GPUs.
Typical use cases include cleaning web crawls, deduplicating large corpora, and preparing multimodal datasets for model training.

Quick Start

Install Nemo Curator and run a GPU-accelerated data-curation pipeline to produce a curated dataset.

Frequently Asked Questions about nemo-curator

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I deduplicate large multimodal datasets for LLM training?▼

You can deduplicate large multimodal datasets using GPU-accelerated fuzzy and semantic deduplication. This process scales across RAPIDS-enabled GPU clusters to clean text, image, video, and audio data for model training.

Do I need RAPIDS and GPU-enabled compute for GPU-accelerated data curation?▼

Yes, GPU-accelerated data curation requires GPU-enabled compute and the Nemo Curator framework. The RAPIDS stack is an optional dependency used to scale data cleaning and deduplication tasks across GPU clusters.

What is the best way to filter NSFW content and redact PII from training corpora?▼

The best way to filter NSFW content and redact PII from training corpora is using automated data curation pipelines. These pipelines apply quality filtering and safety heuristics to clean web crawls before model training.

How does GPU acceleration improve data cleaning for LLM training?▼

GPU acceleration improves data cleaning by distributing fuzzy and semantic deduplication tasks across RAPIDS-enabled clusters. This allows the framework to process large multimodal datasets much faster than CPU-based methods.

What quality filtering heuristics are available for text, image, video, and audio data?▼

GPU-accelerated data curation includes over 30 quality filtering heuristics for text, image, video, and audio data. These heuristics automate the cleaning of web crawls and prepare high-quality multimodal datasets for LLM training.

Can I use Nemo Curator to clean web crawls before preparing multimodal datasets?▼

Yes, you can use Nemo Curator to clean web crawls and prepare multimodal datasets. It automates filtering, deduplication, PII redaction, and NSFW detection across text, image, video, and audio formats for LLM training.

Related Skills

Explore More Data & Analytics Skills→

nemo-curator

Curate multimodal datasets for LLM training with GPU-accelerated pipelines.

fine-tuning-with-trl

Align large language models via reinforcement learning and preference-based fine-tuning workflows.

nemo-curator

Automate GPU-accelerated data curation for LLM training with RAPIDS.

nemo-curator

Automates GPU-accelerated data curation for LLM training datasets.