665,000+ skills indexed

anilcan-karaCOMMUNITY

Explore all skills by this author →

Verified:Website GitHub

nemo-curator

Automate deduplication, quality filtering, PII redaction, and NSFW detection for multimodal LLM training data.

Updated Jun 17, 2026

Search Tags:#multimodal #deduplication #gpu-acceleration #pii-redaction #data-curation #nsfw-detection #nemo-curator

One-click install

npx skills add https://github.com/anilcan-kara/nozich-agent --skill nemo-curator-anilcan-kara

Or copy as Structured Prompt for Agent▼

Please help me install this Agent Skill.
Skill: nemo-curator
Source: https://github.com/anilcan-kara/nozich-agent/tree/main/optional-skills/mlops/nemo-curator
Command: npx skills add https://github.com/anilcan-kara/nozich-agent --skill nemo-curator-anilcan-kara

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires nemo-curator, cudf, dask, rapids, and includes references (resource) components.

What problem does it solve?

GPU-accelerated data curation for LLM training helps teams prepare high-quality training datasets efficiently, reducing time and cost.

Core Features & Use Cases

Supports text, image, video, and audio data with fuzzy deduplication, quality filtering, semantic deduplication, PII redaction, and NSFW detection.
Scales across GPUs with RAPIDS to handle multi-terabyte data and complex pipelines for data cleaning and de-duplication.
Use case: Prepare cleaned, deduplicated, and policy-compliant training data for large language models and multimodal models.

Quick Start

Run Nemo Curator on your dataset to apply quality filters and deduplicate content across multimodal data.

Frequently Asked Questions about nemo-curator

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I automate multimodal data curation for LLM training datasets?▼

GPU-accelerated data curation uses RAPIDS to scale processing across GPUs, handling multi-terabyte datasets efficiently. It applies fuzzy and semantic deduplication, quality filtering, PII redaction, and NSFW detection to reduce time and cost.

Can I use RAPIDS and Dask for GPU-accelerated deduplication on large datasets?▼

Yes, RAPIDS and Dask enable GPU-accelerated processing to handle multi-terabyte data pipelines. This integration allows you to perform complex deduplication and data cleaning tasks at scale much faster than CPU-based methods.

Does GPU-accelerated data curation support PII redaction and NSFW detection for images and videos?▼

Yes, data curation supports text, image, video, and audio data. It applies modular filters to perform PII redaction and NSFW detection, ensuring your large language model and multimodal training pipelines remain policy-compliant.

What is the best way to prepare cleaned and deduplicated training data for multimodal models?▼

The best way to prepare cleaned training data is to apply modular quality filters and fuzzy deduplication across your datasets. This approach ensures high-quality, policy-compliant inputs for large language models and multimodal models.

Do I need Parquet or JSONL formats for GPU-accelerated data curation pipelines?▼

You should use Parquet or JSONL formats for your data curation pipelines. The system integrates with these workflows to seamlessly apply quality filtering, deduplication, and PII redaction across your multimodal data.

Related Skills

Explore More Data & Analytics Skills→

nemo-curator

Curate large multimodal datasets for LLM training with GPU-accelerated deduplication and filtering.

axolotl

Fine-tune Large Language Models with Axolotl using YAML configurations.

nemo-curator

Deduplicate and filter large-scale LLM training data with GPU-optimized pipelines.

nemo-curator

Deduplicate and filter LLM training datasets with GPU acceleration.