skills.rest
thisismynewfmail-ui
thisismynewfmail-uiCOMMUNITY
Explore all skills by this author →
GitHub

nemo-curator

Curate multimodal datasets with quality filtering, deduplication, and PII redaction.

Updated Apr 12, 2026
Search Tags:#deduplication#gpu-acceleration#pii-redaction#data-curation#nsfw-detection#nemo-curator
One-click install
npx skills add https://github.com/thisismynewfmail-ui/Monika-agent --skill nemo-curator-thisismynewfmail-ui
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: nemo-curator
Source: https://github.com/thisismynewfmail-ui/Monika-agent/tree/main/optional-skills/mlops/nemo-curator
Command: npx skills add https://github.com/thisismynewfmail-ui/Monika-agent --skill nemo-curator-thisismynewfmail-ui

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Speeds up the preparation and cleaning of high-quality multimodal datasets for large language model training by automating deduplication, quality filtering, and sensitive-data redaction.

Core Features & Use Cases

  • High-speed exact, fuzzy, and semantic deduplication for text and multimodal data
  • Quality filtering across 30+ heuristics to remove low-quality content
  • PII redaction and NSFW detection to protect privacy and safety
  • Multimodal support (text, image, video, audio) with GPU-accelerated pipelines
  • Scalable, GPU-accelerated processing across clusters for large datasets
  • Ready-to-use pipelines for preparing training data for Nemotron / LLMs

Quick Start

Start by configuring Nemo Curator on your dataset to perform quality filtering, deduplication, and multimodal curation across text, images, videos, and audio.

Frequently Asked Questions about nemo-curator

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I accelerate data deduplication and quality filtering for LLM training datasets?▼

GPU-accelerated data curation speeds up dataset preparation by applying exact, fuzzy, and semantic deduplication alongside quality filtering across text, image, video, and audio corpora.

Can I use GPU-accelerated data curation with Parquet and JSONL input formats?▼

Yes, GPU-accelerated data curation supports input formats like Parquet, JSONL, and CSV, processing them through scalable pipelines to output curated Parquet or JSONL files.

Does multimodal data curation support PII redaction and NSFW detection?▼

Multimodal data curation includes built-in PII redaction and NSFW detection to protect privacy and safety while preparing large language model training data.

What is the best way to remove duplicate entries across large multimodal corpora?▼

Using GPU-accelerated RAPIDS-compatible tooling provides high-speed exact, fuzzy, and semantic deduplication to efficiently remove duplicate entries across large multimodal corpora.

Do I need RAPIDS-compatible GPU tooling to run these data curation pipelines?▼

Yes, RAPIDS-compatible GPU acceleration is required to run these end-to-end multimodal data curation pipelines for quality filtering, deduplication, and PII redaction at scale.

How does semantic deduplication work for text and multimodal data?▼

Semantic deduplication works by applying GPU-accelerated processing across text and multimodal data to identify and remove contextually similar content, improving dataset quality for LLM training.

Related Skills

Explore More Data & Analytics Skills→
nelohenriqnelohenriq
1

nemo-curator

Curate large multimodal datasets with GPU-accelerated filtering, deduplication, and PII redaction.

Community
Advanced
skkeoriwskkeoriw

unsloth

Optimize LLM parameters and reduce VRAM usage during fine-tuning.

Community
Advanced
AissenLiuAissenLiu

nemo-curator

Automate GPU-accelerated data curation workflows for large multimodal datasets.

Community
Advanced
SignmanalSignmanal
1

nemo-curator

Curate large-scale LLM training datasets with GPU-accelerated filtering and deduplication.

Community
Advanced