skills.rest
jacardl
jacardlCOMMUNITY
Explore all skills by this author →
GitHub

nemo-curator

Deduplicate, filter, and redact PII from multimodal datasets with GPU acceleration.

Updated Apr 16, 2026
Search Tags:#multimodal#gpu#deduplication#pii-redaction#data-curation#ml-training#quality-filtering
One-click install
npx skills add https://github.com/jacardl/New-Radar --skill nemo-curator-jacardl
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: nemo-curator
Source: https://github.com/jacardl/New-Radar/tree/main/backend/frameworks/hermes-agent/optional-skills/mlops/nemo-curator
Command: npx skills add https://github.com/jacardl/New-Radar --skill nemo-curator-jacardl

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Curate high-quality training data for ML models by automating deduplication, quality filtering, and PII redaction across large, multimodal datasets to improve data quality and reduce manual curation time.

Core Features & Use Cases

  • Exact, fuzzy, and semantic deduplication to remove near-duplicate and paraphrased content across texts and multimodal data.
  • 30+ quality filters for content quality, language, NSFW, and structural checks, plus PII redaction for privacy.
  • Multimodal curation (text, images, video, audio) with GPU-accelerated pipelines and distributed processing capabilities.
  • Scales across GPU clusters with RAPIDS, enabling large-scale data preparation for high-quality training datasets.
  • Use cases include cleaning web-scraped data and preparing large training corpora for LLMs.

Quick Start

Install Nemo Curator with CUDA support and run a basic deduplication pipeline to clean a sample dataset.

Frequently Asked Questions about nemo-curator

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I perform deduplication and quality filtering on large web-scraped datasets for LLM training?▼

Data curation for LLM training removes near-duplicate content using exact, fuzzy, and semantic deduplication alongside 30+ quality filters, ensuring high-quality training corpora from web-scraped data.

What is GPU-accelerated data curation and when do I need it for multimodal datasets?▼

GPU-accelerated data curation scales processing across GPU clusters using RAPIDS to handle large multimodal datasets, enabling efficient cleaning of text, images, video, and audio for machine learning.

Does Nemo Curator support PII redaction for text and multimodal data?▼

PII redaction is supported during data curation to remove personally identifiable information, applying to text and multimodal data to ensure privacy compliance in training datasets.

Can I use RAPIDS for distributed processing to clean large-scale training corpora?▼

RAPIDS enables GPU-accelerated distributed processing for data curation, allowing you to scale quality filtering and deduplication pipelines across GPU clusters for large-scale training corpora preparation.

What's the best way to filter NSFW content and apply structural checks during dataset preparation?▼

Dataset preparation applies 30+ quality filters that perform NSFW content detection, language filtering, and structural checks, automating content quality control for high-quality ML training data.

Do I need CUDA support to run fuzzy and semantic deduplication pipelines?▼

CUDA support is required for GPU-accelerated processing, as the data curation pipelines rely on RAPIDS to perform exact, fuzzy, and semantic deduplication efficiently across GPU clusters.

Related Skills

Explore More Data & Analytics Skills→
ovachieverovachiever
52

nemo-curator

Curate multimodal LLM training data with GPU-accelerated filtering and deduplication.

Community
Advanced
nelohenriqnelohenriq
1

nemo-curator

Curate large multimodal datasets with GPU-accelerated filtering, deduplication, and PII redaction.

Community
Advanced
OpenAnalystIncOpenAnalystInc
6

Prompt Optimizer

Analyze agent prompts and quality feedback to refine instructions.

Official
Advanced
hhhi21ghhhi21g

nemo-curator

Automates GPU-accelerated deduplication, quality filtering, and PII redaction for LLM training datasets.

Community
Advanced