skills.rest
Rheasilvia
RheasilviaCOMMUNITY
Explore all skills by this author →
GitHub

nemo-curator

Curate large-scale LLM training datasets with GPU-accelerated deduplication, filtering, and PII redaction.

Updated Jun 25, 2026
Search Tags:#deduplication#data curation#pii redaction#gpu acceleration#llm training#nemo-curator
One-click install
npx skills add https://github.com/Rheasilvia/hermes-desktop --skill nemo-curator-rheasilvia
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: nemo-curator
Source: https://github.com/Rheasilvia/hermes-desktop/tree/main/optional-skills/mlops/nemo-curator
Command: npx skills add https://github.com/Rheasilvia/hermes-desktop --skill nemo-curator-rheasilvia

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires nemo-curator, cudf, dask, rapids, and includes references (resource) components.

What problem does it solve?

This Skill addresses the massive computational bottleneck in preparing high-quality LLM training data by providing GPU-accelerated tools for deduplication, filtering, and PII redaction.

Core Features & Use Cases

  • High-Speed Curation: Perform fuzzy deduplication and quality filtering up to 16 times faster than CPU-based alternatives.
  • Multi-modal Support: Process text, image, video, and audio datasets within a unified pipeline architecture.
  • Use Case: When preparing a multi-terabyte web-scraped dataset like Common Crawl, use this Skill to remove toxic content, redact PII, and deduplicate records efficiently across a GPU cluster.

Quick Start

Use the nemo-curator skill to run a deduplication pipeline on the dataset located in the current directory.

Frequently Asked Questions about nemo-curator

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I accelerate LLM training data deduplication for large datasets?▼

GPU-accelerated deduplication processes large-scale LLM training datasets up to 16 times faster than CPU alternatives by leveraging RAPIDS and multi-GPU clusters to remove duplicate records efficiently.

Does nemo-curator support PII redaction and quality filtering for Common Crawl data?▼

Yes, PII redaction and quality filtering are supported natively to clean web-scraped datasets like Common Crawl, removing toxic content and sensitive information during the GPU-accelerated curation pipeline.

Can I use nemo-curator for multi-modal data curation including images and audio?▼

Multi-modal data curation supports text, image, video, and audio datasets within a unified pipeline architecture, enabling high-performance preparation workflows across diverse data types.

What prerequisites are needed to scale data curation across a GPU cluster?▼

Scaling GPU-accelerated data curation across multi-GPU clusters requires integration with NVIDIA RAPIDS, Dask, and Ray to achieve near-linear scaling performance during dataset processing.

How much faster is GPU-accelerated data curation compared to CPU-based alternatives?▼

GPU-accelerated curation performs fuzzy deduplication and quality filtering up to 16 times faster than CPU-based alternatives by utilizing RAPIDS and cuDF for parallel processing.

What is the best way to prepare multi-terabyte web-scraped datasets for LLM training?▼

The best way to prepare multi-terabyte web-scraped datasets is using a unified GPU-accelerated pipeline to concurrently redact PII, filter toxic content, and deduplicate records across distributed clusters.

Related Skills

Explore More Data & Analytics Skills→
ovachieverovachiever
52

nemo-curator

Curate multimodal LLM training data with GPU-accelerated filtering and deduplication.

Community
Advanced
davpatel605-beepdavpatel605-beep

simpo-training

Configure and run reference-free SimPO preference alignment training for LLMs.

Community
Advanced
ClawLink-HQClawLink-HQ
117

google-contacts-address-book

Manage Google Contacts and contact groups via the Google People API.

Official
Intermediate
zeronx798zeronx798

nemo-curator

Cure and deduplicate large-scale multimodal datasets for LLM training on GPU clusters.

Community
Advanced