skills.rest
icyzh
icyzhCOMMUNITY
Explore all skills by this author →
Website𝕏@icyzhbeGitHub

nemo-curator

Curate large-scale LLM training datasets with GPU-accelerated deduplication, filtering, and PII redaction.

1|Updated Jul 31, 2026
Search Tags:#deduplication#data curation#pii redaction#gpu acceleration#llm training#nemo-curator
One-click install
npx skills add https://github.com/icyzh/hermes-web --skill nemo-curator-icyzh
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: nemo-curator
Source: https://github.com/icyzh/hermes-web/tree/main/optional-skills/mlops/nemo-curator
Command: npx skills add https://github.com/icyzh/hermes-web --skill nemo-curator-icyzh

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires nemo-curator, cudf, dask, rapids, and includes references (resource) components.

What problem does it solve?

This Skill addresses the massive computational bottleneck of preparing high-quality training data for Large Language Models by leveraging GPU acceleration to perform deduplication, filtering, and PII redaction at scale.

Core Features & Use Cases

  • High-Speed Deduplication: Perform exact, fuzzy, and semantic deduplication up to 16 times faster than CPU-based alternatives.
  • Multi-Modal Processing: Curate diverse datasets including text, images, video, and audio with specialized heuristic and classifier-based filters.
  • Use Case: When preparing a multi-terabyte web-scraped dataset like Common Crawl, use this Skill to remove toxic content, redact sensitive PII, and eliminate near-duplicate documents efficiently on a GPU cluster.

Quick Start

Use the nemo-curator skill to run a deduplication pipeline on the dataset located in the current directory.

Frequently Asked Questions about nemo-curator

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I accelerate deduplication for large-scale LLM training datasets?▼

GPU-accelerated deduplication uses RAPIDS libraries to process large-scale LLM training datasets up to 16 times faster than CPU-based alternatives, performing exact, fuzzy, and semantic matching efficiently.

Can I curate multi-modal data like images and audio for LLM training?▼

Multi-modal data curation supports text, images, video, and audio modalities. It applies specialized heuristic and classifier-based filters to prepare diverse production-grade training datasets.

What's the best way to redact PII from web-scraped datasets like Common Crawl?▼

PII redaction removes sensitive personal information from multi-terabyte web-scraped datasets like Common Crawl. It leverages GPU clusters to efficiently redact sensitive data during high-throughput preparation pipelines.

Does GPU-accelerated data curation require Ray and Dask architectures?▼

High-throughput GPU-accelerated data curation relies on Ray-based architectures and RAPIDS libraries like cudf and dask to distribute workloads across GPU clusters for production-grade dataset preparation.

Why use GPU-accelerated filtering instead of CPU-based data curation tools?▼

GPU-accelerated filtering overcomes massive computational bottlenecks in data curation. It removes toxic content and near-duplicate documents significantly faster than CPU-based alternatives, satisfying high-throughput processing requirements.

When should I not use GPU-accelerated data curation for dataset preparation?▼

GPU-accelerated data curation is designed for massive computational bottlenecks and multi-terabyte datasets. It requires a GPU cluster and dependencies like cudf and dask, making it unsuitable for smaller scale or single-CPU processing environments.

Related Skills

Explore More Data & Analytics Skills→
SalesforceCommerceCloudSalesforceCommerceCloud
51

b2c-ods

Provision, manage, and monitor on-demand B2C sandboxes using the b2c CLI.

Official
Intermediate
nelohenriqnelohenriq
1

nemo-curator

Curate large multimodal datasets with GPU-accelerated filtering, deduplication, and PII redaction.

Community
Advanced
JKhyroJKhyro

nemo-curator

Filter, deduplicate, and redact multimodal training datasets with GPU acceleration.

Community
Advanced
OpenLAIROpenLAIR
1.0k

nemo-curator

Automate GPU-accelerated data curation for LLM training datasets.

Official
Advanced