skills.rest
zangjeicy
zangjeicyCOMMUNITY
Explore all skills by this author →
GitHub

nemo-curator

Perform GPU-accelerated fuzzy deduplication, quality filtering, and PII redaction for LLM training datasets.

Updated Jul 13, 2026
Search Tags:#data processing#deduplication#data curation#nemo#gpu acceleration#llm training
One-click install
npx skills add https://github.com/zangjeicy/Hermes --skill nemo-curator-zangjeicy
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: nemo-curator
Source: https://github.com/zangjeicy/Hermes/tree/main/optional-skills/mlops/nemo-curator
Command: npx skills add https://github.com/zangjeicy/Hermes --skill nemo-curator-zangjeicy

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires nemo-curator, cudf, dask, rapids, and includes references (resource) components.

What problem does it solve?

This skill addresses the high computational cost and complexity of preparing massive, high-quality datasets for LLM training by leveraging GPU acceleration to perform deduplication, filtering, and cleaning tasks.

Core Features & Use Cases

  • GPU-Accelerated Processing: Utilizes RAPIDS and CUDA to perform fuzzy deduplication and quality filtering up to 16x faster than CPU-based alternatives.
  • Multimodal Support: Provides specialized pipelines for cleaning and curating text, image, video, and audio datasets.
  • Use Case: Use this skill to process terabytes of raw web-scraped data, such as Common Crawl, by chaining heuristic filters, PII redaction, and semantic deduplication to produce a clean, high-quality training corpus.

Quick Start

Use the nemo-curator skill to run a quality filtering and deduplication pipeline on the dataset located in the input directory.

Frequently Asked Questions about nemo-curator

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I perform GPU-accelerated deduplication on large-scale LLM training datasets?▼

GPU-accelerated fuzzy deduplication for LLM training datasets is performed using RAPIDS and CUDA to process data up to 16x faster than CPU alternatives, ensuring high-throughput data cleaning pipelines.

Can I use RAPIDS and Dask for multimodal data curation including image and video?▼

Yes, multimodal data curation using RAPIDS and Dask supports specialized cleaning pipelines across text, image, video, and audio datasets for comprehensive LLM training preparation.

What is the best way to redact PII and apply quality filtering to Common Crawl data?▼

The best way to redact PII and apply quality filtering to Common Crawl data is by chaining heuristic filters, PII redaction, and semantic deduplication within a GPU-accelerated pipeline.

Does fuzzy deduplication for terabyte-scale web-scraped data require a GPU cluster?▼

Fuzzy deduplication for terabyte-scale web-scraped data relies on GPU clusters using RAPIDS to achieve near-linear performance scaling for high-throughput data processing pipelines.

How does GPU acceleration improve data curation compared to standard CPU processing?▼

GPU acceleration improves data curation over standard CPU processing by utilizing RAPIDS and CUDA, delivering up to 16x faster performance for fuzzy deduplication and quality filtering tasks.

What are the limitations of using Dask for PII redaction and semantic deduplication?▼

Using Dask for PII redaction and semantic deduplication requires compatible GPU hardware and RAPIDS dependencies to maintain near-linear performance scaling across large-scale multimodal datasets.

Related Skills

Explore More Data & Analytics Skills→
gagan114662gagan114662

nemo-curator

Curtate multimodal datasets for LLM training with GPU-accelerated deduplication and filtering.

Community
Advanced
sheawinklersheawinkler
78

nemo-curator

Deduplicate, filter, and redact multi-modal training data with GPU acceleration.

Community
Advanced
ldzhhxxldzhhxx
1

nemo-curator

Automate GPU-accelerated data curation for LLM training with RAPIDS.

Community
Advanced
SignmanalSignmanal
1

nemo-curator

Curate large-scale LLM training datasets with GPU-accelerated filtering and deduplication.

Community
Advanced