nemo-curator

Name: nemo-curator
Availability: InStock
Author: zangjeicy

Perform GPU-accelerated fuzzy deduplication, quality filtering, and PII redaction for LLM training datasets.

Updated Jul 13, 2026

Search Tags:#data processing #deduplication #data curation #nemo #gpu acceleration #llm training

Ready to run

npx skills add https://github.com/zangjeicy/Hermes --skill nemo-curator-zangjeicy

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: nemo-curator
Source: https://github.com/zangjeicy/Hermes/tree/main/optional-skills/mlops/nemo-curator
Command: npx skills add https://github.com/zangjeicy/Hermes --skill nemo-curator-zangjeicy

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires nemo-curator, cudf, dask, rapids, and includes references (resource) components.

What problem does it solve?

This skill addresses the high computational cost and complexity of preparing massive, high-quality datasets for LLM training by leveraging GPU acceleration to perform deduplication, filtering, and cleaning tasks.

Core Features & Use Cases

GPU-Accelerated Processing: Utilizes RAPIDS and CUDA to perform fuzzy deduplication and quality filtering up to 16x faster than CPU-based alternatives.
Multimodal Support: Provides specialized pipelines for cleaning and curating text, image, video, and audio datasets.
Use Case: Use this skill to process terabytes of raw web-scraped data, such as Common Crawl, by chaining heuristic filters, PII redaction, and semantic deduplication to produce a clean, high-quality training corpus.

Quick Start

Use the nemo-curator skill to run a quality filtering and deduplication pipeline on the dataset located in the input directory.

Frequently Asked Questions about nemo-curator

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I perform GPU-accelerated deduplication on large-scale LLM training datasets?▼

GPU-accelerated fuzzy deduplication for LLM training datasets is performed using RAPIDS and CUDA to process data up to 16x faster than CPU alternatives, ensuring high-throughput data cleaning pipelines.

Can I use RAPIDS and Dask for multimodal data curation including image and video?▼

Yes, multimodal data curation using RAPIDS and Dask supports specialized cleaning pipelines across text, image, video, and audio datasets for comprehensive LLM training preparation.

What is the best way to redact PII and apply quality filtering to Common Crawl data?▼

The best way to redact PII and apply quality filtering to Common Crawl data is by chaining heuristic filters, PII redaction, and semantic deduplication within a GPU-accelerated pipeline.

Does fuzzy deduplication for terabyte-scale web-scraped data require a GPU cluster?▼

Fuzzy deduplication for terabyte-scale web-scraped data relies on GPU clusters using RAPIDS to achieve near-linear performance scaling for high-throughput data processing pipelines.

How does GPU acceleration improve data curation compared to standard CPU processing?▼

GPU acceleration improves data curation over standard CPU processing by utilizing RAPIDS and CUDA, delivering up to 16x faster performance for fuzzy deduplication and quality filtering tasks.

What are the limitations of using Dask for PII redaction and semantic deduplication?▼

Using Dask for PII redaction and semantic deduplication requires compatible GPU hardware and RAPIDS dependencies to maintain near-linear performance scaling across large-scale multimodal datasets.