skills.rest
zeronx798
zeronx798COMMUNITY
Explore all skills by this author →
GitHub

nemo-curator

Cure and deduplicate large-scale multimodal datasets for LLM training on GPU clusters.

Updated Jul 13, 2026
Search Tags:#data processing#deduplication#data curation#gpu acceleration#llm training#nemo-curator
One-click install
npx skills add https://github.com/zeronx798/demo-hermes-agent --skill nemo-curator-zeronx798
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: nemo-curator
Source: https://github.com/zeronx798/demo-hermes-agent/tree/main/optional-skills/mlops/nemo-curator
Command: npx skills add https://github.com/zeronx798/demo-hermes-agent --skill nemo-curator-zeronx798

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires nemo-curator, cudf, dask, rapids, and includes references (resource) components.

What problem does it solve?

This Skill addresses the massive computational bottleneck of preparing high-quality training datasets for Large Language Models by offloading heavy data processing tasks to GPUs.

Core Features & Use Cases

  • High-Speed Deduplication: Perform exact, fuzzy, and semantic deduplication up to 16 times faster than CPU-based alternatives.
  • Advanced Quality Filtering: Apply over 30 heuristic filters and AI-based classifiers to remove low-quality, toxic, or irrelevant content.
  • Multimodal Support: Process text, image, video, and audio datasets within a unified, scalable pipeline.
  • Use Case: Use this tool to clean and deduplicate terabytes of web-scraped data for pre-training a foundation model, ensuring high data quality while minimizing infrastructure costs.

Quick Start

Use the nemo-curator skill to perform fuzzy deduplication on the dataset located in the input directory using the GPU cluster.

Frequently Asked Questions about nemo-curator

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I accelerate fuzzy deduplication for large-scale LLM training datasets?▼

You can accelerate fuzzy deduplication for LLM training datasets by offloading MinHash-based processing to distributed GPU clusters, achieving up to 16 times faster performance compared to CPU-based alternatives.

What is GPU-accelerated data curation and how does it handle multimodal datasets?▼

GPU-accelerated data curation is a parallelized pipeline that cleans text, image, video, and audio datasets, applying heuristic filtering and PII redaction to prepare high-quality inputs for foundation model pre-training.

Do I need a distributed GPU cluster to run data curation workflows?▼

You need a distributed GPU cluster powered by RAPIDS, cuDF, and Dask to achieve near-linear scaling and high-throughput processing for terabytes of web-scraped data.

Can I redact PII and filter toxic content during data preparation?▼

You can redact PII and filter toxic content during data preparation using GPU-parallelized curation, which applies over 30 heuristic filters and AI-based classifiers to remove low-quality or irrelevant content.

What's the best way to reduce infrastructure costs when cleaning terabytes of pre-training data?▼

The best way to reduce infrastructure costs when cleaning terabytes of pre-training data is offloading curation tasks to GPUs, which significantly reduces total cost of ownership compared to CPU-based pipelines.

Related Skills

Explore More Data & Analytics Skills→
Monjyu1101Monjyu1101
1

nemo-curator

Curate large multimodal datasets for LLM training with GPU-accelerated deduplication and filtering.

Community
Advanced
sheawinklersheawinkler
78

nemo-curator

Deduplicate, filter, and redact multi-modal training data with GPU acceleration.

Community
Advanced
aselimcaselimc

dask

Scale pandas and NumPy workflows to larger-than-memory datasets with parallel and distributed computing.

Community
Advanced
SignmanalSignmanal
1

nemo-curator

Curate large-scale LLM training datasets with GPU-accelerated filtering and deduplication.

Community
Advanced