skills.rest
gagan114662
gagan114662COMMUNITY
Explore all skills by this author →
GitHub

nemo-curator

Curtate multimodal datasets for LLM training with GPU-accelerated deduplication and filtering.

Updated Jan 30, 2026
Search Tags:#data processing#deduplication#data curation#rapids#gpu acceleration#llm training
One-click install
npx skills add https://github.com/gagan114662/content_books --skill nemo-curator-gagan114662
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: nemo-curator
Source: https://github.com/gagan114662/content_books/tree/main/AI-research-SKILLs/05-data-processing/nemo-curator
Command: npx skills add https://github.com/gagan114662/content_books --skill nemo-curator-gagan114662

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires nemo-curator, cudf, dask, rapids, and includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill automates and accelerates the complex process of preparing high-quality training data for Large Language Models (LLMs), significantly reducing the time and computational cost associated with data curation.

Core Features & Use Cases

  • GPU-Accelerated Processing: Leverages NVIDIA's RAPIDS and Dask for massive speedups in deduplication and filtering across multiple GPUs.
  • Multimodal Support: Handles text, image, video, and audio data for comprehensive dataset preparation.
  • Advanced Curation Techniques: Includes fuzzy and semantic deduplication, quality filtering with 30+ heuristics, PII redaction, and NSFW detection.
  • Use Case: Prepare a massive web-scraped dataset for LLM training by efficiently removing near-duplicate documents, filtering out low-quality content, and redacting sensitive information, all at scale across a GPU cluster.

Quick Start

Use the nemo-curator skill to prepare a dataset by applying quality filters and exact deduplication.

Frequently Asked Questions about nemo-curator

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I accelerate data curation for LLM training across multiple GPUs?▼

GPU-accelerated LLM data curation uses RAPIDS and Dask to scale quality filtering and deduplication across multiple GPUs, significantly reducing processing time for large training corpora.

What is fuzzy and semantic deduplication in dataset preparation?▼

Fuzzy and semantic deduplication identifies and removes near-duplicate documents based on content similarity rather than exact text matches, ensuring higher data diversity in your training set.

Does NeMo Curator support multimodal data processing?▼

Yes, multimodal data curation handles text, image, video, and audio data within a single workflow, enabling comprehensive dataset preparation across different content formats.

Can I redact PII and filter NSFW content during data curation?▼

Yes, PII redaction and NSFW detection are built into the curation workflow, alongside quality filtering with 30+ heuristics, ensuring sensitive and inappropriate content is scrubbed from training data.

Do I need Dask and RAPIDS installed to use GPU-accelerated data curation?▼

Yes, RAPIDS, cuDF, and Dask are required dependencies, as the workflow distributes data processing workloads across GPUs using these frameworks to achieve massive speedups.

What is the best way to filter low-quality content from web-scraped datasets?▼

The best way is applying quality filtering with 30+ heuristics during GPU-accelerated curation, which combines fuzzy deduplication and PII redaction to prepare high-quality training data at scale.

Related Skills

Explore More Software Engineering Skills→
richardnguyen0715richardnguyen0715

nemo-curator

GPU-accelerate LLM training data curation with fuzzy deduplication, quality filtering, and PII redaction.

Community
Advanced
BagelHoleBagelHole
46

llm-fine-tuning

Set up QLoRA, LoRA, and full fine-tuning infrastructure with Hugging Face TRL and Axolotl.

Community
Advanced
ovachieverovachiever
52

nemo-curator

Curate multimodal LLM training data with GPU-accelerated filtering and deduplication.

Community
Advanced
twjarviszyra-webtwjarviszyra-web
3

nemo-curator

Clean and deduplicate large-scale training datasets with GPU acceleration.

Community
Advanced