skills.rest
tianhao909
tianhao909COMMUNITY
Explore all skills by this author →
GitHub

nemo-curator

Curate LLM training data with GPU-accelerated deduplication and filtering.

1|Updated Feb 21, 2026
Search Tags:#data quality#multimodal#deduplication#data curation#rapids#gpu acceleration#llm training
One-click install
npx skills add https://github.com/tianhao909/AI-Research-SKILLs-cn --skill nemo-curator-tianhao909
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: nemo-curator
Source: https://github.com/tianhao909/AI-Research-SKILLs-cn/tree/main/05-data-processing/nemo-curator
Command: npx skills add https://github.com/tianhao909/AI-Research-SKILLs-cn --skill nemo-curator-tianhao909

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires nemo-curator, cudf, dask, rapids, and includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill addresses the critical need for high-quality, clean, and deduplicated datasets essential for training Large Language Models (LLMs), significantly accelerating the data preparation process.

Core Features & Use Cases

  • GPU-Accelerated Processing: Leverages NVIDIA's RAPIDS for 16x faster fuzzy deduplication and efficient scaling across GPUs.
  • Comprehensive Curation: Includes quality filtering (30+ heuristics), semantic deduplication, PII redaction, and NSFW detection.
  • Multimodal Support: Capable of curating text, image, video, and audio data.
  • Use Case: Prepare a massive dataset of web-scraped text for LLM training by removing near-duplicate documents, filtering out low-quality content, and redacting sensitive personal information, all on a GPU cluster for maximum speed.

Quick Start

Use the nemo-curator skill to prepare LLM training data by applying quality filters and fuzzy deduplication to a dataset of text documents.

Frequently Asked Questions about nemo-curator

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I deduplicate and filter large-scale text data for LLM training?▼

GPU-accelerated data curation uses RAPIDS and Dask to scale fuzzy and semantic deduplication across multiple GPUs, achieving 16x faster processing for large-scale LLM training data preparation.

What is GPU-accelerated fuzzy deduplication and how does it work for dataset preparation?▼

GPU-accelerated data curation uses RAPIDS and Dask to scale fuzzy and semantic deduplication across multiple GPUs, achieving 16x faster processing for large-scale LLM training data preparation.

Can I redact PII and detect NSFW content across multimodal datasets?▼

You need a GPU cluster environment with RAPIDS, cuDF, and Dask configured to enable distributed processing and GPU acceleration for large-scale multimodal dataset curation.

Do I need a GPU cluster to use RAPIDS for data curation?▼

You need a GPU cluster environment with RAPIDS, cuDF, and Dask configured to enable distributed processing and GPU acceleration for large-scale multimodal dataset curation.

What's the best way to scale semantic deduplication across multiple GPUs?▼

Scaling semantic deduplication across multiple GPUs is best handled by distributing the workload with Dask and utilizing RAPIDS libraries to process large datasets efficiently.

Does data curation support quality filtering for image and audio modalities?▼

Quality filtering capabilities extend beyond text to support multimodal data curation, allowing you to process and clean image, video, and audio datasets for comprehensive model training.

Related Skills

Explore More Data & Analytics Skills→
richardnguyen0715richardnguyen0715

nemo-curator

GPU-accelerate LLM training data curation with fuzzy deduplication, quality filtering, and PII redaction.

Community
Advanced
afel6afel6

nemo-curator

Automate multimodal LLM data curation with deduplication, filtering, and PII redaction on GPU clusters.

Community
Advanced
projectedanxprojectedanx
1

nemo-curator

Deduplicate and filter large-scale LLM training data with GPU-optimized pipelines.

Community
Advanced
Quill-AgentQuill-Agent
3

nemo-curator

Accelerate LLM training data curation with GPU-based deduplication and quality filtering.

Community
Advanced