skills.rest
zhuangbiaowei
zhuangbiaoweiCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

nemo-curator

Curate multimodal LLM training datasets with GPU-accelerated deduplication and filtering.

2|Updated Feb 4, 2026
Search Tags:#multimodal#data processing#deduplication#data curation#gpu acceleration#llm training
One-click install
npx skills add https://github.com/zhuangbiaowei/smart_bot --skill nemo-curator-zhuangbiaowei
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: nemo-curator
Source: https://github.com/zhuangbiaowei/smart_bot/tree/main/skills/nemo-curator
Command: npx skills add https://github.com/zhuangbiaowei/smart_bot --skill nemo-curator-zhuangbiaowei

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires nemo-curator, cudf, dask, rapids, and includes references (resource) components.

What problem does it solve?

This Skill addresses the challenge of preparing massive, high-quality datasets required for training Large Language Models (LLMs), especially when dealing with multimodal data and the need for speed and scale.

Core Features & Use Cases

  • GPU-Accelerated Curation: Leverages NVIDIA's RAPIDS for significantly faster data processing (e.g., 16x faster fuzzy deduplication).
  • Multimodal Support: Handles text, image, video, and audio data.
  • Advanced Filtering & Deduplication: Includes fuzzy deduplication, quality filtering with 30+ heuristics, semantic deduplication, PII redaction, and NSFW detection.
  • Use Case: Prepare a high-quality, diverse dataset for training a new LLM by cleaning and deduplicating terabytes of web-scraped text and image data, ensuring it's free of PII and low-quality content.

Quick Start

Install the necessary NeMo Curator components for text curation on CUDA 12.

Frequently Asked Questions about nemo-curator

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I accelerate fuzzy deduplication for large LLM training datasets?▼

GPU-accelerated data curation tools use NVIDIA RAPIDS to handle multimodal data curation, supporting text, images, video, and audio formats for building high-quality LLM training datasets.

Can I use GPU acceleration for text and image data curation simultaneously?▼

Yes, multimodal data curation supports processing text, image, video, and audio data simultaneously using GPU acceleration via RAPIDS and Dask for distributed computing.

What is semantic deduplication and how does it clean LLM training data?▼

Semantic deduplication identifies and removes contextually similar data points in LLM training datasets, combining with quality filtering and PII redaction to ensure high data diversity and compliance.

Does nemo-curator require NVIDIA GPUs and CUDA to run data processing tasks?▼

Yes, GPU-accelerated data curation requires NVIDIA GPUs and CUDA because it depends on RAPIDS and cuDF for performance optimizations like 16x faster fuzzy deduplication.

How do I redact PII and detect NSFW content in web-scraped datasets?▼

Redact PII and detect NSFW content in web-scraped datasets by applying built-in curation modules that perform quality filtering and content classification during GPU-accelerated data processing.

What's the best way to filter low-quality text data for LLM training?▼

The best way to filter low-quality text data for LLM training is applying quality filtering with 30+ heuristics and classifiers during GPU-accelerated curation to ensure high dataset standards.

Related Skills

Explore More Software Engineering Skills→
LynxLabVNLynxLabVN

nemo-curator

Accelerate LLM training dataset curation with GPU computing and RAPIDS.

Official
Advanced
SignmanalSignmanal
1

nemo-curator

Curate large-scale LLM training datasets with GPU-accelerated filtering and deduplication.

Community
Advanced
HarriesHarries

nemo-curator

Clean and deduplicate large multi-modal datasets for LLM training with GPU acceleration.

Community
Advanced
NITISH-gitbitNITISH-gitbit

nemo-curator

Deduplicate and filter LLM training datasets with GPU acceleration.

Community
Advanced