665,000+ skills indexed

DoanNgocCuongCOMMUNITY

Explore all skills by this author →

Verified:Website GitHub

nemo-curator

Accelerates LLM dataset curation with GPU-based deduplication, filtering, PII redaction, and NSFW detection.

Updated Aug 23, 2026

Search Tags:#data processing #deduplication #data curation #rapids #gpu acceleration #llm training #neMo curator

One-click install

npx skills add https://github.com/DoanNgocCuong/continuous-training-pipeline_T3_2026 --skill nemo-curator-doanngoccuong

Or copy as Structured Prompt for Agent▼

Please help me install this Agent Skill.
Skill: nemo-curator
Source: https://github.com/DoanNgocCuong/continuous-training-pipeline_T3_2026/tree/main/.claude/skills/nemo-curator
Command: npx skills add https://github.com/DoanNgocCuong/continuous-training-pipeline_T3_2026 --skill nemo-curator-doanngoccuong

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires nemo-curator, cudf, dask, rapids, and includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill addresses the challenge of preparing large, high-quality datasets for training Large Language Models (LLMs) efficiently and at scale.

Core Features & Use Cases

GPU-Accelerated Curation: Leverages NVIDIA's NeMo Curator toolkit for significantly faster data processing (e.g., 16x faster fuzzy deduplication).
Comprehensive Data Cleaning: Includes fuzzy and semantic deduplication, quality filtering (30+ heuristics), PII redaction, and NSFW detection.
Multimodal Support: Capable of curating text, image, video, and audio data.
Use Case: Prepare a massive web scrape dataset for LLM training by removing near-duplicate documents, filtering out low-quality content, and redacting sensitive information, all accelerated by GPUs.

Quick Start

Install the nemo-curator library with text processing capabilities for CUDA 12 by running the command: uv pip install "nemo-curator[text_cuda12]".

Frequently Asked Questions about nemo-curator

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I accelerate fuzzy deduplication for large-scale LLM training datasets?▼

You can accelerate fuzzy deduplication for LLM datasets using GPU acceleration via RAPIDS, achieving up to 16x faster processing compared to traditional methods.

Can I use GPU acceleration for PII redaction and NSFW detection in multimodal data?▼

Yes, GPU acceleration supports PII redaction and NSFW detection across multimodal data formats including text, image, video, and audio for comprehensive LLM data curation.

What is the best way to filter low-quality content from web scrape data for LLM training?▼

The best way to filter low-quality content from web scrape data is using quality filtering, which applies over 30 heuristics to clean and curate datasets efficiently for LLM training.

Does data curation with RAPIDS require specific CUDA dependencies?▼

Yes, data curation with RAPIDS requires specific CUDA dependencies, and you can install the necessary text processing capabilities for CUDA 12 using the specified installation command.

How do I scale data preparation tasks across multiple GPUs for LLM training?▼

You scale data preparation tasks across multiple GPUs for LLM training by utilizing Dask and RAPIDS, which distribute the workload to handle massive datasets efficiently.

Related Skills

Explore More Software Engineering Skills→

nemo-curator

Curate large multimodal datasets for LLM training with GPU-accelerated deduplication and filtering.

nemo-curator

Deduplicate, filter, and redact multi-modal training data with GPU acceleration.

twjarviszyra-web

nemo-curator

Clean and deduplicate large-scale training datasets with GPU acceleration.

nemo-curator

Process text, image, video, and audio data with GPU-accelerated curation.