nemo-curator

Name: nemo-curator
Availability: InStock
Author: mesferali

Accelerate LLM training data curation with GPU-intensive operations for text, image, video, and audio.

Updated Jan 12, 2026

Search Tags:#data processing #deduplication #data curation #rapids #gpu acceleration #llm training

Ready to run

npx skills add https://github.com/MesferAli/XCircle --skill nemo-curator-mesferali

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: nemo-curator
Source: https://github.com/MesferAli/XCircle/tree/main/.claude/skills/nemo-curator
Command: npx skills add https://github.com/MesferAli/XCircle --skill nemo-curator-mesferali

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires nemo_curator, cudf, dask, rapids, and includes references (resource) components.

What problem does it solve?

This Skill automates the complex and time-consuming process of preparing high-quality datasets for training Large Language Models (LLMs), significantly accelerating data curation pipelines.

Core Features & Use Cases

GPU-Accelerated Processing: Leverages NVIDIA RAPIDS for 16x faster deduplication and efficient scaling across GPUs.
Multimodal Support: Handles text, image, video, and audio data curation.
Advanced Filtering: Includes quality heuristics, PII redaction, NSFW detection, and semantic deduplication.
Use Case: Prepare a massive dataset of text and images for a multimodal LLM by cleaning web scrapes, removing duplicates, redacting sensitive information, and filtering low-quality content, all on a GPU cluster.

Quick Start

Use the nemo-curator skill to perform fuzzy deduplication on the dataset located at '/data/raw_text_data'.

Frequently Asked Questions about nemo-curator

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I accelerate LLM data curation and deduplication for large datasets?▼

You can accelerate LLM data curation by using GPU-intensive operations with NVIDIA RAPIDS, achieving up to 16x faster deduplication and efficient scaling across multiple GPUs for massive text and multimodal datasets.

Does GPU-accelerated data processing support multimodal datasets like images and video?▼

Yes, GPU-accelerated data processing supports multimodal datasets, handling text, image, video, and audio data curation to prepare comprehensive training data for multimodal LLMs.

What is the best way to perform fuzzy and semantic deduplication on web scraped text?▼

The best way to perform fuzzy and semantic deduplication on web scraped text is using GPU-accelerated operations that scale efficiently across GPUs, removing exact and near-duplicate content rapidly.

Can I redact PII and filter NSFW content during LLM training data preparation?▼

Yes, you can redact PII and filter NSFW content during LLM training data preparation by applying advanced quality filtering and sensitive information redaction features directly within the curation pipeline.

Do I need Dask and cuDF to scale data curation across multiple GPUs?▼

Yes, you need Dask and cuDF dependencies to scale data curation across multiple GPUs, as these frameworks enable the distributed computing and GPU acceleration required for high-throughput dataset preparation.

How do I perform fuzzy deduplication on raw text data for LLM training?▼

You can perform fuzzy deduplication on raw text data by executing GPU-accelerated operations that process the dataset at '/data/raw_text_data', removing near-duplicates efficiently using NVIDIA RAPIDS.