skills.rest
hochoa13
hochoa13COMMUNITY
Explore all skills by this author →
GitHub

nemo-curator

Curate multimodal datasets for LLM training with GPU-accelerated deduplication and filtering.

Updated Mar 13, 2026
Search Tags:#data processing#deduplication#data curation#rapids#gpu acceleration#llm training#nemo curator
One-click install
npx skills add https://github.com/hochoa13/Asistente-Harlest --skill nemo-curator-hochoa13
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: nemo-curator
Source: https://github.com/hochoa13/Asistente-Harlest/tree/main/skills/mlops/evaluation/nemo-curator
Command: npx skills add https://github.com/hochoa13/Asistente-Harlest --skill nemo-curator-hochoa13

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill addresses the challenge of efficiently preparing large-scale, high-quality datasets required for training Large Language Models (LLMs), significantly reducing processing time and cost through GPU acceleration.

Core Features & Use Cases

  • GPU-Accelerated Deduplication: Features fuzzy and semantic deduplication that is up to 16x faster than CPU-based methods.
  • Multi-Modal Support: Capable of curating text, image, video, and audio data.
  • Quality & Safety Filtering: Includes PII redaction, NSFW detection, and over 30 quality heuristics.
  • Use Case: Prepare a massive dataset of web-scraped text for LLM training by cleaning, deduplicating, and filtering out low-quality or sensitive content using GPU resources.

Quick Start

Use the nemo-curator skill to perform fuzzy deduplication on a dataset of text documents.

Frequently Asked Questions about nemo-curator

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I accelerate data curation and deduplication for large-scale LLM training datasets?▼

GPU-accelerated data curation speeds up preparing large-scale datasets for LLM training by utilizing libraries like NeMo Curator and RAPIDS to process terabytes of text, images, video, and audio efficiently across multiple GPUs.

What is the best way to perform fuzzy and semantic deduplication on web-scraped text data?▼

Fuzzy and semantic deduplication on web-scraped text is best handled using GPU-powered libraries, which perform these operations up to 16x faster than traditional CPU-based methods while scaling across multiple GPUs.

Can I redact PII and detect NSFW content during the data curation process?▼

PII redaction and NSFW detection can be performed directly during the data curation process, alongside cleaning and filtering out low-quality content using over 30 quality heuristics.

Does GPU-accelerated data curation support multimodal data like video and audio?▼

GPU-accelerated data curation supports multimodal data by efficiently processing and curating not just text, but also images, video, and audio formats for comprehensive LLM training preparation.

Do I need multiple GPUs to process terabytes of data for LLM training preparation?▼

Multiple GPUs are required to scale efficiently when processing terabytes of data for LLM training preparation, allowing the curation pipeline to handle massive datasets through GPU acceleration.

Related Skills

Explore More Software Engineering Skills→
dawsonblockdawsonblock

nemo-curator

Automates GPU-accelerated data curation for LLM training datasets.

Community
Advanced
OpenLAIROpenLAIR
1.0k

training-llms-megatron

Train large language models with Megatron-Core parallelism on multi-node GPU clusters.

Official
Advanced
samuelmukotisamuelmukoti

nemo-curator

Curate multimodal LLM training data with GPU-accelerated filtering, deduplication, and redaction.

Community
Advanced
NITISH-gitbitNITISH-gitbit

nemo-curator

Deduplicate and filter LLM training datasets with GPU acceleration.

Community
Advanced