skills.rest
informatico-madrid
informatico-madridCOMMUNITY
Explore all skills by this author →
GitHub

nemo-curator

Prepare LLM training data with GPU-accelerated curation tools.

1|Updated Feb 10, 2026
Search Tags:#deduplication#data curation#rapids#gpu acceleration#llm training#multimodal data
One-click install
npx skills add https://github.com/informatico-madrid/Architect-Expert-Gap-Forge --skill nemo-curator-informatico-madrid
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: nemo-curator
Source: https://github.com/informatico-madrid/Architect-Expert-Gap-Forge/tree/main/.github/skills/nemo-curator
Command: npx skills add https://github.com/informatico-madrid/Architect-Expert-Gap-Forge --skill nemo-curator-informatico-madrid

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires nemo-curator, cudf, dask, rapids, and includes references (resource) components.

What problem does it solve?

This Skill addresses the challenge of preparing massive, high-quality datasets for training Large Language Models, especially when dealing with multimodal data and the need for efficient, GPU-accelerated processing.

Core Features & Use Cases

  • GPU-Accelerated Curation: Leverages NVIDIA's RAPIDS for significant speedups in deduplication, filtering, and PII redaction.
  • Multimodal Support: Handles text, image, video, and audio data curation.
  • Advanced Filtering & Deduplication: Offers fuzzy and semantic deduplication, quality filtering with 30+ heuristics, PII redaction, and NSFW detection.
  • Use Case: Prepare a large corpus of web-scraped text and images for LLM training by removing duplicates, filtering low-quality content, and redacting sensitive information, all significantly faster using GPUs.

Quick Start

Use the nemo-curator skill to prepare LLM training data by applying quality filters and exact deduplication to a dataset.

Frequently Asked Questions about nemo-curator

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I accelerate LLM training data preparation with GPU curation tools?▼

You can accelerate LLM training data preparation by using GPU-powered curation tools that leverage NVIDIA RAPIDS for significant speedups in fuzzy deduplication, quality filtering, and PII redaction across large-scale corpora.

What is GPU-accelerated fuzzy deduplication for multimodal datasets?▼

GPU-accelerated fuzzy deduplication is a data curation process that uses RAPIDS to rapidly identify and remove near-duplicate entries across multimodal datasets, including text, image, video, and audio files for LLM training.

Can I use RAPIDS and Dask for large-scale text and image data curation?▼

Yes, you can use RAPIDS and Dask to scale data curation efficiently across multiple GPUs. This combination handles large-scale processing of text, image, video, and audio data for LLM training datasets.

How do I filter low-quality content and redact PII in web-scraped datasets?▼

You can filter low-quality content and redact PII by applying over 30 quality filtering heuristics alongside built-in PII redaction and NSFW detection modules during the data curation workflow.

Do I need NVIDIA GPUs to run large-scale dataset deduplication and quality filtering?▼

Yes, GPU acceleration using RAPIDS is required to achieve the significant speedups in deduplication, filtering, and redaction designed for preparing massive, high-quality multimodal datasets for LLM training.

Related Skills

Explore More Software Engineering Skills→
projectedanxprojectedanx
1

nemo-curator

Deduplicate and filter large-scale LLM training data with GPU-optimized pipelines.

Community
Advanced
pmcdowallpmcdowall

fine-tuning-with-trl

Align large language models via reinforcement learning and preference-based fine-tuning workflows.

Community
Advanced
brainworkupbrainworkup
1

optimize-for-gpu

Port CPU-bound Python code to GPU-accelerated RAPIDS libraries.

Community
Advanced
NITISH-gitbitNITISH-gitbit

nemo-curator

Deduplicate and filter LLM training datasets with GPU acceleration.

Community
Advanced