skills.rest
vikrant-project
vikrant-projectCOMMUNITY
Explore all skills by this author →
GitHub

nemo-curator

Curate multimodal data for LLM training with GPU-accelerated filtering and deduplication.

2|Updated Jun 8, 2026
Search Tags:#quality control#GPU acceleration#deduplication#LLM training#data curation
One-click install
npx skills add https://github.com/vikrant-project/devil-agent-ai-platform --skill nemo-curator-vikrant-project
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: nemo-curator
Source: https://github.com/vikrant-project/devil-agent-ai-platform/tree/main/agent_core/optional-skills/mlops/nemo-curator
Command: npx skills add https://github.com/vikrant-project/devil-agent-ai-platform --skill nemo-curator-vikrant-project

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires nemo-curator, cudf, dask, rapids, and includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill streamlines the preparation of high-quality training data for Large Language Models (LLMs), addressing the challenges of large, diverse, and multimodal datasets.

Core Features & Use Cases

  • Multi-Modal Curation: Process text, images, video, and audio data with dedicated tools for each modality.
  • Quality Control: Apply 30+ heuristic filters for text, including language identification, quality scoring, and NSFW detection.
  • Deduplication: Perform fast and efficient deduplication, including exact, fuzzy, and semantic methods.
  • PII Redaction: Safely remove personally identifiable information to protect privacy.
  • Use Case: Use this Skill to clean and prepare data from web scrapes like Common Crawl for training an LLM, ensuring high-quality, diverse, and privacy-compliant datasets.

Quick Start

Use the nemo-curator skill to deduplicate and filter text data from 'web_data.parquet'.

Frequently Asked Questions about nemo-curator

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I deduplicate and filter large text datasets for LLM training?▼

You can curate LLM training data by applying GPU-accelerated exact, fuzzy, and semantic deduplication alongside 30+ heuristic quality filters to clean raw text datasets efficiently.

What is GPU-accelerated data curation and how does it scale for multimodal datasets?▼

GPU-accelerated data curation uses RAPIDS and Dask to distribute processing across GPU clusters, enabling scalable quality filtering and deduplication for large text, image, video, and audio datasets.

Can I use RAPIDS and Dask to redact PII from Common Crawl data?▼

Yes, you can use this Skill with RAPIDS and Dask to redact personally identifiable information from Common Crawl data, ensuring your LLM training datasets remain privacy-compliant.

Does GPU-accelerated data curation support image, video, and audio processing?▼

Yes, GPU-accelerated data curation supports multimodal processing, providing dedicated tools to filter and curate image, video, and audio data alongside standard text for LLM training.

What's the best way to perform quality control on web scrapes for LLM training?▼

The best way to perform quality control on web scrapes is applying automated heuristic filters for language identification, quality scoring, and NSFW detection to ensure high-quality LLM training datasets.

Do I need a GPU cluster to run deduplication and quality filtering on training data?▼

You need a GPU cluster to leverage the RAPIDS and Dask integration for GPU-accelerated deduplication and quality filtering, ensuring efficient scaling when processing large multimodal datasets.

Related Skills

Explore More Data & Analytics Skills→
ovachieverovachiever
52

nemo-curator

Curate multimodal LLM training data with GPU-accelerated filtering and deduplication.

Community
Advanced
sheawinklersheawinkler
78

nemo-curator

Deduplicate, filter, and redact multi-modal training data with GPU acceleration.

Community
Advanced
ketzal88ketzal88

agent-orchestration-multi-agent-optimize

Profile, coordinate, and throttle workloads across multiple agents in distributed AI pipelines or simulations.

Community
Advanced
NITISH-gitbitNITISH-gitbit

nemo-curator

Deduplicate and filter LLM training datasets with GPU acceleration.

Community
Advanced