skills.rest
valentinuuiuiu
valentinuuiuiuCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

nemo-curator

Curate GPU-accelerated multimodal datasets for LLM training.

Updated Aug 23, 2026
Search Tags:#GPU acceleration#deduplication#quality filtering#data curation#LLM training data
One-click install
npx skills add https://github.com/valentinuuiuiu/vikarma --skill nemo-curator-valentinuuiuiu
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: nemo-curator
Source: https://github.com/valentinuuiuiu/vikarma/tree/main/hermes_agent/optional-skills/mlops/nemo-curator
Command: npx skills add https://github.com/valentinuuiuiu/vikarma --skill nemo-curator-valentinuuiuiu

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires nemo-curator, cudf, dask, rapids, and includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill addresses the need for fast, GPU-accelerated data curation, allowing users to efficiently prepare high-quality training datasets for large language models (LLMs).

Core Features & Use Cases

  • Multi-modal Data Support: Process text, images, video, and audio with support for various file types.
  • GPU Acceleration: Leverage GPU power for fuzzy deduplication, quality filtering, and more.
  • Fuzzy Deduplication: Quickly deduplicate data up to 16x faster than CPU alternatives.
  • Quality Filtering: Apply 30+ heuristic filters for comprehensive dataset refinement.
  • Semantic Deduplication: Remove semantically similar documents for precise dataset creation.
  • PII Redaction: Protect sensitive information by redacting personally identifiable information.
  • NSFW Detection: Identify and filter out inappropriate content to ensure compliance.
  • Use Case: Use NeMo Curator to prepare high-quality LLM training data from Common Crawl web scrapes, web data, or large corpora.

Quick Start

Use the nemo-curator skill to prepare training data from a web scrape using 'common_crawl/*' and save it as 'curated_data.parquet'.

Frequently Asked Questions about nemo-curator

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I curate LLM training data using GPU acceleration?▼

To curate LLM training data with GPU acceleration, you can use this Skill to process large-scale web data, applying fuzzy deduplication and quality filtering across multiple GPUs to prepare high-quality datasets.

Does GPU-accelerated fuzzy deduplication work faster than CPU for large text corpora?▼

GPU-accelerated fuzzy deduplication works significantly faster for large text corpora, operating up to 16x faster than standard CPU alternatives while preparing datasets for large language models.

What is the best way to prepare Common Crawl scrapes for LLM training?▼

The best way to prepare Common Crawl scrapes for LLM training is to apply GPU-accelerated quality filtering and semantic deduplication, refining web data into curated datasets saved as parquet files.

Can I use this data curation workflow for image, video, and audio modalities?▼

Yes, you can use this data curation workflow for image, video, and audio modalities, as it supports multi-modal data processing alongside text for comprehensive large language model dataset preparation.

Do I need GPU resources and Dask to run large-scale data curation?▼

Yes, you need GPU resources and dependencies like Dask, cuDF, and RAPIDS to run large-scale data curation, as the workflow requires GPU acceleration and distributed computing to process web data efficiently.

How does semantic deduplication remove similar documents from LLM training data?▼

Semantic deduplication removes semantically similar documents from LLM training data by identifying and filtering overlapping content, ensuring precise dataset creation and reducing redundancy across large-scale corpora.

Related Skills

Explore More Software Engineering Skills→
projectedanxprojectedanx
1

nemo-curator

Deduplicate and filter large-scale LLM training data with GPU-optimized pipelines.

Community
Advanced
BagelHoleBagelHole
46

llm-fine-tuning

Set up QLoRA, LoRA, and full fine-tuning infrastructure with Hugging Face TRL and Axolotl.

Community
Advanced
sayalinvidiasayalinvidia

accelerated-computing-cudf

Migrate pandas workflows to GPU-accelerated cuDF pipelines.

Community
Advanced
Yangel-hideYangel-hide

nemo-curator

Automate GPU-accelerated deduplication and quality filtering for multimodal LLM training data.

Community
Advanced