skills.rest
ldzhhxx
ldzhhxxCOMMUNITY
Explore all skills by this author →
GitHub

nemo-curator

Automate GPU-accelerated data curation for LLM training with RAPIDS.

1|1|Updated May 9, 2026
Search Tags:#GPU acceleration#deduplication#LLM training#quality filtering#PII redaction#data curation
One-click install
npx skills add https://github.com/ldzhhxx/Hermes_offline_v2 --skill nemo-curator-ldzhhxx
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: nemo-curator
Source: https://github.com/ldzhhxx/Hermes_offline_v2/tree/main/hermes-agent/optional-skills/mlops/nemo-curator
Command: npx skills add https://github.com/ldzhhxx/Hermes_offline_v2 --skill nemo-curator-ldzhhxx

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires nemo-curator, cudf, dask, rapids, and includes scripts (resource) and references (resource) and assets (resource) components.

What problem does it solve?

This Skill automates the process of preparing high-quality training datasets for LLMs by providing advanced data curation tools for text, image, video, and audio data, ensuring faster and higher-quality dataset preparation.

Core Features & Use Cases

  • Multi-modal Data Curation: Supports text, images, video, and audio, allowing users to clean and prepare complex datasets.
  • Fuzzy Deduplication: Removes near-duplicate content up to 16 times faster than CPU-based methods.
  • Quality Filtering: Applies more than 30 heuristic filters to remove low-quality content.
  • Semantic Deduplication: Uses embeddings to find semantically similar documents and remove them.
  • PII Redaction: Identifies and redacts personally identifiable information to ensure privacy.
  • NSFW Detection: Filters out non-safe-for-work content to maintain standards.
  • Use Cases: Suitable for web scraping, cleaning datasets, deduplicating large corpora, and preparing datasets for LLM training.

Quick Start

Install nemo-curator and run the command 'nemo-curator text-curate "input-data.csv" -o "output-data.csv" --min-words 50 --max-words 50000' to filter and deduplicate text data.

Frequently Asked Questions about nemo-curator

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I accelerate fuzzy deduplication for large text datasets?▼

GPU-accelerated fuzzy deduplication removes near-duplicate content up to 16 times faster than CPU-based methods by utilizing RAPIDS and CUDA compatible libraries.

What is the best way to redact PII and filter NSFW content during LLM data curation?▼

Data curation for LLM training includes built-in PII redaction to ensure privacy and NSFW detection to filter out non-safe-for-work content automatically.

How do I perform semantic deduplication on a training corpus?▼

Semantic deduplication finds semantically similar documents using embeddings and removes them, ensuring higher-quality dataset preparation for LLMs.

Does GPU-accelerated data curation work with multi-modal data like images and video?▼

Multi-modal data curation supports text, images, video, and audio, allowing users to clean and prepare complex datasets across GPU clusters.

Do I need CUDA and RAPIDS installed to run data curation scripts?▼

Yes, scaling data curation across GPU clusters requires CUDA and compatible libraries like RAPIDS, cudf, and dask to execute the acceleration.

How do I apply quality filtering to remove low-quality content from web scraping data?▼

Quality filtering applies more than 30 heuristic filters to remove low-quality content from web scraping data or large corpora efficiently.

Related Skills

Explore More Software Engineering Skills→
markwang2658markwang2658
31

nemo-curator

Automates GPU-accelerated data curation for LLM training datasets with deduplication and quality filtering.

Community
Advanced
pmcdowallpmcdowall

fine-tuning-with-trl

Align large language models via reinforcement learning and preference-based fine-tuning workflows.

Community
Advanced
ovachieverovachiever
52

nemo-curator

Curate multimodal LLM training data with GPU-accelerated filtering and deduplication.

Community
Advanced
Yangel-hideYangel-hide

nemo-curator

Automate GPU-accelerated deduplication and quality filtering for multimodal LLM training data.

Community
Advanced