skills.rest
MLT-OSS
MLT-OSSOFFICIAL
Explore all skills by this author →
WebsiteGitHub

nemo-curator

Curate LLM training datasets with GPU-accelerated deduplication, filtering, and PII redaction.

14|5|Updated Apr 9, 2026
Search Tags:#deduplication#quality filtering#data curation#pii redaction#gpu acceleration#nemo curator#llm training data
One-click install
npx skills add https://github.com/MLT-OSS/hermes-agent-go --skill nemo-curator-mlt-oss
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: nemo-curator
Source: https://github.com/MLT-OSS/hermes-agent-go/tree/main/optional-skills/mlops/nemo-curator
Command: npx skills add https://github.com/MLT-OSS/hermes-agent-go --skill nemo-curator-mlt-oss

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires nemo-curator, cudf, dask, rapids, and includes references (resource) components.

What problem does it solve? Preparing high-quality training data from massive web scrapes or raw corpora is slow and expensive on CPUs, and low-quality or duplicate content degrades LLM training outcomes. ## Core Features & Use Cases - GPU-Accelerated Deduplication: Exact, fuzzy (MinHash + LSH), and semantic deduplication running up to 16× faster than CPU on multi-terabyte datasets. - Quality Filtering & Safety: 30+ heuristic filters, quality classifiers, NSFW detection, and PII redaction for text, image, video, and audio data. - Use Case: Curate a Common Crawl dump by filtering short or repetitive documents, removing near-duplicates with fuzzy dedup, redacting emails and phone numbers, and exporting clean Parquet files for LLM pretraining. ## Quick Start Ask the agent to install nemo-curator with CUDA support and build a pipeline that filters, deduplicates, and redacts PII from your Parquet dataset.

Frequently Asked Questions about nemo-curator

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I deduplicate a large text dataset for LLM training?▼

Use NeMo Curator's FuzzyDuplicates module, which applies MinHash and LSH to remove near-duplicates. On GPUs it processes 8TB in about 7.5 hours, roughly 16× faster than CPU-based deduplication.

What is the difference between exact, fuzzy, and semantic deduplication?▼

Exact deduplication removes identical documents via hashing, fuzzy deduplication catches near-duplicates using MinHash and LSH, and semantic deduplication uses embedding similarity to detect paraphrases. Fuzzy dedup offers the best speed-to-recall trade-off for large corpora.

Does NeMo Curator work without a GPU?▼

Yes, a CPU-only installation is available via the cpu extra, but it is significantly slower. GPU acceleration with RAPIDS and cuDF delivers 10-16× speedups for deduplication and filtering workloads.

Can NeMo Curator process image, video, and audio datasets?▼

Yes, it supports multimodal curation including aesthetic and NSFW filtering for images, scene detection and clip extraction for video, and ASR transcription with WER filtering for audio.

When should I use datatrove or dolma instead of NeMo Curator?▼

Use datatrove or dolma when you lack GPU infrastructure or need a CPU-based open-source pipeline. NeMo Curator is the better fit when processing multi-terabyte datasets where GPU speedups reduce cost and runtime substantially.

Related Skills

Explore More Data & Analytics Skills→
thisismynewfmail-uithisismynewfmail-ui

nemo-curator

Curate multimodal datasets with quality filtering, deduplication, and PII redaction.

Community
Advanced
K-Dense-AIK-Dense-AI
41.1k

waypoint-bio

Embed, fine-tune, and benchmark microbiome taxonomic profiles with Outpost Bio's Waypoint foundation models.

Official
Advanced
hanuminhanumin

exploratory-data-analysis

Profile biomedical research datasets and detect outliers via statistical tests.

Community
Intermediate
leesk212leesk212
1

cuml-machine-learning

Train and evaluate GPU-accelerated machine learning models on tabular data with cuML.

Community
Intermediate