Curate multimodal LLM training data with GPU-accelerated filtering and deduplication.
npx skills add https://github.com/ovachiever/droid-tings --skill nemo-curator
Please help me install this Agent Skill. Skill: nemo-curator Source: https://github.com/ovachiever/droid-tings/tree/main/skills/nemo-curator Command: npx skills add https://github.com/ovachiever/droid-tings --skill nemo-curator
NeMo Curator provides GPU-accelerated data curation for large multimodal datasets used in LLM training, enabling fast deduplication, high-quality filtering, PII redaction, and NSFW detection across text, image, video, and audio data.
Install Nemo Curator, load 600+ tools, and run a simple corpus curation pipeline that filters low-quality documents, deduplicates, redacts PII, and saves to Parquet.
High-intent search queries and answers about installing and using this skill.
GPU-accelerated deduplication identifies and removes duplicate records across massive datasets 16× faster than CPU methods. Nemo Curator supports exact, fuzzy, and semantic deduplication, scaling across GPU clusters to handle multi-terabyte corpora like RedPajama v2 efficiently.
Multimodal data curation applies 30+ quality heuristics across all four formats simultaneously. Nemo Curator processes text, image, video, and audio through unified filtering pipelines, enabling consistent quality standards across heterogeneous training datasets.
PII redaction and NSFW detection run as GPU-accelerated pipeline stages before training data export. Nemo Curator redacts sensitive information and filters unsafe content at scale, protecting privacy and safety compliance without manual review.
RAPIDS and Dask enable distributed processing across GPU clusters for stage-based pipelines. Nemo Curator leverages multi-GPU scaling to parallelize filtering, deduplication, and redaction across nodes, reducing curation time for massive corpora.
Semantic deduplication finds duplicate meaning beyond exact text matches, catching near-duplicates exact matching misses. Choose semantic deduplication for web-scraped or noisy corpora; exact matching suffices for clean, deduplicated sources already prepared.
Input data integrates from web scrapes and large text corpora, typically as raw documents or structured formats. Nemo Curator outputs curated datasets to Parquet, enabling direct ingestion into LLM training frameworks after all filtering and deduplication stages.
projectedanxDeduplicate and filter large-scale LLM training data with GPU-optimized pipelines.
overviewlabsCurate multi-modal datasets for LLM training with GPU-accelerated filters and deduplication.
richardnguyen0715GPU-accelerate LLM training data curation with fuzzy deduplication, quality filtering, and PII redaction.
SufficientDaikonDetect and deduplicate text content using normalization, SHA-256 fingerprinting, and Jaccard similarity.
