Curate and deduplicate LLM training datasets on GPUs with NeMo Curator.
npx skills add https://github.com/xu1713/openhorse --skill nemo-curator-xu1713
Please help me install this Agent Skill. Skill: nemo-curator Source: https://github.com/xu1713/openhorse/tree/main/openhorse/openhorse/optional-skills/mlops/nemo-curator Command: npx skills add https://github.com/xu1713/openhorse --skill nemo-curator-xu1713
High-intent search queries and answers about installing and using this skill.
Use NeMo Curator's FuzzyDuplicates module with MinHash and LSH to remove near-duplicates at scale. On GPUs it processes 8TB in about 7.5 hours, roughly 16× faster than CPU-based deduplication.
Exact dedup removes identical documents via hashing, fuzzy dedup uses MinHash+LSH for near-duplicates with about 95% recall, and semantic dedup uses embedding similarity for paraphrases. Fuzzy dedup offers the best speed-to-quality trade-off for large datasets.
NeMo Curator is GPU-accelerated with RAPIDS and supports multimodal curation, making it faster for very large datasets. Datatrove is CPU-based and open-source, better suited when GPU infrastructure is unavailable.
Yes, it includes image curation with aesthetic scoring, NSFW filtering, and CLIP embeddings, plus video curation with scene detection, clip extraction, and InternVideo2 embeddings. Audio curation supports ASR transcription and WER filtering.
Yes, install the CPU-only extra with pip install nemo-curator[cpu], but processing is significantly slower. GPU acceleration with CUDA 12 delivers 10-16× speedups on deduplication and filtering workloads.
Initialize a Dask CUDA cluster with LocalCUDACluster or get_client with cluster_type set to gpu and your desired worker count. Processing then scales near-linearly across the available GPU nodes.
big4council-progCurate multimodal training corpora with deduplication, filtering, and PII redaction.
K-Dense-AIEmbed, fine-tune, and benchmark microbiome taxonomic profiles with Outpost Bio's Waypoint foundation models.
Estrella-231Organize biological datasets with versioned, queryable artifacts and lineage tracking.
MesferAliProcesses large-scale ML datasets with distributed streaming across CPUs and GPUs.
