Automate GPU-accelerated curation of multimodal LLM training datasets.
npx skills add https://github.com/objval/hermes-agent --skill nemo-curator-objval
Please help me install this Agent Skill. Skill: nemo-curator Source: https://github.com/objval/hermes-agent/tree/main/optional-skills/mlops/nemo-curator Command: npx skills add https://github.com/objval/hermes-agent --skill nemo-curator-objval
GPU-accelerated data curation for LLM training. Supports text/image/video/audio. Features fuzzy deduplication (16× faster), quality filtering (30+ heuristics), semantic deduplication, PII redaction, NSFW detection. Scales across GPUs with RAPIDS. Use for preparing high-quality training datasets, cleaning web data, or deduplicating large corpora.
Run the Nemo Curator pipeline on your dataset to produce a cleaned, de-duplicated, and redacted version ready for model fine-tuning.
High-intent search queries and answers about installing and using this skill.
GPU-accelerated data curation applies exact, fuzzy, and semantic deduplication to multimodal datasets. It processes text, image, video, and audio data across distributed GPU clusters to prepare cleaned training corpora.
PII redaction and NSFW detection sanitize web scrape data by identifying and removing sensitive or unsafe content. This ensures the resulting Parquet or JSONL datasets are safe for LLM fine-tuning.
Yes, data curation pipelines scale across distributed GPU clusters using RAPIDS and Dask. This architecture processes massive datasets by applying quality filtering and deduplication efficiently across multiple GPUs.
The best way to apply quality filtering is using over 30 built-in heuristics tailored for multimodal data. This automates the cleaning of web data to produce high-quality training datasets.
Yes, fuzzy deduplication works with Parquet and JSONL formats. The pipeline reads and writes these formats while running 16 times faster than standard methods by leveraging RAPIDS and Dask.
You need GPU clusters to achieve GPU-accelerated data curation and scale fuzzy deduplication across large corpora. The system leverages RAPIDS and Dask to distribute the workload across available GPUs.
KarlinskySDeduplicate, filter, redact PII, and remove NSFW content from large multimodal corpora.
jolequesMerge interaction logs and documentation into Vertex AI Gemini JSONL fine-tuning datasets.
AissenLiuAutomate GPU-accelerated data curation workflows for large multimodal datasets.
dawsonblockAutomates GPU-accelerated data curation for LLM training datasets.
