Filter, deduplicate, and redact web-scale datasets for LLM training.
npx skills add https://github.com/JamesFincher/gengar --skill nemo-curator-jamesfincher
Please help me install this Agent Skill. Skill: nemo-curator Source: https://github.com/JamesFincher/gengar/tree/main/optional-skills/mlops/nemo-curator Command: npx skills add https://github.com/JamesFincher/gengar --skill nemo-curator-jamesfincher
NeMo Curator removes low-quality, duplicate, and sensitive content from large web-derived datasets so your LLM training data is cleaner and more useful.
Use the nemo-curator skill to curate a raw dataset and write a deduplicated, filtered Parquet output you can feed into your next LLM training run.
High-intent search queries and answers about installing and using this skill.
To clean and deduplicate LLM training data, you can apply heuristic quality filters, GPU-accelerated exact and fuzzy matching, and PII redaction to raw web corpora, outputting a filtered Parquet dataset ready for model training.
GPU-accelerated deduplication processes large text corpora using RAPIDS, cuDF, and Dask to distribute MinHash+LSH and embedding similarity computations across GPUs, rapidly eliminating exact, near-duplicate, and semantically similar documents.
Yes, NeMo Curator requires a GPU-capable execution environment to leverage NVIDIA RAPIDS and Dask modules for distributed processing, ensuring sufficient throughput for web-scale dataset filtering and deduplication tasks.
The best way to remove PII and NSFW content from multimodal training data is to apply automated redaction and detection pipelines that process text, images, video, and audio records to ensure safety hygiene before model training.
Yes, you can use Dask and RAPIDS to distribute data curation workflows on Common Crawl, enabling GPU-accelerated filtering, deduplication, and sensitive content redaction across large-scale multimodal records.
Heuristic classifiers filter short, noisy, and repetitive documents using rule-based scores, but they may miss nuanced semantic quality issues that ML-based classifiers catch, requiring combined approaches for optimal corpus preparation.
Clay-HHKCurate and deduplicate large-scale multimodal corpora with GPU acceleration.
nelohenriqCurate large multimodal datasets with GPU-accelerated filtering, deduplication, and PII redaction.
richardnguyen0715GPU-accelerate LLM training data curation with fuzzy deduplication, quality filtering, and PII redaction.
NITISH-gitbitDeduplicate and filter LLM training datasets with GPU acceleration.
