Filter, deduplicate, and redact large Parquet corpora on GPUs.
npx skills add https://github.com/qcmuu/AI-Research-Skills --skill nemo-curator-qcmuu
Please help me install this Agent Skill. Skill: nemo-curator Source: https://github.com/qcmuu/AI-Research-Skills/tree/main/05-data-processing/nemo-curator Command: npx skills add https://github.com/qcmuu/AI-Research-Skills --skill nemo-curator-qcmuu
NeMo Curator eliminates the time-consuming work of cleaning, filtering, and deduplicating massive web-scale datasets so your LLM training corpus is higher quality and safer.
Use the nemo-curator skill to run a GPU curation pipeline that filters, deduplicates, and redacts a large Parquet corpus into a cleaned output dataset.
High-intent search queries and answers about installing and using this skill.
You can clean and deduplicate Parquet datasets for LLM training by applying GPU-accelerated quality filtering, multi-strategy deduplication, and PII redaction to produce high-quality, training-ready outputs.
Filtering low-quality content from Common Crawl data is best achieved by applying over 30 heuristic filters and optional quality classifiers to remove undesirable text before model training.
GPU acceleration with RAPIDS significantly improves data curation throughput by running high-throughput workflows across single- and multi-GPU settings for text and multimodal inputs.
To redact personally identifiable information before training, you apply PII redaction modules during the data curation pipeline, ensuring privacy-safe preprocessing across your web-scale corpora.
You can perform fuzzy deduplication using MinHash and LSH, alongside semantic embedding-based deduplication, to effectively remove exact and near-duplicates from web-scale corpora.
You need NVIDIA GPUs because the curation pipeline requires RAPIDS-based execution and dependencies like cudf and dask to accelerate the filtering and deduplication workflows.
Clay-HHKCurate and deduplicate large-scale multimodal corpora with GPU acceleration.
twjarviszyra-webConfigure Axolotl YAML for LLM fine-tuning and evaluation workflows.
jleechanorgAutomate GPU-accelerated filtering, deduplication, PII redaction, and NSFW detection on multimodal datasets.
carterwayneskhizeineCurate noisy training data with GPU-accelerated deduplication and PII redaction.
