Curate text, image, video, and audio data for LLM training datasets.
npx skills add https://github.com/superfhp/lumi-agent --skill nemo-curator-superfhp
Please help me install this Agent Skill. Skill: nemo-curator Source: https://github.com/superfhp/lumi-agent/tree/main/optional-skills/mlops/nemo-curator Command: npx skills add https://github.com/superfhp/lumi-agent --skill nemo-curator-superfhp
This Skill addresses the challenge of preparing high-quality training data for LLMs, offering GPU-accelerated data curation for text, image, video, and audio data.
Use the nemo-curator skill to deduplicate and filter text data from the 'corpus.txt' file.
High-intent search queries and answers about installing and using this skill.
You can deduplicate and filter text data from a corpus file using GPU-accelerated data curation. The process leverages RAPIDS to execute tasks like fuzzy deduplication, quality filtering, and PII redaction on your dataset.
Yes, GPU-accelerated data curation supports multimodal datasets for LLM training. It provides specific curation features for text, images, video, and audio data to ensure high-quality inputs across different data formats.
Yes, you need RAPIDS, Dask, and cudf installed to run GPU-accelerated deduplication for LLM training. These dependencies provide the necessary GPU acceleration and distributed computing environment for processing large-scale corpora.
The best way to clean web data and remove PII before LLM training is using GPU-accelerated data curation. It applies quality filtering, PII redaction, and NSFW detection to ensure your web-scraped corpora are safe and high-quality.
Semantic deduplication works by identifying and removing contextually similar entries when curating large text corpora. This GPU-accelerated data curation process goes beyond exact matching to ensure higher data diversity for LLM training.
richardnguyen0715GPU-accelerate LLM training data curation with fuzzy deduplication, quality filtering, and PII redaction.
pmcdowallAlign large language models via reinforcement learning and preference-based fine-tuning workflows.
Clay-HHKCurate and deduplicate large-scale multimodal corpora with GPU acceleration.
nelohenriqCurate large multimodal datasets with GPU-accelerated filtering, deduplication, and PII redaction.
