Process text, image, video, and audio data with GPU-accelerated curation.
npx skills add https://github.com/519lab/thoth-agent --skill nemo-curator-519lab
Please help me install this Agent Skill. Skill: nemo-curator Source: https://github.com/519lab/thoth-agent/tree/main/optional-skills/mlops/nemo-curator Command: npx skills add https://github.com/519lab/thoth-agent --skill nemo-curator-519lab
This Skill streamlines the preparation of high-quality training data for LLMs, automating data curation tasks like deduplication, filtering, and redaction.
Use nemo-curator to prepare a dataset by running the following commands:
uv pip install "nemo-curator[all_cuda12]"
from nemo_curator import ScoreFilter, Modify
from nemo_curator.datasets import DocumentDataset
df = pd.DataFrame({"text": ["Example text", "Another example"]})
dataset = DocumentDataset(df)
filtered = ScoreFilter(lambda doc: len(doc["text"].split()) > 5)(dataset)
filtered.to_parquet("curated_data/")
High-intent search queries and answers about installing and using this skill.
GPU-accelerated data curation for LLM training automates deduplication and quality filtering using CUDA. By leveraging GPUs, it significantly reduces processing time for large multimodal datasets.
Yes, GPU-accelerated data curation supports multimodal data including text, images, video, and audio. It automates PII redaction alongside deduplication and quality filtering to prepare datasets for LLM training.
Yes, nemo-curator requires the nemo-curator library and CUDA for GPU acceleration. It depends on RAPIDS, cuDF, and Dask to execute fast data curation tasks like filtering and deduplication.
Using GPU-accelerated data curation streamlines the preparation of high-quality training data. It automates the cleaning, deduplication, and filtering of web data to ensure optimal LLM training datasets.
You can use the ScoreFilter function from the nemo-curator library to filter documents based on text length. It processes DocumentDataset objects and exports the curated data to parquet format.
LynxLabVNAccelerate LLM training dataset curation with GPU computing and RAPIDS.
ricableAccelerate LLM fine-tuning with LoRA and QLoRA on local hardware.
markwang2658Automates GPU-accelerated data curation for LLM training datasets with deduplication and quality filtering.
NITISH-gitbitDeduplicate and filter LLM training datasets with GPU acceleration.