Retrieve datasets from HuggingFace Hub and format them as JSONL.
npx skills add https://github.com/cas-bigdatalab/piflow --skill remote-formatter
Please help me install this Agent Skill. Skill: remote_formatter Source: https://github.com/cas-bigdatalab/piflow/tree/main/workspace/skills/remote_formatter Command: npx skills add https://github.com/cas-bigdatalab/piflow --skill remote-formatter
This Skill automates the process of loading datasets from HuggingFace Hub, streamlining data preparation for analysis and model training.
Load the 'rotten_tomatoes' dataset from HuggingFace Hub and format it as JSONL:
python scripts/run_remote_formatter.py --dataset_path "rotten_tomatoes" --output_path /path/to/output.jsonl
High-intent search queries and answers about installing and using this skill.
To load datasets from HuggingFace Hub and convert them to JSONL, you can use the remote_formatter script to automatically download and format the data into JSONL for machine learning workflows.
Automating data preprocessing for HuggingFace datasets involves retrieving the dataset and formatting it. This Skill handles the retrieval and JSONL formatting automatically, streamlining data preparation for analysis and model training.
Yes, you need to install the data_juicer library and set up a Python environment. This dependency is required to run the remote_formatter script and load datasets from HuggingFace Hub.
Yes, you can load private datasets from HuggingFace Hub. The remote_formatter supports retrieving both public and private datasets and automates their conversion into JSONL format.
To download a HuggingFace dataset and format it as JSONL, run the Python script with arguments specifying the dataset path and output path, such as passing the dataset name and the target JSONL file location.
EurecaMomentClean and preprocess datasets using Data-Juicer tools.
InverterNetworkCurate large-scale LLM training datasets with GPU-accelerated deduplication, filtering, and PII redaction.
mindmorassIdentify data patterns, anomalies, and actionable insights from datasets.
dathereConvert tabular data files between CSV, TSV, Excel, JSONL, and Parquet formats.