data_splitter

Name: data_splitter
Availability: InStock
Author: cas-bigdatalab

Split JSONL datasets into subsets by specified ratios.

541|171|Updated May 3, 2018

Search Tags:#jsonl #dataset management #data splitting #test set #training set

Ready to run

npx skills add https://github.com/cas-bigdatalab/piflow --skill data-splitter

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: data_splitter
Source: https://github.com/cas-bigdatalab/piflow/tree/main/workspace/skills/data_splitter
Command: npx skills add https://github.com/cas-bigdatalab/piflow --skill data-splitter

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) components.

What problem does it solve?

This Skill simplifies the process of splitting JSONL datasets into smaller subsets based on specified ratios, making it easier to manage and analyze data.

Core Features & Use Cases

JSONL Splitting: Divide JSONL data into multiple subsets based on given ratios.
Random Splitting: Support random splitting for better data distribution.
Stratified Splitting: Maintain category distribution when splitting by a specific field.
Use Case: For data scientists who need to create training, validation, and test sets from a large JSONL dataset.

Quick Start

Use the data_splitter skill to split the 'data.jsonl' file into training, validation, and test sets with ratios 0.7, 0.2, and 0.1 respectively.

Frequently Asked Questions about data_splitter

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I split a JSONL dataset into training and test sets?▼

To split a JSONL dataset into training and test sets, this Skill divides the data into multiple subsets based on specified ratios, such as 0.7 for training and 0.3 for testing. It requires Python and specific command-line arguments for execution.

What is stratified splitting for JSONL data?▼

Stratified splitting for JSONL data is a mechanism that maintains the category distribution of a specific field when dividing the dataset. This ensures that the resulting training, validation, and test sets accurately reflect the original data's proportions.

Can I create validation sets with random splitting from a JSONL file?▼

Yes, you can create validation sets with random splitting from a JSONL file. This Skill supports random splitting to ensure better data distribution across your training, validation, and test subsets for machine learning workflows.

Do I need Python to manage dataset splitting for machine learning?▼

Yes, you need Python to manage dataset splitting for machine learning using this Skill. It is designed as a data preprocessing task executed through Python and specific command-line arguments to generate the required subsets.