mindspeed-llm-data-prep

What problem does it solve?

This Skill simplifies and automates the process of preparing diverse datasets for training MindSpeed-LLM, reducing manual effort and minimizing errors.

Core Features & Use Cases

• Data Conversion and Formatting: Converts raw datasets such as Alpaca, ShareGPT, and pretraining texts into model-ready formats.
• Flexible Data Handling: Supports multiple input formats including JSONL, JSON, Parquet, CSV, TXT, and Arrow.
• Use Case: For instance, preparing a large ChatGPT-like conversation dataset into optimized packed binary files suitable for fine-tuning large language models.

Quick Start

Import your dataset, specify the tokenizers and handlers, then run preprocessing commands to generate training-ready files.