hai-datapipe

Name: hai-datapipe
Availability: InStock
Author: hai-series

Clean text and chunk content into structured pipelines for AI processing.

3|7|Updated Jan 28, 2026

Search Tags:#rag #pipeline #nlp #chunking #data-preprocessing #text-cleaning #datapipe

Ready to run

npx skills add https://github.com/hai-series/hai-framework --skill hai-datapipe

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: hai-datapipe
Source: https://github.com/hai-series/hai-framework/tree/main/packages/cli/templates/skills/hai-datapipe
Command: npx skills add https://github.com/hai-series/hai-framework --skill hai-datapipe

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Text data often arrives noisy and unstructured, requiring repetitive cleaning, chunking, and pipeline orchestration before AI processing. This Skill provides a deterministic, inline solution to preprocess and structure text for downstream tasks.

Core Features & Use Cases

Text Cleaning: remove HTML tags, URLs, emails, and normalize whitespace.
Flexible Chunking: multiple modes (sentence, paragraph, markdown, word, character, custom) to fit downstream models.
Pipeline Orchestration: chain clean, transform, and chunk steps into a reusable workflow for RAG readiness or document ingestion.
Use Case: prepare large collections of web content for embedding and indexing by a vector store.

Quick Start

Invoke the datapipe API to clean and chunk text, then run a pipeline on your raw input to obtain structured chunks.

Frequently Asked Questions about hai-datapipe

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I clean and chunk text for RAG pipelines?▼

Text cleaning for RAG involves removing HTML tags, URLs, and emails while normalizing whitespace. You can apply these transformations inline before chunking to ensure your document ingestion pipeline receives structured, noise-free input.

What is the best way to chunk markdown for document ingestion?▼

Markdown chunking splits text based on structural elements like headers and paragraphs. Using a deterministic pipeline with a markdown chunking mode preserves document semantics for accurate retrieval in RAG workflows.

Can I use custom chunking modes for text preprocessing?▼

Custom chunking modes allow you to define specific splitting logic beyond standard sentence or paragraph boundaries. This ensures the processed text aligns with the exact token limits and formatting requirements of your target AI models.

Does this text preprocessing pipeline require complex initialization?▼

The pipeline operates with no-init usage, meaning you can invoke the datapipe API directly without extensive configuration. This deterministic approach ensures consistent text cleaning and chunking results for AI-ready data.

How do I prepare web content for a vector store?▼

Preparing web content for a vector store requires removing HTML tags, URLs, and emails before normalizing whitespace. A deterministic pipeline automates this cleaning and chunking to produce structured, AI-ready data.