skills.rest
hai-series
hai-seriesOFFICIAL
Explore all skills by this author →
GitHub

hai-datapipe

Clean text and chunk content into structured pipelines for AI processing.

3|7|Updated Jan 28, 2026
Search Tags:#rag#pipeline#nlp#chunking#data-preprocessing#text-cleaning#datapipe
One-click install
npx skills add https://github.com/hai-series/hai-framework --skill hai-datapipe
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: hai-datapipe
Source: https://github.com/hai-series/hai-framework/tree/main/packages/cli/templates/skills/hai-datapipe
Command: npx skills add https://github.com/hai-series/hai-framework --skill hai-datapipe

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Text data often arrives noisy and unstructured, requiring repetitive cleaning, chunking, and pipeline orchestration before AI processing. This Skill provides a deterministic, inline solution to preprocess and structure text for downstream tasks.

Core Features & Use Cases

  • Text Cleaning: remove HTML tags, URLs, emails, and normalize whitespace.
  • Flexible Chunking: multiple modes (sentence, paragraph, markdown, word, character, custom) to fit downstream models.
  • Pipeline Orchestration: chain clean, transform, and chunk steps into a reusable workflow for RAG readiness or document ingestion.
  • Use Case: prepare large collections of web content for embedding and indexing by a vector store.

Quick Start

Invoke the datapipe API to clean and chunk text, then run a pipeline on your raw input to obtain structured chunks.

Frequently Asked Questions about hai-datapipe

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I clean and chunk text for RAG pipelines?▼

Text cleaning for RAG involves removing HTML tags, URLs, and emails while normalizing whitespace. You can apply these transformations inline before chunking to ensure your document ingestion pipeline receives structured, noise-free input.

What is the best way to chunk markdown for document ingestion?▼

Markdown chunking splits text based on structural elements like headers and paragraphs. Using a deterministic pipeline with a markdown chunking mode preserves document semantics for accurate retrieval in RAG workflows.

Can I use custom chunking modes for text preprocessing?▼

Custom chunking modes allow you to define specific splitting logic beyond standard sentence or paragraph boundaries. This ensures the processed text aligns with the exact token limits and formatting requirements of your target AI models.

Does this text preprocessing pipeline require complex initialization?▼

The pipeline operates with no-init usage, meaning you can invoke the datapipe API directly without extensive configuration. This deterministic approach ensures consistent text cleaning and chunking results for AI-ready data.

How do I prepare web content for a vector store?▼

Preparing web content for a vector store requires removing HTML tags, URLs, and emails before normalizing whitespace. A deterministic pipeline automates this cleaning and chunking to produce structured, AI-ready data.

Related Skills

Explore More Software Engineering Skills→
zjykzkzjykzk

data-ingest

Ingest raw text data into an Obsidian wiki as searchable knowledge.

Community
Intermediate
Greenmamba29Greenmamba29

vector-embedder

Generate and store vector embeddings for semantic search pipelines.

Community
Intermediate
zhangzhang-111-izhangzhang-111-i
2

rag-architect

Designs RAG pipelines with chunking, embedding, retrieval, and evaluation strategies.

Community
Advanced
firef1iefiref1ie

agentpmt-tool-text-manipulation-and-converter-d789e9

Automate text manipulation and conversion tasks via the AgentPMT external API.

Community
Intermediate