skills.rest
pvliesdonk
pvliesdonkCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

data-patterns

Provide patterns for RAG data pipelines with chunking, validation, and evaluation.

3|Updated Feb 15, 2026
Search Tags:#embeddings#llm#rag#vector database#data pipeline#data validation#chunking
One-click install
npx skills add https://github.com/pvliesdonk/agents.md --skill data-patterns
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: data-patterns
Source: https://github.com/pvliesdonk/agents.md/tree/main/skills/data-patterns
Command: npx skills add https://github.com/pvliesdonk/agents.md --skill data-patterns

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill provides essential patterns and best practices for building reliable and efficient data pipelines for Retrieval Augmented Generation (RAG) systems, addressing challenges in data preparation, storage, and validation.

Core Features & Use Cases

  • Chunking Strategies: Offers various methods (fixed-size, semantic, document-aware) for optimal text splitting.
  • Vector Store Selection: Guides users on choosing the right vector database based on scale, features, and hosting needs.
  • Data Validation: Implements robust validation using Pydantic and Pandera for structured data and embeddings.
  • Schema Evolution: Defines strategies for managing changes in data schemas over time.
  • RAG Evaluation: Provides metrics and approaches for assessing retrieval and end-to-end RAG performance.
  • Use Case: When developing a RAG system for customer support documentation, this Skill helps select the best chunking strategy for technical articles, choose an appropriate vector store like Qdrant for scalability, and implement validation to ensure data quality before indexing.

Quick Start

Use the data-patterns skill to explore chunking strategies for technical documentation.

Frequently Asked Questions about data-patterns

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
What is the best way to choose a chunking strategy for my RAG data pipeline?▼

RAG data pipeline chunking strategy selection depends on document structure, with options including fixed-size, semantic, and document-aware splitting to optimize text retrieval. Document-aware chunking is typically best for technical articles, ensuring context boundaries are maintained for accurate LLM generation.

How do I validate embeddings and structured data before indexing in a vector database?▼

Validate embeddings and structured data in a RAG pipeline using Pydantic and Pandera to enforce schemas before indexing into a vector database. This ensures structured data and embedding quality, preventing corrupted vectors from degrading LLM retrieval performance.

How do I select the right vector database for a production-ready RAG system?▼

Vector database selection for production RAG systems requires evaluating scale, features, and hosting needs to match your specific data throughput requirements. Choosing a scalable vector store like Qdrant ensures the infrastructure handles high-volume retrieval efficiently.

What metrics are used for RAG evaluation and assessing retrieval performance?▼

RAG evaluation metrics assess both retrieval accuracy and end-to-end generation performance to measure how well the LLM pipeline functions. Applying these metrics helps identify anti-patterns and ensures the RAG system delivers robust, high-quality results.

How do I manage schema evolution in a RAG data pipeline?▼

Manage schema evolution in a RAG data pipeline by defining strategies that accommodate changes in data schemas over time without breaking existing vector indexes. This involves structured validation updates to ensure continuous data quality during embedding model optimization.

Related Skills

Explore More Software Engineering Skills→
AngelP17AngelP17
1

ml-pipeline-workflow

Automate end-to-end ML pipelines from data ingestion to deployment.

Community
Advanced
HemantSudarshanHemantSudarshan

rag-engineer

Orchestrate document retrieval and generation for end-to-end RAG workflows.

Community
Advanced
dlantsdlants
7

pkb

Index local Markdown files and retrieve semantic search results with contextual information.

Community
Advanced
rfdiosuaorfdiosuao
30

vector-db

Guide vector database operations, embedding selection, indexing, and RAG pipeline design.

Community
Advanced