skills.rest
ancoleman
ancolemanCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

ai-data-engineering

Build RAG pipelines, feature stores, and embedding generation for AI/ML systems.

503|73|Updated Nov 13, 2025
Search Tags:#embeddings#rag#vector database#orchestration#data engineering#feature store
One-click install
npx skills add https://github.com/ancoleman/ai-design-components --skill ai-data-engineering
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: ai-data-engineering
Source: https://github.com/ancoleman/ai-design-components/tree/main/skills/ai-data-engineering
Command: npx skills add https://github.com/ancoleman/ai-design-components --skill ai-data-engineering

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires langchain, langchain-core, langchain-openai, langchain-voyageai, langchain-qdrant, qdrant-client, ragas, datasets, feast, dagster, dagster-webserver, lakefs-client, and includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill enables the creation and management of robust data infrastructure essential for AI/ML systems, including RAG pipelines, feature stores, and embedding generation.

Core Features & Use Cases

  • RAG Pipelines: Build end-to-end RAG systems from ingestion to evaluation.
  • Feature Stores: Implement ML feature serving with Feast to prevent training-serving skew.
  • Embedding Generation: Create high-quality embeddings using state-of-the-art models.
  • Orchestration: Manage complex data workflows with Dagster or Prefect.
  • Use Case: Develop a RAG pipeline for customer support documentation, enabling semantic search and question answering over your knowledge base.

Quick Start

Use the ai-data-engineering skill to set up a basic RAG pipeline by chunking documents and generating embeddings.

Frequently Asked Questions about ai-data-engineering

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I build a RAG pipeline for semantic search over my documentation?▼

Building a RAG pipeline for semantic search requires chunking documents, generating embeddings, and storing them in a vector database. This Skill orchestrates that workflow using LangChain and Qdrant, end-to-end.

What is the best way to prevent training-serving skew in ML feature pipelines?▼

Preventing training-serving skew in ML feature pipelines is achieved by implementing a feature store. This Skill uses Feast to provide consistent feature serving for both training and online inference.

Can I use Dagster for orchestrating AI data workflows with LangChain?▼

Yes, you can use Dagster for orchestrating AI data workflows with LangChain. This Skill integrates Dagster to manage complex data workflows, including RAG pipelines and embedding generation tasks.

Does this approach support data versioning for RAG pipelines and feature stores?▼

This approach supports data versioning for RAG pipelines and feature stores through lakefs-client. It manages data versions to ensure reproducibility across your AI and ML data infrastructure workflows.

How do I evaluate RAG pipelines after generating embeddings?▼

To evaluate RAG pipelines after generating embeddings, this Skill integrates ragas with datasets. It provides evaluation mechanisms to measure the retrieval and generation quality of your workflows.

When do I need a vector database like Qdrant for AI data infrastructure?▼

You need a vector database like Qdrant for AI data infrastructure when building RAG pipelines or semantic search applications. It stores and retrieves high-quality embeddings generated by models like VoyageAI.

Related Skills

Explore More Software Engineering Skills→
RajanChavadaRajanChavada
3

data-ml-project

Coordinate data ingestion, preprocessing, training, evaluation, and deployment workflows.

Community
Advanced
ivanshamaevivanshamaev
14

feature-store

Build a Feast feature repository with Entities, FeatureViews, and FeatureServices.

Community
Advanced
Himanshu040604Himanshu040604

embedding-strategies

Select and optimize embedding models for semantic search and RAG applications.

Community
Advanced
masermediagroup-stackmasermediagroup-stack
1

ai-data-engineering

Automate AI data pipelines, feature stores, and embedding workflows.

Community
Advanced