ai-data-engineering

AI data pipelines for RAG and ML serving require orchestrated data transformations, embedding generation, and feature stores. This Skill automates end-to-end workflows, specifying integration patterns for reproducibility, data health, and monitoring across production platforms.

Orchestrating AI data engineering workflows with Dagster, Prefect, or Airflow demands robust pipeline architecture and versioning. This Skill provides architecture patterns and guardrails to ensure reproducibility, consistent feature serving, and reliable data transformations.

Generating embeddings and managing chunking strategies for AI workloads involves end-to-end data transformations. This Skill guides embedding workflows using Voyage AI and LangChain, integrating with Qdrant vector stores for reliable retrieval flows.

Feast and LakeFS ensure reproducibility in ML feature stores by enabling data versioning and consistent feature serving. This Skill specifies integration patterns for these tools, applying guardrails for data health and monitoring across production AI platforms.

This AI data engineering approach supports production-scale vector stores and evaluation by automating scalable embedding workflows. It specifies tooling integration patterns and monitoring guardrails to maintain data health and reproducibility across complex AI platforms.