data-engineering

Build Medallion ELT pipelines producing null-free Gold datasets with Polars and Pydantic.

Updated Mar 2, 2022

Search Tags:#sqlalchemy #polars #medallion architecture #openlineage #elt #pydantic validation #duckdb ducklake

Ready to run

npx skills add https://github.com/ysuurme/azure_hello_world --skill data-engineering-ysuurme

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: data-engineering
Source: https://github.com/ysuurme/azure_hello_world/tree/main/.agents/skills/data-engineering
Command: npx skills add https://github.com/ysuurme/azure_hello_world --skill data-engineering-ysuurme

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill prevents brittle, ad-hoc ETL by providing a structured Medallion pipeline that validates data boundaries and enforces quality gates so downstream systems can reliably consume Gold-tier datasets.

Core Features & Use Cases

Design and implement Mini-Medallion ELT pipelines: Raw → Bronze → Silver → Gold with clear tier boundaries and deterministic data flow.
Enforce data quality and schema contracts: Pydantic validation at boundaries, null-free Gold outputs, and robust upsert/idempotency rules.
Transform and package data for ML consumption: Use Polars for transformations and produce query-ready tables or Parquet datasets backed by SQLite (prototype) or DuckLake (production).

Quick Start

Use the data-engineering skill to implement the Medallion pipeline that converts your raw source files into null-free Gold datasets using Polars transformations and Pydantic-validated contracts.

Frequently Asked Questions about data-engineering

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

What is a Medallion ELT pipeline and how does it produce ML-ready datasets?▼

A Medallion ELT pipeline processes raw data through Bronze upserts, Silver cleaning, and Gold zero-null enforcement to produce validated, query-ready datasets for ML consumption. It applies Pydantic boundary validation and Polars transformations.

How do I build a data pipeline with Polars and Pydantic validation?▼

You can build a reliable pipeline by using Polars for data transformations and applying Pydantic validation at tier boundaries. This approach enforces schema contracts and ensures null-free Gold outputs backed by SQLite or DuckLake.

Can I use DuckLake as a backend for Medallion architecture pipelines?▼

Yes, DuckLake serves as a production backend for Medallion architecture pipelines. SQLite is used for prototyping, while DuckLake supports production-scale Gold datasets outputted as query-ready tables or Parquet files.

What is the best way to enforce data quality and schema contracts in ELT?▼

The best way to enforce data quality in ELT is using Pydantic validation at boundary layers and applying zero-null enforcement on Gold datasets. This prevents brittle pipelines by ensuring deterministic data flow.

How do I track data lineage events across Bronze, Silver, and Gold tiers?▼

You can track data lineage by emitting OpenLineage events throughout the Bronze, Silver, and Gold transformation stages. This ensures integration-testable stage outputs and transparent data flow across tier boundaries.

When should I use SQLAlchemy Core versus ORM in a tiered data pipeline?▼

In a tiered data pipeline, SQLAlchemy Core and ORM usage varies by tier to handle upserts and idempotency rules effectively. This structured approach maintains deterministic data flow across Bronze, Silver, and Gold layers.