skills.rest
ivanshamaev
ivanshamaevCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

medallion_architecture

Design Bronze, Silver, and Gold Iceberg lakehouse layers with MERGE-based deduplication.

14|1|Updated May 5, 2026
Search Tags:#deduplication#iceberg#medallion#cdc#data lakehouse#data quality gates#merge upsert
One-click install
npx skills add https://github.com/ivanshamaev/de-agent-skills --skill medallion-architecture-ivanshamaev
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: medallion_architecture
Source: https://github.com/ivanshamaev/de-agent-skills/tree/main/skills/medallion_architecture
Command: npx skills add https://github.com/ivanshamaev/de-agent-skills --skill medallion-architecture-ivanshamaev

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Medallion architecture turns messy, source-native lakehouse data into reliable, queryable datasets by enforcing a disciplined progression from raw ingestion to cleansed truth and then business-ready serving tables.

Core Features & Use Cases

  • Define Bronze/Silver/Gold responsibilities: model an immutable raw archive (Bronze), a typed and deduplicated operational truth (Silver), and consumer-optimized outputs (Gold).
  • Implement end-to-end load patterns: generate DDL/DML for Iceberg-based Parquet lakehouses, including incremental pipelines, MERGE-based upserts, and streaming/CDC micro-batch designs.
  • Handle deduplication and correctness: apply row_number, MERGE semantics, hash/CDC ordering, watermark strategies, and schema evolution while adding data quality gates between layers.
  • Tune storage for performance: set partitioning/sorting per layer (e.g., ingestion date vs business event date vs reporting period) and support late-arriving data with safe reprocessing/backfills.

Quick Start

Use the medallion_architecture skill to design your Bronze/Silver/Gold Iceberg tables and incremental MERGE + dedup approach for a CDC or batch ingestion scenario.

Frequently Asked Questions about medallion_architecture

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I design Bronze, Silver, and Gold layers for an Iceberg data lakehouse?▼

Medallion architecture standardizes lakehouse data flows by modeling Bronze as an immutable raw archive, Silver as typed and deduplicated operational truth, and Gold as consumer-optimized outputs on Iceberg/Parquet.

What is the best way to deduplicate data and handle late arrivals in a data lakehouse?▼

Deduplication in a data lakehouse is handled using row_number, MERGE semantics, hash/CDC ordering, and watermark strategies, while safe reprocessing and backfills manage late-arriving data across layers.

How do I build incremental CDC pipelines with MERGE-based upserts in Parquet lakehouses?▼

Incremental CDC pipelines in Parquet lakehouses use streaming and micro-batch designs with deterministic MERGE-based upsert logic to update tables, ensuring schema evolution and data quality gates between layers.

Does the medallion architecture pattern support both batch and streaming ingestion sources?▼

Yes, the medallion architecture pattern supports batch, streaming, and CDC sources within data lakehouse designs, generating layer-specific DDL/DML patterns for incremental pipelines on Iceberg tables.

How should I configure partitioning and sorting strategies for each medallion layer?▼

Partitioning and sorting are tuned per medallion layer purpose: ingestion date for Bronze, business event date for Silver, and reporting period for Gold, optimizing storage performance and query efficiency.

Related Skills

Explore More Data & Analytics Skills→
augustoheissaugustoheiss

agency-data-engineer

Design idempotent ETL/ELT pipelines with medallion lakehouse architectures.

Community
Advanced
henkisdabrohenkisdabro
77

ga4-bigquery

Export GA4 event data to BigQuery and query nested fields with SQL.

Community
Advanced
scumunnascumunna
1

pipeline-orchestration-and-data-quality

Orchestrate marketing data pipelines with idempotent loads and data quality checks.

Community
Advanced
erichowenserichowens
181

data-pipeline-engineer

Automate ETL/ELT pipeline design, validation, and operation across medallion architectures.

Community
Advanced