e2e-medallion-architecture

What problem does it solve?

Implement end-to-end Medallion Architecture (Bronze/Silver/Gold) lakehouse patterns in Microsoft Fabric using PySpark, Delta Lake, and Fabric Pipelines. Use when the user wants to: (1) design a Bronze/Silver/Gold data lakehouse, (2) set up multi-layer workspace with lakehouses for each tier, (3) build ingestion-to-analytics pipelines with data quality enforcement, (4) optimize Spark configurations per medallion layer, (5) orchestrate Bronze-to-Silver-to-Gold flows via notebooks. Triggers: "medallion architecture", "bronze silver gold", "lakehouse layers", "e2e data pipeline", "end-to-end lakehouse", "data lakehouse pattern", "multi-layer lakehouse", "build medallion", "setup medallion".

Core Features & Use Cases

• Design and deploy separate Bronze, Silver, and Gold lakehouses across dedicated workspaces for governance and performance.
• Bind notebooks to lakehouses and orchestrate end-to-end workflows from ingestion to analytics, including validation and governance checks.
• Enforce data quality, schema conformance, and optimization patterns per medallion layer (Bronze write-heavy, Silver balanced, Gold read-heavy).
• Automate workspace and lakehouse provisioning, notebook deployment, and sequential execution (Bronze → Silver → Gold) with verification and lifecycle management.
• Enable downstream consumption through Direct Lake connections to analytics and BI tools (e.g., Power BI) with end-to-end traceability.

Quick Start

Set up Bronze/Silver/Gold workspaces and lakehouses, deploy notebooks, bind them, and execute Bronze → Silver → Gold end-to-end.