What problem does it solve?
Implement end-to-end Medallion Architecture (Bronze/Silver/Gold) lakehouse patterns in Microsoft Fabric using PySpark, Delta Lake, and Fabric Pipelines. Use when the user wants to: (1) design a Bronze/Silver/Gold data lakehouse, (2) set up multi-layer workspace with lakehouses for each tier, (3) build ingestion-to-analytics pipelines with data quality enforcement, (4) optimize Spark configurations per medallion layer, (5) orchestrate Bronze-to-Silver-to-Gold flows via notebooks. Triggers: "medallion architecture", "bronze silver gold", "lakehouse layers", "e2e data pipeline", "end-to-end lakehouse", "data lakehouse pattern", "multi-layer lakehouse", "build medallion", "setup medallion".
Core Features & Use Cases
- Design and deploy separate Bronze, Silver, and Gold lakehouses across dedicated workspaces for governance and performance.
- Bind notebooks to lakehouses and orchestrate end-to-end workflows from ingestion to analytics, including validation and governance checks.
- Enforce data quality, schema conformance, and optimization patterns per medallion layer (Bronze write-heavy, Silver balanced, Gold read-heavy).
- Automate workspace and lakehouse provisioning, notebook deployment, and sequential execution (Bronze → Silver → Gold) with verification and lifecycle management.
- Enable downstream consumption through Direct Lake connections to analytics and BI tools (e.g., Power BI) with end-to-end traceability.
Quick Start
Set up Bronze/Silver/Gold workspaces and lakehouses, deploy notebooks, bind them, and execute Bronze → Silver → Gold end-to-end.
