fabric-medallion

Create separate Lakehouses or schemas for Bronze, Silver, and Gold layers. Ingest raw data into Bronze with metadata columns, cleanse and deduplicate into Silver using MERGE upserts, and build star schema tables in Gold with V-Order and OPTIMIZE for Direct Lake consumption.

Use the DeltaTable API merge method with whenMatchedUpdateAll and whenNotMatchedInsertAll on a key column like order_id. Read incremental Bronze data filtered by ingestion timestamp, apply Silver transformations, then execute the merge against the target Delta table.

Use Tables for structured data consumed by Power BI or the SQL Analytics Endpoint, since they provide ACID transactions and Direct Lake support. Use Files only for raw landing zones, archives, and reference files in any format.

Fallback happens when the semantic model uses T-SQL views, tables lack V-Order, data contains complex types like arrays, or tables have too many small files. Materialize views as Delta tables, enable V-Order, flatten nested data in Silver, and run OPTIMIZE regularly.

Use shortcuts when data already exists in ADLS Gen2, S3, Dataverse, or another Fabric workspace and you want zero-copy real-time access. Use Copy Activity when you need local storage, lowest query latency, or data isolated from source changes.

Bronze typically retains 30-90 days of raw data, Silver retains 1-2 years of cleansed data, and Gold retains 3 or more years for compliance. Enforce retention by running VACUUM with matching RETAIN HOURS values on each Delta table.