e2e-medallion-architecture

Create one workspace with a schema-enabled lakehouse containing bronze, silver, and gold schemas, then deploy one notebook per layer. Bind each notebook to the lakehouse, execute Bronze to Silver to Gold sequentially via the Fabric REST API, and connect Power BI to the Gold layer.

Use Materialized Lake Views when the transformation is expressible in Spark SQL and benefits from declarative incremental refresh; they require a schema-enabled lakehouse. Use notebooks when you need PySpark UDFs, complex Python logic, or full control over write strategies.

No, Fabric Spark cannot access arbitrary external HTTP or HTTPS URLs. Land data in the lakehouse Files folder first using a Fabric Pipeline Copy activity, the OneLake API, or OneLake shortcuts, then read from the lakehouse-relative path.

Notebook creation fails silently when the .ipynb payload is missing "outputs": [] or "execution_count": null on code cells, or when no default lakehouse is bound. Include metadata.dependencies.lakehouse with the lakehouse ID, name, and workspace ID in the updateDefinition payload.

Enable V-Order with spark.sql.parquet.vorder.default=true and Optimize Write with spark.databricks.delta.optimizeWrite.enabled=true and binSize of 1g before writes. Run OPTIMIZE with ZORDER on frequently filtered columns after writes to improve Direct Lake and SQL endpoint read performance.