e2e-medallion-architecture

Create a schema-enabled lakehouse with bronze, silver, and gold schemas, then deploy one PySpark notebook per layer via the Fabric REST API. Bind each notebook to the lakehouse, execute them sequentially Bronze to Gold, and connect Power BI to the Gold layer using Direct Lake.

Land raw files in the lakehouse Files folder, write Bronze tables with metadata columns like ingestion timestamp and batch ID, apply deduplication and validation in Silver, and pre-aggregate metrics in Gold. Orchestrate the layers with a Fabric pipeline using sequential notebook activities.

Prefer one schema-enabled lakehouse with bronze, silver, and gold schemas, which supports Materialized Lake Views and row-level security out of box. Use separate lakehouses or workspaces only when schema-enabled lakehouses are unavailable or governance requires isolation.

No, Fabric Spark cannot read from arbitrary HTTP or HTTPS URLs. Land external data in the lakehouse Files folder first using a pipeline Copy activity, OneLake API upload, or OneLake shortcut, then read from the lakehouse-relative path.

Notebook failures often come from invalid .ipynb structure, such as missing "outputs": [] or "execution_count": null on code cells, or from a missing default lakehouse binding. Ensure metadata.dependencies.lakehouse includes the lakehouse ID, name, and workspace ID.

Discover the Gold lakehouse SQL endpoint from the lakehouse properties, wait for provisioning to succeed, then create a semantic model in Direct Lake mode referencing the Gold Delta tables. Build a report on that model and validate with DAX queries.