databricks-spark-declarative-pipelines

Databricks Lakeflow SDP requires using CREATE OR REFRESH STREAMING TABLE syntax for streaming ingestion, never CREATE OR REPLACE, to correctly build bronze and silver layer objects in a medallion architecture.

SCD Type 2 in Databricks declarative pipelines requires careful sequence_by column typing and key selection to track historical changes. The pipeline applies correct SDP routing rules to maintain temporal query accuracy for slowly changing dimensions.

Databricks Lakeflow SDP supports serverless Auto CDC ingestion by default. It applies safe routing and validation to ensure change data capture patterns correctly populate streaming tables without manual infrastructure configuration.

Building a medallion architecture pipeline with Spark declarative pipelines uses streaming tables for bronze and silver transformations, then materialized views for gold aggregations. The pipeline enforces correct CREATE OR REFRESH syntax for each layer.

Databricks Lakeflow SDP transformations can use Python via pyspark.pipelines code as an alternative to SQL. You must explicitly select the SQL or Python language path to ensure the generated pipeline satisfies SDP syntax requirements correctly.

Databricks SDP materialized views fail to refresh when using incorrect syntax like CREATE OR REPLACE. Correct SDP syntax requires CREATE OR REFRESH MATERIALIZED VIEW to properly execute gold layer aggregations in Lakeflow pipelines.