databricks-spark-declarative-pipelines

To create a Databricks streaming table, use `CREATE OR REFRESH STREAMING TABLE` syntax and enforce `STREAM` usage when reading from Auto Loader file ingestion sources to build correct serverless pipelines.

Spark Declarative Pipelines require `CREATE OR REFRESH` rather than `CREATE OR REPLACE`, use streaming tables for streaming reads, and default to serverless with Unity Catalog for medallion architecture transformations.

Implement CDC and SCD Type 2 flows in Databricks Lakeflow by using SQL for simple tasks or `pyspark.pipelines` Python patterns for complex parametrized needs, applying medallion-ready pipeline patterns for silver and gold transformations.

Databricks Lakeflow supports serverless ingestion with Unity Catalog by defaulting to serverless environments, allowing you to build bronze-to-silver-to-gold medallion architectures while validating outputs via pipeline run results and table stats.

Databricks declarative pipelines fail with `CREATE OR REPLACE` because mandatory SDP syntax constraints require `CREATE OR REFRESH` instead, distinguishing between streaming tables and materialized views to ensure correct pipeline execution.

Use SQL for simple Spark Declarative Pipeline tasks and Python with `pyspark.pipelines` for complex or parametrized needs, including CDC patterns, to route logic appropriately based on transformation complexity.