databricks-spark-declarative-pipelines

To create a Databricks Spark declarative pipeline for a medallion architecture, initialize a standalone project using the databricks pipelines init command with a config file defining your catalog and schema. This automatically generates bronze to silver to gold templates with proper naming conventions and multi-schema patterns.

The correct syntax for creating streaming tables in Databricks Lakeflow pipelines is `CREATE OR REFRESH`. Using this declarative syntax for streaming tables and materialized views avoids legacy `CREATE OR REPLACE` commands and ensures proper pipeline updates.

Yes, you can configure serverless compute and liquid clustering for Spark declarative pipelines. The setup automatically applies serverless defaults, liquid clustering, and best-practice table properties during pipeline creation and updates without manual configuration.

Yes, this approach supports change data capture and SCD Type 2 workflows in both SQL and Python. It provides detailed patterns for AUTO CDC, SCD Type 2, and change-data-capture flows, including automatic schema handling for Auto Loader and CDC workflows.

The best way to migrate from Delta Live Tables to Spark declarative pipelines is by using enforced `CREATE OR REFRESH` syntax and serverless compute defaults. This approach eliminates manual configuration guesswork and reduces syntax errors during the migration process.

Automatic schema handling with Auto Loader in declarative pipelines is designed specifically for CDC workflows and serverless compute environments. Limitations arise when manual schema overrides are required, as the pipeline defaults to automatic schema handling and serverless configurations.