databricks-spark-declarative-pipelines

To build a Databricks medallion architecture pipeline using serverless compute, define your bronze, silver, and gold assets as streaming tables and materialized views. You can structure and deploy these using Asset Bundles for consistent multi-environment promotion.

Auto Loader ingestion and AUTO CDC in Spark Declarative Pipelines work by automatically processing streaming data sources and applying SCD Type 1 or Type 2 changes. This captures data modifications directly without manual delta tracking logic.

The best way to migrate from DLT to Databricks serverless pipelines is to use the provided migration guidance. You transition existing definitions to Spark Declarative Pipelines, utilizing Asset Bundles for configuration and serverless compute for execution.

Yes, you can manage Databricks SDP pipeline lifecycle using MCP. The workflow includes initializing your project, customizing transformations, and applying MCP-based management to handle deployments across different environments.

You deploy Databricks pipelines across multiple environments by initializing an SDP project with databricks pipelines init. You then customize your transformations and deploy the Asset Bundle directly to your target development environment.

You should use SCD Type 2 in a serverless Databricks pipeline when tracking historical data changes is required. The AUTO CDC feature handles SCD Type 2 automatically within your silver or gold layers to maintain historical accuracy.