databricks-spark-declarative-pipelines

Automating Databricks pipeline deployments involves using Databricks Asset Bundles and the modern Python API to configure and update SDP/LDP pipelines. This approach enables multi-environment deployments through variable-driven configurations and MCP tooling.

Migrating from DLT to serverless Databricks pipelines requires setting up SDP/LDP configurations with serverless compute defaults. This skill facilitates the migration while supporting bronze/silver/gold medallion architecture patterns.

Implementing CDC and SCD Type 2 in a Databricks medallion architecture is handled through SDP/LDP pipeline configurations. The process supports Auto Loader ingestion and applies these patterns across bronze, silver, and gold layers.

Python and SQL are both supported for Databricks serverless pipeline configuration. The setup allows language selection within SDP/LDP pipelines while maintaining Unity Catalog usage for governance.

Managing multi-environment governance for Databricks SDP pipelines utilizes Unity Catalog and variable-driven configurations. This ensures consistent deployment, governance, and maintenance across different environments using Asset Bundles.

Auto Loader ingestion works with serverless Databricks SDP pipelines by integrating ingestion patterns into the pipeline configuration. This supports automated data loading within the bronze layer of the medallion architecture.