databricks-pipelines

To build a Databricks Lakeflow declarative pipeline, you author streaming tables, batch datasets, and materialized views using Python or SQL, then scaffold the project structure and deploy via the CLI. This skill guides you through authoring and deploying these pipelines.

The best way to implement a medallion architecture with Auto Loader and CDC in Databricks is to use Lakeflow declarative pipelines. This skill provides scaffolded patterns and best practices to guide data from bronze to silver and gold reliably.

Yes, you can use Auto Loader and materialized views in the same Databricks Lakeflow pipeline. The skill covers supported dataset types including streams, batch, materialized views, and Auto CDC, allowing you to combine them effectively.

Yes, you need the Databricks CLI to manage your Lakeflow Spark declarative pipelines. This skill includes step-by-step scaffolding, CLI usage, and lifecycle management best practices to reduce risk and accelerate delivery of your data products.

Auto CDC in Lakeflow pipelines automatically processes change data capture events to keep your tables updated. This skill guides you through implementing CDC pipelines and applying end-to-end data quality patterns from bronze to gold.

Declarative pipelines for batch and streaming workflows on Databricks require careful project structure and lifecycle management to ensure safe, scalable implementation. This skill provides recommended patterns and prerequisites to navigate these constraints.