spark-declarative-pipelines

You can build a Databricks data pipeline by initializing a Spark Declarative Pipeline project with Databricks Asset Bundles. This approach supports Auto Loader for cloud files and streaming sources like Kafka or Event Hubs for continuous data ingestion.

The best way to implement CDC and SCD Type 2 in Spark is using declarative pipelines. They natively support SCD Type 1 and Type 2 patterns, enabling efficient tracking of data history and deduplication across your bronze, silver, and gold layers.

Yes, Databricks declarative pipelines fully support both SQL and Python APIs. You can use them together within the same pipeline to define complex transformation logic across your data layers.

Yes, Spark Declarative Pipelines use serverless compute and leverage Liquid Clustering. This combination optimizes data layout and significantly improves query performance for your pipeline workloads.

You set up a multi-environment Databricks pipeline by initializing a new project using the databricks pipelines init command. This creates a structured, production-ready configuration managed via Databricks Asset Bundles.