databricks-spark-declarative-pipelines

You can build a Databricks streaming ingestion pipeline by automating Auto Loader configuration for cloud storage, Kafka, and Event Hub, applying schema evolution, rescue data handling, and authentication best practices for serverless compute.

The best way to implement medallion architecture is automating bronze, silver, and gold layer configurations with deduplication, data quality patterns, and performance tuning using Liquid Clustering on serverless compute.

To handle change data capture in Spark Declarative Pipelines, this skill automates AUTO CDC configuration, slowly changing dimensions, temporal queries, and point-in-time analysis for streaming and batch workloads.

Yes, you need Unity Catalog and serverless compute to use these Spark Declarative Pipelines, as the automated configuration, Asset Bundle deployment, and performance optimization features require these Databricks environment prerequisites.

Yes, you can use both SQL and Python APIs for Databricks Lakeflow pipeline deployment, as this skill supports multi-environment Asset Bundle scaffolding and configuration across both programming interfaces.

To optimize streaming workloads in Databricks, this skill automates state management, windowed aggregations, materialized views, and join optimization to improve throughput for high-performance serverless pipelines.