databricks-spark-declarative-pipelines

Automate Databricks Spark Declarative Pipelines with serverless compute.

1|Updated Dec 15, 2025

Search Tags:#serverless #lakehouse #cdc #databricks #scd2 #sdpldp

Ready to run

npx skills add https://github.com/lucaslessachaves/default --skill databricks-spark-declarative-pipelines-lucaslessachaves

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: databricks-spark-declarative-pipelines
Source: https://github.com/lucaslessachaves/default/tree/main/.claude/skills/databricks-spark-declarative-pipelines
Command: npx skills add https://github.com/lucaslessachaves/default --skill databricks-spark-declarative-pipelines-lucaslessachaves

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Automates the setup and ongoing governance of Databricks Spark Declarative Pipelines (SDP/LDP) using serverless compute, reducing complexity in building and maintaining data pipelines across Bronze/Silver/Gold.

Core Features & Use Cases

Automates creation, configuration, and updates of SDP objects with serverless compute.
Supports streaming ingestion, CDC, SCD Type 2, materialized views, Auto Loader ingestion patterns, and Delta Live Tables integration.
Use cases include building end-to-end medallion architectures, implementing CDC/SCD patterns, and accelerating data pipeline delivery in modern lakehouse environments.

Quick Start

Describe how to set up a serverless SDP pipeline in Databricks, including bronze, silver, and gold layers with CDC and Auto Loader.

Frequently Asked Questions about databricks-spark-declarative-pipelines

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I build a Databricks Delta Live Tables pipeline with serverless compute?▼

To build a serverless Delta Live Tables pipeline, you configure SDP objects across Bronze, Silver, and Gold layers. This approach automates setup and governance while enforcing syntax rules for streaming ingestion and materialized views.

What is the best way to implement SCD Type 2 and CDC in a Databricks lakehouse?▼

Implementing SCD Type 2 and CDC in a Databricks lakehouse is automated through Spark Declarative Pipelines. This enforces SDP syntax rules and provides Python and SQL references for rapid materialized view and streaming ingestion deployment.

Can I use Auto Loader for streaming ingestion in Spark Declarative Pipelines?▼

Yes, Auto Loader is supported for streaming ingestion patterns in Spark Declarative Pipelines. It integrates with serverless compute to automate data pipeline delivery across medallion architecture layers using Python or SQL.

Does serverless Databricks support medallion architecture for Delta Live Tables?▼

Serverless Databricks fully supports medallion architecture for Delta Live Tables. It automates the creation and configuration of Bronze, Silver, and Gold layers, reducing complexity in maintaining modern lakehouse environments.

How do I configure Delta Live Tables across Bronze, Silver, and Gold layers?▼

Configuring Delta Live Tables across layers involves using SDP syntax rules for serverless compute. This provides workflow guidance for building end-to-end medallion architectures with CDC, Auto Loader, and materialized views.

Why use serverless compute for Spark Declarative Pipelines instead of standard clusters?▼

Using serverless compute for Spark Declarative Pipelines reduces complexity in building and maintaining data pipelines. It automates ongoing governance and updates for Delta Live Tables across streaming, CDC, and SCD Type 2 workloads.