skills.rest
datasciencemonkey
datasciencemonkeyCOMMUNITY
Explore all skills by this author →
GitHub

databricks-spark-declarative-pipelines

Automate creation, configuration, and lifecycle management of Databricks SDP pipelines.

27|9|Updated Feb 3, 2026
Search Tags:#serverless#pipeline-management#cdc#databricks#sdp#ldp#autoloader
One-click install
npx skills add https://github.com/datasciencemonkey/coding-agents-databricks-apps --skill databricks-spark-declarative-pipelines-datasciencemonkey
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: databricks-spark-declarative-pipelines
Source: https://github.com/datasciencemonkey/coding-agents-databricks-apps/tree/main/.claude/skills/databricks-spark-declarative-pipelines
Command: npx skills add https://github.com/datasciencemonkey/coding-agents-databricks-apps --skill databricks-spark-declarative-pipelines-datasciencemonkey

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This skill enables teams to automate the end-to-end setup, configuration, and lifecycle management of Databricks Lakeflow Spark Declarative Pipelines (SDP/LDP) using serverless compute. It addresses the challenge of building, updating, and maintaining complex data pipelines across multi-environment deployments.

Core Features & Use Cases

  • Create, configure, and update SDP pipelines with serverless compute in Unity Catalog-enabled Databricks workspaces.
  • Support streaming tables, materialized views, Change Data Capture (AUTO CDC and SCD Type 1/2), and Auto Loader ingestion patterns, enabling robust medallion architectures.
  • Use cases include building new end-to-end pipelines, migrating from DLT, implementing CDC-driven data warehouses, and accelerating data engineering workflows with repeatable bundle-based deployments.

Quick Start

Create a production-ready SDP project using Asset Bundles, then customize transformations and deploy to your multi-environment target.

Frequently Asked Questions about databricks-spark-declarative-pipelines

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I automate Databricks pipeline provisioning for serverless compute?▼

You can automate Databricks pipeline provisioning by using Asset Bundles to configure and deploy serverless SDP pipelines across multiple environments, yielding repeatable end-to-end lifecycle management.

Can I implement Change Data Capture pipelines using Databricks SDP?▼

Yes, you can implement Change Data Capture using Databricks SDP by configuring AUTO CDC and SCD Type 1/2 patterns within your streaming tables to maintain accurate historical data records.

Does this approach support building medallion architectures with Auto Loader?▼

Yes, building medallion architectures is fully supported by combining Auto Loader for bronze ingestion with streaming tables and materialized views for silver and gold transformations.

What is the best way to migrate from DLT to Spark Declarative Pipelines?▼

The best way to migrate from DLT to Spark Declarative Pipelines is to automate the creation and configuration of serverless SDP pipelines, adapting existing transformations to the new framework.

How do I manage multi-environment deployment for Databricks SDP pipelines?▼

You manage multi-environment deployment for Databricks SDP pipelines by utilizing Asset Bundles, which package configurations to provision and update pipeline infrastructure across distinct workspaces.

Related Skills

Explore More Data & Analytics Skills→
databricks-solutionsdatabricks-solutions
1.8k

spark-declarative-pipelines

Automate creation, configuration, and updating of Databricks Lakeflow Spark Declarative Pipelines.

Official
Advanced
teegin-gteegin-g

databricks-spark-declarative-pipelines

Configure Databricks SDP assets with Asset Bundles and serverless compute.

Community
Advanced
juanlamadrid20juanlamadrid20

databricks-spark-declarative-pipelines

Automate creation and configuration of Databricks Lakeflow Spark Declarative Pipelines.

Community
Advanced
slysikslysik

databricks-spark-declarative-pipelines

Automate creation and management of Databricks Lakeflow Spark Declarative Pipelines.

Community
Advanced