skills.rest
mkgs-databricks-demos
mkgs-databricks-demosOFFICIAL
Explore all skills by this author →
GitHub

databricks-spark-declarative-pipelines

Create, configure, and update Databricks Lakeflow Spark Declarative Pipelines using serverless compute.

1|1|Updated Oct 1, 2025
Search Tags:#cdc#databricks#sdp#ldp#autoloader#spark-declarative-pipelines#scd-2
One-click install
npx skills add https://github.com/mkgs-databricks-demos/synthea-on-fhir --skill databricks-spark-declarative-pipelines-mkgs-databricks-demos
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: databricks-spark-declarative-pipelines
Source: https://github.com/mkgs-databricks-demos/synthea-on-fhir/tree/main/.cursor/skills/databricks-spark-declarative-pipelines
Command: npx skills add https://github.com/mkgs-databricks-demos/synthea-on-fhir --skill databricks-spark-declarative-pipelines-mkgs-databricks-demos

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Databricks SDP pipelines are complex to set up and maintain across environments; this skill streamlines creation, configuration, and ongoing updates using serverless compute, CDC, SCD Type 2, Auto Loader patterns, and medallion architecture guidance.

Core Features & Use Cases

  • End-to-end support for bronze/silver/gold data pipelines, streaming tables, materialized views, and auto loader ingestion patterns.
  • Includes AUTO CDC for SCD Type 1/2, data deduplication, and late-arriving data handling to improve data quality and accuracy.
  • Facilitates migration from Delta Live Tables (DLT) to Spark Declarative Pipelines (SDP) and supports multi-environment deployments via Asset Bundles.

Quick Start

Initialize a new SDP project with databricks pipelines init, add your SQL or Python transformations under transformations/, and deploy with databricks bundle deploy.

Frequently Asked Questions about databricks-spark-declarative-pipelines

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I create Databricks Spark Declarative Pipelines for bronze/silver/gold medallion architectures?▼

To create Databricks Spark Declarative Pipelines for a bronze/silver/gold medallion architecture, initialize a new project with databricks pipelines init, add SQL or Python transformations under transformations/, and deploy using databricks bundle deploy with serverless compute.

What is the best way to migrate Delta Live Tables workflows to Spark Declarative Pipelines?▼

Migrating Delta Live Tables to Spark Declarative Pipelines is supported directly. You can transition existing streaming ingestion and transformations to the serverless SDP framework using either the Python dp API or SQL workflows integrated with Unity Catalog.

How do I implement Auto Loader and CDC SCD Type 2 workflows in Databricks SDP?▼

Implementing Auto Loader and CDC SCD Type 2 workflows in Databricks SDP involves using AUTO CDC for SCD Type 1/2 alongside built-in data deduplication and late-arriving data handling to ensure streaming data quality and accuracy.

Does Databricks serverless compute support both Python and SQL for Spark Declarative Pipelines?▼

Yes, Databricks serverless compute fully supports both Python SDP using the dp API and SQL workflows for Spark Declarative Pipelines. Language selection rules are applied to configure your streaming tables and materialized views appropriately.

Can I deploy Databricks SDP pipelines across multiple environments using Asset Bundles?▼

Yes, you can deploy Databricks SDP pipelines across multiple environments using Asset Bundles. This approach streamlines the creation, configuration, and ongoing updates of serverless compute pipelines for new projects.

Why use Unity Catalog integration when configuring serverless Databricks pipelines?▼

Unity Catalog integration is used when configuring serverless Databricks pipelines to centrally manage data governance and access. It ensures secure operations across bronze/silver/gold layers during streaming ingestion and CDC workflows.

Related Skills

Explore More Data & Analytics Skills→
databricks-solutionsdatabricks-solutions
1.8k

spark-declarative-pipelines

Automate creation, configuration, and updating of Databricks Lakeflow Spark Declarative Pipelines.

Official
Advanced
LaurentPRAT-DBLaurentPRAT-DB

spark-declarative-pipelines

Automate Databricks Lakeflow Spark Declarative Pipelines using SQL or Python.

Community
Advanced
tomztomz

azure-databricks

Provision Azure Databricks workspaces with Unity Catalog governance via az CLI.

Community
Advanced
slysikslysik

databricks-spark-declarative-pipelines

Automate creation and management of Databricks Lakeflow Spark Declarative Pipelines.

Community
Advanced