databricks-spark-declarative-pipelines

Create and manage Databricks Lakeflow Spark Declarative Pipelines for streaming and batch workflows.

Updated Mar 25, 2026

Search Tags:#spark #cdc #databricks #sdp #scd-type-2 #lakeflow #auto-loader

Ready to run

npx skills add https://github.com/JingyiWng/databricks_ai_dev_kit_price_watcher --skill databricks-spark-declarative-pipelines-jingyiwng

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: databricks-spark-declarative-pipelines
Source: https://github.com/JingyiWng/databricks_ai_dev_kit_price_watcher/tree/main/.claude/skills/databricks-spark-declarative-pipelines
Command: npx skills add https://github.com/JingyiWng/databricks_ai_dev_kit_price_watcher --skill databricks-spark-declarative-pipelines-jingyiwng

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Helps teams design, create, configure, and operate Databricks Lakeflow Spark Declarative Pipelines (SDP/LDP) with consistent serverless defaults and Asset Bundle project structure, reducing misconfiguration, fragile migrations from DLT, and ad hoc pipeline maintenance.

Core Features & Use Cases

Project initialization & deployment: Use databricks pipelines init and Asset Bundles to scaffold multi-environment projects and deploy pipelines via databricks bundle deploy.
Ingestion & streaming patterns: Guidance for Auto Loader (read_files/cloudFiles), Kafka/Event Hub/Kinesis read_stream patterns, schema hints, and rescue/quarantine handling.
CDC, SCD and deduplication: Recipes for AUTO CDC, SCD Type 1/2, sequence_by best practices, and querying historical SCD tables.
Python & SQL guidance: Modern pyspark.pipelines (dp) API recommendations, migration from legacy dlt, and rules for choosing SQL vs Python.
Performance and advanced config: Liquid Clustering, materialized view refresh strategies, serverless vs cluster settings, and extra_settings examples.

Quick Start

Initialize a new serverless Databricks Asset Bundle pipeline project with databricks pipelines init and choose the target catalog, schema, and language.

Frequently Asked Questions about databricks-spark-declarative-pipelines

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I create serverless Databricks Lakeflow pipelines using Asset Bundles?▼

To create serverless Databricks Lakeflow pipelines, initialize an Asset Bundle project using databricks pipelines init, configure target Unity Catalog schema and language, then deploy via databricks bundle deploy.

What is the best way to handle SCD Type 2 and CDC flows in Databricks SDP?▼

Handling SCD Type 2 and CDC flows in Databricks SDP requires using AUTO CDC recipes, configuring sequence_by best practices for record ordering, and querying historical SCD tables to track changes accurately.

How do I migrate from legacy DLT to the modern pyspark.pipelines API?▼

Migrating from legacy DLT to modern pyspark.pipelines API requires updating codebases to use the dp module, aligning streaming and batch workflows with serverless defaults, and applying Asset Bundle project structures.

Does Databricks Lakeflow support Auto Loader ingestion from Kafka and Event Hub?▼

Yes, Databricks Lakeflow supports Auto Loader ingestion from Kafka, Event Hub, and Kinesis by utilizing read_stream patterns, applying schema hints, and configuring rescue or quarantine handling for malformed records.

Can I build a bronze/silver/gold medallion architecture with Spark Declarative Pipelines?▼

Yes, you can build a bronze/silver/gold medallion architecture with Spark Declarative Pipelines by defining materialized views, configuring refresh strategies, and applying Liquid Clustering for optimized performance.

Why use serverless defaults instead of cluster settings for Databricks pipelines?▼

Using serverless defaults instead of cluster settings for Databricks pipelines reduces misconfiguration and ad hoc maintenance, providing consistent environments for Auto Loader ingestion and Unity Catalog integration.