skills.rest
JingyiWng
JingyiWngCOMMUNITY
Explore all skills by this author →
GitHub

databricks-spark-declarative-pipelines

Create and manage Databricks Lakeflow Spark Declarative Pipelines for streaming and batch workflows.

Updated Mar 25, 2026
Search Tags:#spark#cdc#databricks#sdp#scd-type-2#lakeflow#auto-loader
One-click install
npx skills add https://github.com/JingyiWng/databricks_ai_dev_kit_price_watcher --skill databricks-spark-declarative-pipelines-jingyiwng
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: databricks-spark-declarative-pipelines
Source: https://github.com/JingyiWng/databricks_ai_dev_kit_price_watcher/tree/main/.claude/skills/databricks-spark-declarative-pipelines
Command: npx skills add https://github.com/JingyiWng/databricks_ai_dev_kit_price_watcher --skill databricks-spark-declarative-pipelines-jingyiwng

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Helps teams design, create, configure, and operate Databricks Lakeflow Spark Declarative Pipelines (SDP/LDP) with consistent serverless defaults and Asset Bundle project structure, reducing misconfiguration, fragile migrations from DLT, and ad hoc pipeline maintenance.

Core Features & Use Cases

  • Project initialization & deployment: Use databricks pipelines init and Asset Bundles to scaffold multi-environment projects and deploy pipelines via databricks bundle deploy.
  • Ingestion & streaming patterns: Guidance for Auto Loader (read_files/cloudFiles), Kafka/Event Hub/Kinesis read_stream patterns, schema hints, and rescue/quarantine handling.
  • CDC, SCD and deduplication: Recipes for AUTO CDC, SCD Type 1/2, sequence_by best practices, and querying historical SCD tables.
  • Python & SQL guidance: Modern pyspark.pipelines (dp) API recommendations, migration from legacy dlt, and rules for choosing SQL vs Python.
  • Performance and advanced config: Liquid Clustering, materialized view refresh strategies, serverless vs cluster settings, and extra_settings examples.

Quick Start

Initialize a new serverless Databricks Asset Bundle pipeline project with databricks pipelines init and choose the target catalog, schema, and language.

Frequently Asked Questions about databricks-spark-declarative-pipelines

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I create serverless Databricks Lakeflow pipelines using Asset Bundles?▼

To create serverless Databricks Lakeflow pipelines, initialize an Asset Bundle project using databricks pipelines init, configure target Unity Catalog schema and language, then deploy via databricks bundle deploy.

What is the best way to handle SCD Type 2 and CDC flows in Databricks SDP?▼

Handling SCD Type 2 and CDC flows in Databricks SDP requires using AUTO CDC recipes, configuring sequence_by best practices for record ordering, and querying historical SCD tables to track changes accurately.

How do I migrate from legacy DLT to the modern pyspark.pipelines API?▼

Migrating from legacy DLT to modern pyspark.pipelines API requires updating codebases to use the dp module, aligning streaming and batch workflows with serverless defaults, and applying Asset Bundle project structures.

Does Databricks Lakeflow support Auto Loader ingestion from Kafka and Event Hub?▼

Yes, Databricks Lakeflow supports Auto Loader ingestion from Kafka, Event Hub, and Kinesis by utilizing read_stream patterns, applying schema hints, and configuring rescue or quarantine handling for malformed records.

Can I build a bronze/silver/gold medallion architecture with Spark Declarative Pipelines?▼

Yes, you can build a bronze/silver/gold medallion architecture with Spark Declarative Pipelines by defining materialized views, configuring refresh strategies, and applying Liquid Clustering for optimized performance.

Why use serverless defaults instead of cluster settings for Databricks pipelines?▼

Using serverless defaults instead of cluster settings for Databricks pipelines reduces misconfiguration and ad hoc maintenance, providing consistent environments for Auto Loader ingestion and Unity Catalog integration.

Related Skills

Explore More Data & Analytics Skills→
ThomazRossitoThomazRossito
31

databricks-bundles

Create and configure Databricks Declarative Automation Bundles for multi-environment deployment.

Community
Advanced
PaldomPaldom
11

databricks-spark-declarative-pipelines

Automate creation, configuration, and updates of Databricks SDP pipelines.

Community
Intermediate
leary-pokenleary-poken
1

databricks-dbsql

Document advanced Databricks SQL patterns for procedural SQL, stored procedures, and governance.

Community
Advanced
camelranchentertainmentcamelranchentertainment
1

spark-engineer

Design and optimize large-scale Apache Spark jobs for scalable data processing.

Community
Advanced