skills.rest
Blackkadder
BlackkadderCOMMUNITY
Explore all skills by this author →
GitHub

databricks-spark-declarative-pipelines

Create and configure Databricks Lakeflow Spark Declarative Pipelines with serverless compute.

Updated Mar 19, 2026
Search Tags:#streaming#spark#cdc#databricks#sdp#scd-type-2#lakeflow
One-click install
npx skills add https://github.com/Blackkadder/databricks-apps-and-agents-workshop --skill databricks-spark-declarative-pipelines-blackkadder
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: databricks-spark-declarative-pipelines
Source: https://github.com/Blackkadder/databricks-apps-and-agents-workshop/tree/main/.claude/skills/databricks-spark-declarative-pipelines
Command: npx skills add https://github.com/Blackkadder/databricks-apps-and-agents-workshop --skill databricks-spark-declarative-pipelines-blackkadder

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Provides a prescriptive, production-oriented workflow to create, configure, and validate Databricks Lakeflow Spark Declarative Pipelines (SDP) so teams can reliably ingest streaming data, apply CDC/SCD patterns, and publish clean analytics tables without manual cluster management or ad-hoc scripts.

Core Features & Use Cases

  • Declarative pipeline creation: Guidance and syntax for CREATE OR REFRESH STREAMING TABLE and CREATE OR REFRESH MATERIALIZED VIEW in SQL and equivalent pyspark.pipelines (dp) patterns in Python.
  • Ingestion patterns: Auto Loader (read_files), Kafka, and file-based streaming with rescue/quarantine patterns, schema hints, and ingestion metadata.
  • CDC / SCD support: AUTO CDC flows and SCD Type 2 patterns with best practices for sequence_by, apply-as-delete ordering, and history tracking (__START_AT / __END_AT).
  • Medallion architecture & minimalism: Recommendations for bronze→silver→gold pipelines, naming conventions, clustering defaults, and when to keep pipelines minimal vs. full medallion chains.
  • Deployment & iteration workflows: Option A (standalone DAB project), Option B (add to existing bundle), and Option C (MCP rapid iteration), plus validation and post-run checks.
  • Real-world example: Ingest JSON orders with Auto Loader into a bronze streaming table, clean and deduplicate in silver, then publish a gold materialized view for daily totals with SCD Type 2 customer history.

Quick Start

Create a serverless SDP that ingests JSON files from /Volumes/my_catalog/my_schema/raw/orders into a bronze streaming table and a gold materialized view that aggregates daily totals.

Frequently Asked Questions about databricks-spark-declarative-pipelines

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I create a Databricks Lakeflow streaming pipeline for bronze to gold medallion architecture?▼

To create a Databricks Lakeflow streaming pipeline, use the CREATE OR REFRESH STREAMING TABLE syntax or pyspark.pipelines (dp) Python API to define bronze, silver, and gold layers. Serverless compute automatically handles cluster management for these declarative pipelines.

How do I implement SCD Type 2 and CDC flows in Databricks Spark Declarative Pipelines?▼

Implement SCD Type 2 and CDC flows in Spark Declarative Pipelines using AUTO CDC patterns. Configure sequence_by and apply-as-delete ordering, then track historical changes using __START_AT and __END_AT metadata fields for accurate data versioning.

What is the best way to ingest Kafka and JSON files into Databricks streaming tables?▼

The best way to ingest Kafka and JSON files into streaming tables is using Auto Loader with read_files. It supports schema hints, ingestion metadata, and rescue or quarantine patterns to handle malformed records reliably during pipeline execution.

Do I need Unity Catalog enabled to use serverless Spark Declarative Pipelines?▼

Yes, Unity Catalog must be enabled in your Databricks workspace to use serverless Spark Declarative Pipelines. The pipeline configuration databricks.yml and validation via MCP tools also expect a Unity Catalog-enabled environment for proper governance.

Can I use materialized views instead of streaming tables for daily aggregation in Lakeflow?▼

Yes, you can use materialized views for daily aggregation in Lakeflow. Use CREATE OR REFRESH MATERIALIZED VIEW SQL syntax to build gold layer aggregations from silver streaming tables, effectively computing daily totals and summary metrics.

When should I keep my Databricks declarative pipeline minimal instead of building a full medallion chain?▼

Keep your Databricks declarative pipeline minimal instead of building a full medallion chain when direct ingestion and simple transformations suffice. Reserve bronze, silver, and gold layers for complex CDC, deduplication, and SCD Type 2 history tracking requirements.

Related Skills

Explore More Data & Analytics Skills→
databricks-solutionsdatabricks-solutions
1.8k

spark-declarative-pipelines

Automate creation, configuration, and updating of Databricks Lakeflow Spark Declarative Pipelines.

Official
Advanced
ice-droid-99ice-droid-99

databricks-spark-declarative-pipelines

Build Databricks Lakeflow Spark Declarative Pipelines with SDP/LDP SQL or pyspark.pipelines code.

Community
Advanced
datasciencemonkeydatasciencemonkey
27

databricks-spark-declarative-pipelines

Automate creation, configuration, and lifecycle management of Databricks SDP pipelines.

Community
Advanced
alessandro9110alessandro9110
4

databricks-spark-declarative-pipelines

Automate Databricks Spark Declarative Pipelines with serverless compute and Unity Catalog.

Community
Advanced