skills.rest
datasciencemonkey
datasciencemonkeyCOMMUNITY
Explore all skills by this author →
GitHub

spark-declarative-pipelines

Automate creation and configuration of Databricks Lakeflow Spark Declarative Pipelines.

27|9|Updated Feb 3, 2026
Search Tags:#serverless#pipeline#cdc#databricks#sdp#scd-type-2#medallion-architecture
One-click install
npx skills add https://github.com/datasciencemonkey/claude-code-cli-bricks --skill spark-declarative-pipelines-datasciencemonkey
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: spark-declarative-pipelines
Source: https://github.com/datasciencemonkey/claude-code-cli-bricks/tree/main/.claude/skills/spark-declarative-pipelines
Command: npx skills add https://github.com/datasciencemonkey/claude-code-cli-bricks --skill spark-declarative-pipelines-datasciencemonkey

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

The Spark Declarative Pipelines skill helps teams quickly set up, configure, and maintain SDP/LDP pipelines on Databricks, reducing boilerplate and enabling serverless scalability.

Core Features & Use Cases

  • Ingest and process streaming data with Auto Loader, read_files and other ingestion patterns
  • Build bronze/silver/gold medallion architectures with streaming, CDC, and SCD Type 2
  • Deploy, configure, and update pipelines using SDP APIs (dp) in a serverless environment with Unity Catalog
  • Real-world use: design a data pipeline that ingests streaming events, transforms them with Spark SQL/Python, and writes to durable layers for analytics

Quick Start

  • Initialize a new SDP bundle using databricks pipelines init
  • Add transformations under src/transformations/ and deploy with databricks bundle deploy
  • Run the pipeline and monitor via the UI or API

Frequently Asked Questions about spark-declarative-pipelines

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I build a medallion architecture with streaming data in Databricks?▼

Build a Databricks medallion architecture by configuring Spark Declarative Pipelines to ingest streaming events with Auto Loader and process them through bronze, silver, and gold layers serverlessly.

How do I handle CDC and SCD Type 2 changes in a serverless Databricks pipeline?▼

Handle CDC and SCD Type 2 changes by applying Spark Declarative Pipelines transformations within your serverless Databricks environment to maintain historical records and capture streaming data changes.

What's the best way to automate Databricks pipeline deployment using serverless compute?▼

Automate Databricks pipeline deployment by initializing an SDP bundle, adding transformations under src/transformations/, and deploying with the databricks bundle deploy command for serverless execution.

Do I need Unity Catalog to run serverless Spark Declarative Pipelines?▼

Yes, you need Unity Catalog along with serverless compute and access to cloud storage sources via read_files to run modern Spark Declarative Pipelines using pyspark.pipelines.

Can I use Auto Loader with read_files for cloud storage ingestion in Spark Declarative Pipelines?▼

Yes, you can use Auto Loader and read_files to ingest and process streaming data from cloud storage sources directly within your serverless Spark Declarative Pipelines.

Related Skills

Explore More Data & Analytics Skills→
JingyiWngJingyiWng

databricks-spark-declarative-pipelines

Create and manage Databricks Lakeflow Spark Declarative Pipelines for streaming and batch workflows.

Community
Advanced
prashsubprashsub
1

project-planning

Generate multi-phase Databricks project plans with YAML manifests.

Community
Advanced
databrickslabsdatabrickslabs
38

databricks-spark-declarative-pipelines

Automate creation, configuration, and optimization of Databricks Lakeflow Spark Declarative Pipelines.

Official
Advanced
m19cm19c

databricks-spark-declarative-pipelines

Automates creation, configuration, and management of Databricks Lakeflow Spark Declarative Pipelines.

Community
Advanced