skills.rest
databricks-solutions
databricks-solutionsOFFICIAL
Explore all skills by this author →
GitHub

spark-declarative-pipelines

Automate creation, configuration, and updating of Databricks Lakeflow Spark Declarative Pipelines.

1.8k|404|Updated Dec 17, 2025
Search Tags:#serverless#medallion#cdc#databricks#scd type 2#auto loader#spark declarative pipelines
One-click install
npx skills add https://github.com/databricks-solutions/ai-dev-kit --skill spark-declarative-pipelines
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: spark-declarative-pipelines
Source: https://github.com/databricks-solutions/ai-dev-kit/tree/main/databricks-skills/spark-declarative-pipelines
Command: npx skills add https://github.com/databricks-solutions/ai-dev-kit --skill spark-declarative-pipelines

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This skill automates the creation, configuration, and updating of Databricks Lakeflow Spark Declarative Pipelines (SDP) using serverless compute. It enables teams to quickly stand up robust, scalable data pipelines with streaming sources, CDC, and modern medallion architectures.

Core Features & Use Cases

  • Automated pipeline bootstrap: Create, configure, and update SDP/LDP workloads with serverless compute.
  • Streaming and CDC support: Ingest streaming data, apply AUTO CDC for SCD Type 1/2, and manage materialized views.
  • End-to-end medallion workflows: Support bronze/silver/gold patterns using Auto Loader and common ingestion patterns.
  • Use Case: Rapidly spin up pipelines for real-time analytics dashboards and data platforms in Databricks.

Quick Start

Invoke the init flow to generate SDP project scaffolding and customize with transformation files.

Frequently Asked Questions about spark-declarative-pipelines

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I automate Databricks pipeline creation for medallion architectures?▼

You can automate the creation, configuration, and updating of Databricks Lakeflow Spark Declarative Pipelines to build end-to-end medallion bronze, silver, and gold data workflows using serverless compute.

How do I handle streaming ingestion and CDC in Databricks Lakeflow pipelines?▼

Databricks Lakeflow pipelines handle streaming ingestion and CDC by applying Auto Loader for data ingestion and supporting SCD Type 1 and Type 2 to manage materialized views and changing data.

Does this approach require Unity Catalog and serverless compute?▼

Yes, automating Lakeflow Spark Declarative Pipelines assumes a Unity Catalog-backed serverless environment to rapidly spin up scalable pipelines for real-time analytics dashboards.

What is the best way to bootstrap a Spark Declarative Pipeline project?▼

The best way to bootstrap a Spark Declarative Pipeline project is to invoke the initialization flow to generate project scaffolding, which you then customize with your specific transformation files.

Can I update existing DLT pipelines using Spark Declarative Pipelines?▼

Yes, you can update existing DLT, LDP, and Lakeflow pipelines by configuring and updating them through automated Spark Declarative Pipelines on serverless compute.

Related Skills

Explore More Data & Analytics Skills→
datasciencemonkeydatasciencemonkey
27

spark-declarative-pipelines

Automate creation and configuration of Databricks Lakeflow Spark Declarative Pipelines.

Community
Advanced
slysikslysik

databricks-spark-declarative-pipelines

Automate creation and management of Databricks Lakeflow Spark Declarative Pipelines.

Community
Advanced
teegin-gteegin-g

databricks-spark-declarative-pipelines

Configure Databricks SDP assets with Asset Bundles and serverless compute.

Community
Advanced
alessandro9110alessandro9110
4

databricks-spark-declarative-pipelines

Automate Databricks Spark Declarative Pipelines with serverless compute and Unity Catalog.

Community
Advanced