skills.rest
andregit2026
andregit2026COMMUNITY
Explore all skills by this author →
GitHub

databricks-spark-declarative-pipelines

Build and update Databricks Lakeflow Spark Declarative Pipelines on serverless compute.

Updated Aug 23, 2026
Search Tags:#serverless#pipelines#cdc#databricks#sdp#ldp#lakeflow
One-click install
npx skills add https://github.com/andregit2026/Databricks_DQ_Business --skill databricks-spark-declarative-pipelines-andregit2026
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: databricks-spark-declarative-pipelines
Source: https://github.com/andregit2026/Databricks_DQ_Business/tree/main/.claude/skills/databricks-general-skill-spark-declarative-pipelines
Command: npx skills add https://github.com/andregit2026/Databricks_DQ_Business --skill databricks-spark-declarative-pipelines-andregit2026

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Enables teams to rapidly build, configure, and update Databricks Lakeflow Spark Declarative Pipelines (SDP/LDP) using serverless compute, simplifying the deployment of scalable data pipelines and reducing setup toil.

Core Features & Use Cases

  • Create, configure, and update Lakeflow SDP/LDP pipelines to run on serverless compute.
  • Handle streaming tables, materialized views, CDC, SCD Type 2, and Auto Loader ingestion patterns across medallion Bronze/Silver/Gold architectures.
  • Use cases include building new data pipelines and evolving existing ones within Delta Live Tables integrations and Databricks Lakehouse environments.

Quick Start

Initialize or update a Lakeflow SDP/LDP pipeline on Databricks using serverless compute, and I will generate the project structure and configuration you need.

Frequently Asked Questions about databricks-spark-declarative-pipelines

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I build serverless Databricks Spark Declarative Pipelines for a medallion architecture?▼

Build serverless Databricks Spark Declarative Pipelines by generating project structures for streaming tables and materialized views across Bronze, Silver, and Gold layers. The skill provides configuration for serverless compute to streamline scalable data pipeline deployment.

Can I implement CDC and SCD Type 2 patterns in Databricks Lakeflow pipelines?▼

Yes, Databricks Lakeflow pipelines support CDC and SCD Type 2 patterns natively. The skill generates the necessary configuration to handle change data capture and slowly changing dimensions within your medallion architecture data workflows.

Does this approach support both Python and SQL for Auto Loader ingestion in Delta Live Tables?▼

Auto Loader ingestion supports deployment using either Python or SQL within Delta Live Tables integrations. This allows teams to build and update Lakeflow pipelines using their preferred language for serverless compute workloads.

What is the best way to configure multi-environment Asset Bundles for Databricks Lakeflow?▼

The best way to configure multi-environment Asset Bundles for Databricks Lakeflow is through generated best-practice configurations. The skill outputs the required project structure to manage deployments across different serverless environments using Unity Catalog.

Why use serverless compute for Spark Declarative Pipelines instead of standard clusters?▼

Serverless compute for Spark Declarative Pipelines reduces setup toil and simplifies the deployment of scalable data pipelines. It allows teams to rapidly build and update Lakeflow workloads without managing underlying infrastructure.

Do I need Unity Catalog to manage materialized views in Databricks Lakeflow SDP?▼

Unity Catalog provides best-practice governance for materialized views and streaming tables in Databricks Lakeflow SDP. The skill generates configurations aligned with Unity Catalog to ensure secure data management within serverless pipelines.

Related Skills

Explore More Data & Analytics Skills→
mkgs-databricks-demosmkgs-databricks-demos

databricks-ml-training

Train, register, and serve ML models on Databricks with MLflow and Unity Catalog.

Official
Advanced
teegin-gteegin-g

databricks-spark-declarative-pipelines

Configure Databricks SDP assets with Asset Bundles and serverless compute.

Community
Advanced
databricksdatabricks
251

databricks-pipelines

Develop batch and streaming data pipelines on Databricks with Lakeflow Spark Declarative Pipelines.

Official
Advanced
ThomazRossitoThomazRossito
31

databricks-spark-declarative-pipelines

Create and configure Databricks Lakeflow Spark Declarative Pipelines with serverless compute.

Community
Advanced