databricks-spark-declarative-pipelines

Automate creation and configuration of Databricks Lakeflow Spark Declarative Pipelines.

Updated Mar 12, 2026

Search Tags:#streaming #etl #data pipelines #spark #databricks #sdp #lakeflow

Ready to run

npx skills add https://github.com/juanlamadrid20/coda --skill databricks-spark-declarative-pipelines-juanlamadrid20

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: databricks-spark-declarative-pipelines
Source: https://github.com/juanlamadrid20/coda/tree/main/.claude/skills/databricks-spark-declarative-pipelines
Command: npx skills add https://github.com/juanlamadrid20/coda --skill databricks-spark-declarative-pipelines-juanlamadrid20

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill automates the creation, configuration, and updating of Databricks Lakeflow Spark Declarative Pipelines (SDP/LDP), streamlining data engineering workflows.

Core Features & Use Cases

Pipeline Creation & Management: Automates the setup and lifecycle management of SDP pipelines using Databricks Asset Bundles or MCP tools.
Data Ingestion & Streaming: Supports Auto Loader for file ingestion and various streaming sources like Kafka and Event Hubs.
Data Transformation & Quality: Facilitates building streaming tables, materialized views, implementing CDC, SCD Type 2, and data quality checks.
Use Case: A data engineer needs to build a new streaming pipeline to ingest real-time sensor data, process it, and store aggregated metrics. This Skill can initialize the project, define the pipeline structure, and deploy it to Databricks.

Quick Start

Use the databricks pipelines init command to create a new Spark Declarative Pipeline project.

Frequently Asked Questions about databricks-spark-declarative-pipelines

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I build Databricks streaming pipelines with Auto Loader and Kafka?▼

This Skill automates the creation of Databricks Lakeflow Spark Declarative Pipelines using serverless compute. It manages streaming tables, materialized views, and Auto Loader ingestion patterns through Databricks Asset Bundles or MCP tools.

Can I implement SCD Type 2 and CDC in Databricks Lakeflow pipelines?▼

Yes, you can implement SCD Type 2 and CDC in Databricks Lakeflow pipelines. This Skill facilitates these data transformations alongside streaming tables and materialized views, automating the configuration of declarative pipelines using serverless compute.

What is the best way to automate Databricks data pipeline creation?▼

The best way to automate Databricks data pipeline creation is using Databricks Asset Bundles. This Skill initializes Spark Declarative Pipeline projects and automates their lifecycle management, configuration, and updates via MCP tools.

Does this support Databricks Asset Bundles for initializing pipeline projects?▼

Yes, it supports Databricks Asset Bundles for project initialization. You can use the databricks pipelines init command to create a new Spark Declarative Pipeline project structure, define the pipeline, and deploy it to Databricks.

How do I manage streaming tables and materialized views in Databricks SDP?▼

You manage streaming tables and materialized views in Databricks SDP by defining them within your declarative pipeline configuration. This Skill automates the setup and updating of these structures on serverless compute.