databricks-spark-declarative-pipelines

Name: databricks-spark-declarative-pipelines
Availability: InStock
Author: zack2626-ok

Automate creation and management of Databricks Spark Declarative Pipelines.

Updated Jun 11, 2026

Search Tags:#serverless #data pipelines #spark #databricks #delta live tables

Ready to run

npx skills add https://github.com/Zack2626-ok/DATN_Website-Dat-Ban-Va-Quan-Ly-Nha-Hang --skill databricks-spark-declarative-pipelines-zack2626-ok

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: databricks-spark-declarative-pipelines
Source: https://github.com/Zack2626-ok/DATN_Website-Dat-Ban-Va-Quan-Ly-Nha-Hang/tree/main/.windsurf/skills/databricks-spark-declarative-pipelines
Command: npx skills add https://github.com/Zack2626-ok/DATN_Website-Dat-Ban-Va-Quan-Ly-Nha-Hang --skill databricks-spark-declarative-pipelines-zack2626-ok

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill simplifies the process of creating, configuring, and updating Databricks Lakeflow Spark Declarative Pipelines, enabling efficient data ingestion, transformation, and aggregation in a serverless environment.

Core Features & Use Cases

Streamlined Pipeline Creation: Automates the creation of Databricks Spark Declarative Pipelines (SDP), reducing manual setup time.
Data Ingestion: Supports various data ingestion patterns, including streaming tables, materialized views, CDC, SCD Type 2, and Auto Loader.
Use Case: Ideal for building data pipelines, working with Delta Live Tables, processing streaming data, implementing change data capture, and designing bronze/silver/gold medallion architectures.

Quick Start

Use the databricks-spark-declarative-pipelines skill to create a new SDP for ingesting streaming data from a Kafka source into a streaming table.

Frequently Asked Questions about databricks-spark-declarative-pipelines

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I build serverless data pipelines with Spark Declarative Pipelines on Databricks?▼

You can build serverless data pipelines by using declarative SQL-like syntax to automate the creation and management of Spark Declarative Pipelines on Databricks. This approach handles data ingestion, transformation, and incremental processing without manual infrastructure setup.

What is the best way to implement a medallion architecture with Delta Live Tables?▼

Implementing a medallion architecture with Delta Live Tables is achieved by defining data transformations using declarative syntax. This automates the flow of data through bronze, silver, and gold layers for structured ingestion and aggregation in a serverless environment.

Does Databricks Lakeflow support streaming ingestion and CDC for Delta Live Tables?▼

Yes, Databricks Lakeflow supports streaming ingestion and change data capture for Delta Live Tables. You can configure these pipelines to process streaming data, materialized views, and Auto Loader patterns using declarative syntax on Spark 3.x.

How do I automate data transformation and incremental processing in a serverless Spark environment?▼

Automating data transformation and incremental processing in a serverless Spark environment requires defining the logic through declarative pipelines. The system automatically manages the execution and orchestration of streaming and batch transformations on Databricks Lakeflow.

Do I need a specific Spark version to parse declarative SQL-like syntax for data pipelines?▼

Yes, you need Spark 3.x along with Databricks Lakeflow to properly parse and execute the declarative SQL-like syntax used for creating and managing these serverless data pipelines.