skills.rest
slysik
slysikCOMMUNITY
Explore all skills by this author →
GitHub

databricks-spark-declarative-pipelines

Automate creation and management of Databricks Lakeflow Spark Declarative Pipelines.

Updated Mar 10, 2026
Search Tags:#streaming#data engineering#etl#spark#lakehouse#databricks#declarative pipelines
One-click install
npx skills add https://github.com/slysik/databricks-claude-coding --skill databricks-spark-declarative-pipelines-slysik
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: databricks-spark-declarative-pipelines
Source: https://github.com/slysik/databricks-claude-coding/tree/main/.claude/skills/databricks-spark-declarative-pipelines
Command: npx skills add https://github.com/slysik/databricks-claude-coding --skill databricks-spark-declarative-pipelines-slysik

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill automates the creation, configuration, and management of Databricks Lakeflow Spark Declarative Pipelines (SDP), enabling efficient data ingestion, transformation, and processing for streaming and batch workloads.

Core Features & Use Cases

  • Pipeline Creation & Management: Initializes new projects using Databricks Asset Bundles or manages existing pipelines via CLI tools.
  • Data Ingestion: Supports Auto Loader for cloud files and streaming sources like Kafka and Event Hubs.
  • Data Transformation: Implements streaming tables, materialized views, CDC, and SCD Type 2 patterns.
  • Performance Optimization: Leverages Liquid Clustering and advanced configuration for efficient data processing.
  • Use Case: Set up a new data pipeline to ingest streaming order data from Kafka, clean and transform it into a silver layer, and aggregate daily sales metrics in a gold layer, all managed via a production-ready Asset Bundle.

Quick Start

Use the databricks pipelines init command to scaffold a new Spark Declarative Pipeline project.

Frequently Asked Questions about databricks-spark-declarative-pipelines

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I build a Databricks streaming pipeline with Auto Loader and Kafka?▼

You can build a Databricks streaming pipeline using Spark Declarative Pipelines to ingest data from Auto Loader and Kafka, then transform it through bronze, silver, and gold layers using serverless compute.

What is the best way to implement change data capture in a Databricks lakehouse?▼

Implementing change data capture in a Databricks lakehouse is automated using Spark Declarative Pipelines, which natively supports CDC and SCD Type 2 patterns to process streaming and batch data efficiently.

How do I set up a medallion architecture with Delta Live Tables?▼

Setting up a medallion architecture with Delta Live Tables involves using Spark Declarative Pipelines to define streaming tables and materialized views for bronze, silver, and gold data layers on serverless compute.

Can I manage Databricks pipeline configurations using Databricks Asset Bundles?▼

Yes, you can manage Databricks pipeline configurations using Databricks Asset Bundles to scaffold projects, initialize Spark Declarative Pipelines, and handle deployment via CLI tools.

Does Spark Declarative Pipelines support SCD Type 2 and materialized views?▼

Yes, Spark Declarative Pipelines supports SCD Type 2 and materialized views, enabling efficient data transformations and change data capture within Databricks lakehouse architectures.

How do I optimize streaming table performance with Liquid Clustering in Databricks?▼

You optimize streaming table performance in Databricks by leveraging Liquid Clustering within Spark Declarative Pipelines, applying advanced configurations to improve data processing efficiency.

Related Skills

Explore More Software Engineering Skills→
BlackkadderBlackkadder

databricks-spark-declarative-pipelines

Create and configure Databricks Lakeflow Spark Declarative Pipelines with serverless compute.

Community
Advanced
patrikboroschpatrikborosch
14

e2e-medallion-architecture

Automate Bronze/Silver/Gold medallion lakehouse pipelines in Microsoft Fabric.

Community
Advanced
mirakuimirakui

databricks-spark-declarative-pipelines

Automate Databricks SDP/LDP pipeline configuration and lifecycle across environments.

Community
Advanced
luokai25luokai25

data-engineering

Design and implement scalable data pipelines with Apache Spark and dbt workflows.

Community
Advanced