skills.rest
alessandro9110
alessandro9110COMMUNITY
Explore all skills by this author →
GitHub

databricks-spark-declarative-pipelines

Automate Databricks Spark Declarative Pipelines with serverless compute and Unity Catalog.

4|2|Updated Feb 16, 2026
Search Tags:#streaming#etl#pipelines#lakehouse#cdc#databricks
One-click install
npx skills add https://github.com/alessandro9110/Speech-To-Text-With-Databricks --skill databricks-spark-declarative-pipelines-alessandro9110
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: databricks-spark-declarative-pipelines
Source: https://github.com/alessandro9110/Speech-To-Text-With-Databricks/tree/main/.claude/skills/databricks-spark-declarative-pipelines
Command: npx skills add https://github.com/alessandro9110/Speech-To-Text-With-Databricks --skill databricks-spark-declarative-pipelines-alessandro9110

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill automates the creation, configuration, and management of Databricks Lakeflow Spark Declarative Pipelines (SDP/LDP), enabling efficient data engineering workflows.

Core Features & Use Cases

  • Pipeline Creation & Management: Set up new pipelines using Databricks Asset Bundles or manual workflows.
  • Data Ingestion & Transformation: Supports Auto Loader, streaming sources, and complex transformations with SQL or Python.
  • Change Data Capture (CDC): Implements SCD Type 1/2 for tracking data history and deduplication.
  • Performance Optimization: Leverages Liquid Clustering and advanced configurations for efficient data processing.
  • Use Case: Automate the ingestion of streaming data from Kafka, process it through bronze, silver, and gold layers, and manage the entire pipeline lifecycle using Databricks Asset Bundles for CI/CD.

Quick Start

Use the databricks-spark-declarative-pipelines skill to initialize a new Python pipeline project named 'customer_orders_pipeline' in the current directory.

Frequently Asked Questions about databricks-spark-declarative-pipelines

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I build a Databricks medallion architecture with streaming data?▼

Build a Databricks medallion architecture by using Spark Declarative Pipelines to ingest streaming data and process it sequentially through bronze, silver, and gold layers. This automates the data engineering workflow using serverless compute and Unity Catalog.

How do I implement change data capture in Databricks pipelines?▼

Implement change data capture in Databricks pipelines by configuring SCD Type 1 and Type 2 within Spark Declarative Pipelines. This manages tracking data history and handles deduplication automatically across your streaming tables and materialized views.

What is the best way to automate Databricks pipeline deployment?▼

Automate Databricks pipeline deployment by packaging Spark Declarative Pipelines into Databricks Asset Bundles. This facilitates CI/CD orchestration and manages the entire lifecycle of your Auto Loader ingestion patterns and data transformations.

Can I use Auto Loader for data ingestion in Delta Live Tables?▼

Yes, you can use Auto Loader for data ingestion in Delta Live Tables. Spark Declarative Pipelines support Auto Loader alongside streaming sources to automate file ingestion and schema inference directly into your lakehouse environment.

Does Databricks support serverless compute for declarative pipelines?▼

Yes, Databricks supports serverless compute for declarative pipelines. Spark Declarative Pipelines leverage serverless compute combined with Liquid Clustering and Unity Catalog to optimize performance and governance for your ETL workflows.

Related Skills

Explore More Software Engineering Skills→
JonathanMitchell1234JonathanMitchell1234

databricks-expert

Provides expert guidance and code examples for configuring and operating the Databricks platform.

Community
Advanced
patrikboroschpatrikborosch
14

e2e-medallion-architecture

Automate Bronze/Silver/Gold medallion lakehouse pipelines in Microsoft Fabric.

Community
Advanced
LaurentPRAT-DBLaurentPRAT-DB

databricks-ml

Automate end-to-end machine learning workflows on Databricks with MLflow and Unity Catalog.

Community
Advanced
databricksdatabricks
251

databricks-pipelines

Develop batch and streaming data pipelines on Databricks with Lakeflow Spark Declarative Pipelines.

Official
Advanced