skills.rest
LaurentPRAT-DB
LaurentPRAT-DBCOMMUNITY
Explore all skills by this author →
GitHub

spark-declarative-pipelines

Automate Databricks Lakeflow Spark Declarative Pipelines using SQL or Python.

Updated Feb 27, 2026
Search Tags:#streaming#sql#etl#data pipelines#spark#cdc#databricks
One-click install
npx skills add https://github.com/LaurentPRAT-DB/LPT_claude_config --skill spark-declarative-pipelines-laurentprat-db
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: spark-declarative-pipelines
Source: https://github.com/LaurentPRAT-DB/LPT_claude_config/tree/main/skills/spark-declarative-pipelines
Command: npx skills add https://github.com/LaurentPRAT-DB/LPT_claude_config --skill spark-declarative-pipelines-laurentprat-db

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill automates the creation, configuration, and management of Databricks Lakeflow Spark Declarative Pipelines (SDP/LDP), enabling efficient data ingestion, transformation, and CDC/SCD implementation.

Core Features & Use Cases

  • Pipeline Orchestration: Create and update pipelines using SQL or Python.
  • Data Ingestion: Supports Auto Loader, Kafka, Event Hubs, and Kinesis.
  • Change Data Capture: Implements SCD Type 1 and Type 2 using AUTO CDC.
  • Use Case: Set up a new streaming pipeline to ingest real-time sensor data from Kafka, clean and deduplicate it, and store historical changes in a Slowly Changing Dimension table for analytics.

Quick Start

Use the spark-declarative-pipelines skill to initialize a new SQL pipeline project for Databricks.

Frequently Asked Questions about spark-declarative-pipelines

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I build Databricks data pipelines using SQL or Python?▼

You can build Databricks data pipelines by defining data ingestion, transformations, and materialized views declaratively in SQL or Python. This approach supports Auto Loader, streaming tables, and medallion architectures to automate pipeline creation and management.

What is the best way to implement SCD Type 2 and CDC in Spark streaming pipelines?▼

Implementing SCD Type 2 and CDC in Spark streaming pipelines is best handled using AUTO CDC features in Databricks Declarative Pipelines. This automates tracking historical data changes and maintaining Slowly Changing Dimension tables for analytics.

Can I use Databricks Declarative Pipelines for real-time Kafka ingestion and deduplication?▼

Yes, Databricks Declarative Pipelines support real-time data ingestion from Kafka, Event Hubs, and Kinesis using Auto Loader. You can clean, deduplicate, and transform streaming data efficiently within your pipelines.

Does Databricks Lakeflow support serverless compute for Spark declarative pipelines?▼

Databricks Lakeflow fully supports serverless compute for Spark Declarative Pipelines. This allows you to run automated data ingestion and transformations without managing underlying cluster infrastructure, optimizing pipeline deployment.

What are the limitations when automating Databricks pipeline configurations declaratively?▼

Automating Databricks pipeline configurations declaratively is limited to supported compute types and specific data sources like Kafka and Auto Loader. Complex custom logic outside standard SQL or Python declarative syntax may require alternative integration approaches.

Related Skills

Explore More Software Engineering Skills→
teegin-gteegin-g

databricks-spark-declarative-pipelines

Configure Databricks SDP assets with Asset Bundles and serverless compute.

Community
Advanced
Aradhya0510Aradhya0510
3

databricks-spark-declarative-pipelines

Automates creation, configuration, and updating of Databricks Lakeflow Spark Declarative Pipelines.

Community
Advanced
ScottHMcKeanScottHMcKean

databricks-spark-declarative-pipelines

Automate creation and management of Databricks Lakeflow Spark Declarative Pipelines.

Community
Advanced
PaldomPaldom
1

spark-declarative-pipelines

Automate Databricks Spark Declarative Pipelines with serverless compute and Asset Bundles.

Community
Advanced