skills.rest
ScottHMcKean
ScottHMcKeanCOMMUNITY
Explore all skills by this author →
GitHub

databricks-spark-declarative-pipelines

Automate creation and management of Databricks Lakeflow Spark Declarative Pipelines.

3|Updated Jan 2, 2025
Search Tags:#python#streaming#sql#etl#pipelines#cdc#databricks
One-click install
npx skills add https://github.com/ScottHMcKean/genai_notebooks --skill databricks-spark-declarative-pipelines-scotthmckean
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: databricks-spark-declarative-pipelines
Source: https://github.com/ScottHMcKean/genai_notebooks/tree/main/.cursor/skills/databricks-spark-declarative-pipelines
Command: npx skills add https://github.com/ScottHMcKean/genai_notebooks --skill databricks-spark-declarative-pipelines-scotthmckean

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill automates the creation, configuration, and management of Databricks Lakeflow Spark Declarative Pipelines (SDP), streamlining data ingestion, transformation, and analytics workflows.

Core Features & Use Cases

  • Pipeline Creation & Management: Initializes new projects using Databricks Asset Bundles or manages existing pipelines via MCP tools.
  • Data Ingestion: Supports Auto Loader for cloud files and streaming sources like Kafka and Event Hubs.
  • Data Transformation: Implements complex streaming patterns, SCD Type 1/2, and CDC using SQL or Python.
  • Performance Optimization: Leverages Liquid Clustering and advanced configuration for efficient data processing.
  • Use Case: Set up a new, production-ready data pipeline for ingesting streaming sensor data, cleaning it, performing windowed aggregations, and storing results in a gold layer table, all configured for serverless compute.

Quick Start

Use the databricks-spark-declarative-pipelines skill to initialize a new SQL pipeline project named 'customer_orders_pipeline' in the current directory.

Frequently Asked Questions about databricks-spark-declarative-pipelines

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I build a Databricks streaming pipeline with Auto Loader and serverless compute?▼

Yes, Databricks Spark Declarative Pipelines support SCD Type 2 and CDC implementations using SQL or Python. The Skill automates pipeline configuration to handle these complex data transformation patterns natively.

Can I implement SCD Type 2 and CDC in Databricks declarative pipelines using Python?▼

Yes, Databricks Spark Declarative Pipelines support SCD Type 2 and CDC implementations using SQL or Python. The Skill automates pipeline configuration to handle these complex data transformation patterns natively.

What's the best way to manage existing Databricks Lakeflow pipelines without starting from scratch?▼

Managing existing Databricks Lakeflow pipelines is handled through integrated MCP tools. This allows you to configure and update streaming tables and materialized views without needing to initialize a new project from scratch.

How do I optimize streaming sensor data aggregations in a Databricks gold layer table?▼

Optimizing streaming sensor data aggregations in a Databricks gold layer uses Liquid Clustering and advanced pipeline configurations. The Skill automates these settings for efficient windowed aggregations on serverless compute.

Does this declarative pipeline approach support both Kafka and Event Hubs streaming sources?▼

Yes, Databricks declarative pipelines support streaming sources like Kafka and Event Hubs. The Skill configures Auto Loader and streaming ingestion patterns to process data directly from these platforms into your pipeline.

Related Skills

Explore More Software Engineering Skills→
JingyiWngJingyiWng

databricks-spark-declarative-pipelines

Create and manage Databricks Lakeflow Spark Declarative Pipelines for streaming and batch workflows.

Community
Advanced
m19cm19c

databricks-aibi-dashboards

Automate creation, testing, and deployment of Databricks AI/BI dashboards.

Community
Advanced
datasciencemonkeydatasciencemonkey
27

databricks-spark-declarative-pipelines

Automate creation, configuration, and lifecycle management of Databricks SDP pipelines.

Community
Advanced
thbehthbeh

databricks-spark-declarative-pipelines

Automate Spark data pipeline development and execution on Databricks.

Community
Advanced