skills.rest
mirakui
mirakuiCOMMUNITY
Explore all skills by this author →
Website𝕏@mirakuiGitHub

databricks-spark-declarative-pipelines

Automate Databricks SDP/LDP pipeline configuration and lifecycle across environments.

Updated Sep 9, 2017
Search Tags:#pipeline#cdc#databricks#sdp#ldp#scd
One-click install
npx skills add https://github.com/mirakui/dotfiles --skill databricks-spark-declarative-pipelines-mirakui
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: databricks-spark-declarative-pipelines
Source: https://github.com/mirakui/dotfiles/tree/main/claude/skills/databricks-spark-declarative-pipelines
Command: npx skills add https://github.com/mirakui/dotfiles --skill databricks-spark-declarative-pipelines-mirakui

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Automates the configuration, deployment, and lifecycle management of Databricks Lakehouse SDP/LDP pipelines, providing a standardized approach to serverless deployments, CDC, SCD, and Auto Loader ingestion patterns.

Core Features & Use Cases

  • Standardizes language choice (Python or SQL) and enforces serverless defaults for new projects.
  • Covers streaming tables, materialized views, AUTO CDC (SCD Type 1/2), and Auto Loader ingestion patterns.
  • Supports multi-environment asset bundles and medallion architecture (bronze/silver/gold) across dev/stage/prod.

Quick Start

Create a new SDP project skeleton and deploy to the default environment using Asset Bundles.

Frequently Asked Questions about databricks-spark-declarative-pipelines

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I set up serverless Databricks pipelines with Auto Loader and CDC ingestion?▼

You can build serverless Databricks SDP/LDP pipelines by standardizing on Python or SQL, configuring Auto Loader ingestion patterns, and enabling CDC and SCD Type 2 with streaming tables and materialized views.

What is the best way to manage Databricks declarative pipeline configurations across dev, stage, and prod?▼

Managing Databricks declarative pipelines across environments is handled using multi-environment Asset Bundles, allowing you to promote medallion architecture configurations smoothly through dev, stage, and prod.

Can I use Python instead of SQL for Databricks SDP streaming tables and materialized views?▼

Yes, you can use either Python or SQL for Databricks SDP streaming tables and materialized views. The pipeline configuration enforces your language choice while defaulting to serverless compute for new projects.

Does Databricks SDP support SCD Type 2 and Auto CDC for streaming tables?▼

Databricks SDP supports AUTO CDC for implementing SCD Type 1 and Type 2 within streaming tables, automating change data capture and historical data tracking for your Lakehouse pipelines.

How do I create a new Databricks SDP project skeleton and deploy it using Asset Bundles?▼

You can quickly create a new SDP project skeleton and deploy it to your default environment by utilizing Databricks Asset Bundles, which standardizes the deployment and lifecycle automation of serverless pipelines.

When should I use serverless compute for Databricks Lakehouse pipelines over standard clusters?▼

You should use serverless compute for new Databricks Lakehouse SDP/LDP projects to automate lifecycle management and standardize deployments, as serverless is enforced as the default for new pipeline configurations.

Related Skills

Explore More Data & Analytics Skills→
juanlamadrid20juanlamadrid20

databricks-spark-declarative-pipelines

Automate creation and configuration of Databricks Lakeflow Spark Declarative Pipelines.

Community
Advanced
m19cm19c

databricks-aibi-dashboards

Automate creation, testing, and deployment of Databricks AI/BI dashboards.

Community
Advanced
RajuNaikRajuNaik
1

databricks-medallion-architecture

Design and implement Bronze/Silver/Gold medallion data architectures on Databricks.

Community
Advanced
databricksdatabricks
251

databricks-pipelines

Develop batch and streaming data pipelines on Databricks with Lakeflow Spark Declarative Pipelines.

Official
Advanced