skills.rest
Aradhya0510
Aradhya0510COMMUNITY
Explore all skills by this author →
WebsiteGitHub

databricks-spark-declarative-pipelines

Automates creation, configuration, and updating of Databricks Lakeflow Spark Declarative Pipelines.

3|1|Updated May 12, 2025
Search Tags:#streaming#etl#data pipelines#spark#medallion architecture#lakehouse#databricks
One-click install
npx skills add https://github.com/Aradhya0510/databricks-cv-accelerator --skill databricks-spark-declarative-pipelines-aradhya0510
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: databricks-spark-declarative-pipelines
Source: https://github.com/Aradhya0510/databricks-cv-accelerator/tree/main/.github/skills/databricks-spark-declarative-pipelines
Command: npx skills add https://github.com/Aradhya0510/databricks-cv-accelerator --skill databricks-spark-declarative-pipelines-aradhya0510

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill automates the creation, configuration, and updating of Databricks Lakeflow Spark Declarative Pipelines (SDP/LDP), enabling efficient data engineering workflows.

Core Features & Use Cases

  • Pipeline Orchestration: Create and manage serverless data pipelines on Databricks.
  • Data Ingestion: Supports Auto Loader for efficient ingestion from cloud storage, Kafka, Event Hubs, and Kinesis.
  • Data Transformation: Implements streaming tables, materialized views, and advanced patterns like SCD Type 1/2 and CDC.
  • Use Case: Set up a new data pipeline to ingest streaming data from Kafka, clean and transform it into silver tables, and then aggregate it into gold tables for business reporting, all managed through a single, declarative configuration.

Quick Start

Use the databricks-spark-declarative-pipelines skill to initialize a new Python pipeline project named 'customer_orders_pipeline' in the current directory, targeting the 'main' catalog and using personal schemas for development.

Frequently Asked Questions about databricks-spark-declarative-pipelines

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I build a Databricks declarative pipeline for streaming ingestion and Medallion architecture?▼

You can build a Databricks declarative pipeline by configuring serverless compute to manage streaming tables and materialized views, transforming raw data through silver layers into gold tables for business reporting.

How do I implement SCD Type 2 and CDC patterns in Databricks Spark pipelines?▼

Implement SCD Type 2 and CDC patterns in Databricks Spark pipelines by defining declarative data transformations that automatically track historical changes and apply data corrections to streaming tables and materialized views.

Can I use Auto Loader in serverless Databricks pipelines to ingest data from Kafka?▼

Yes, Auto Loader works with serverless Databricks pipelines to ingest streaming data efficiently from cloud storage, Kafka, Event Hubs, and Kinesis directly into your Lakehouse architecture.

What is the best way to automate creating and updating Databricks Lakeflow pipelines?▼

Automate Databricks Lakeflow pipelines by using declarative configurations and Databricks Asset Bundles to initialize, configure, and update serverless data pipelines supporting both Python and SQL languages.

Does the Databricks declarative pipelines skill support both Python and SQL for pipeline configuration?▼

Yes, Databricks declarative pipelines support both Python and SQL languages, allowing you to define Auto Loader ingestion, streaming tables, and materialized views within your preferred development environment.

When should I use serverless compute for Databricks Spark declarative pipelines?▼

Use serverless compute for Databricks Spark declarative pipelines when you need automated orchestration of streaming tables, CDC, and SCD Type 2 transformations without managing underlying infrastructure.

Related Skills

Explore More Software Engineering Skills→
JingyiWngJingyiWng

databricks-spark-declarative-pipelines

Create and manage Databricks Lakeflow Spark Declarative Pipelines for streaming and batch workflows.

Community
Advanced
databricks-solutionsdatabricks-solutions
1.8k

spark-declarative-pipelines

Automate creation, configuration, and updating of Databricks Lakeflow Spark Declarative Pipelines.

Official
Advanced
m19cm19c

databricks-spark-declarative-pipelines

Automates creation, configuration, and management of Databricks Lakeflow Spark Declarative Pipelines.

Community
Advanced
manutejmanutej
61

apache-spark-data-processing

Teach Apache Spark data processing across RDDs, DataFrames, Spark SQL, streaming, and MLlib.

Community
Advanced