skills.rest
mkgs-databricks-demos
mkgs-databricks-demosOFFICIAL
Explore all skills by this author →
GitHub

databricks-pipelines

Automates Lakeflow Spark Declarative Pipelines on Databricks for batch and streaming data.

Updated Jul 4, 2026
Search Tags:#data pipelines#spark#databricks#lakeflow#delta live tables
One-click install
npx skills add https://github.com/mkgs-databricks-demos/aiSkillUpdater --skill databricks-pipelines-mkgs-databricks-demos
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: databricks-pipelines
Source: https://github.com/mkgs-databricks-demos/aiSkillUpdater/tree/main/databricks-pipelines
Command: npx skills add https://github.com/mkgs-databricks-demos/aiSkillUpdater --skill databricks-pipelines-mkgs-databricks-demos

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires databricks-cli, spark, and includes scripts (resource) and references (resource) and assets (resource) components.

What problem does it solve?

This Skill helps users develop and manage Lakeflow Spark Declarative Pipelines (formerly Delta Live Tables) on Databricks, automating the creation, management, and execution of data pipelines.

Core Features & Use Cases

  • Automated Pipeline Development: Streamlines the creation of batch or streaming data pipelines using Python or SQL.
  • Integration with Databricks: Leverages Databricks' platform capabilities for data processing and analytics.
  • Use Case: For a user who needs to process and analyze large volumes of data in real-time, this Skill can help automate the end-to-end data pipeline process, from data ingestion to data transformation and analysis.

Quick Start

Use the databricks-pipelines skill to create a new streaming table from a Kafka source.

Frequently Asked Questions about databricks-pipelines

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I create Lakeflow Spark declarative pipelines on Databricks?▼

You can create Lakeflow Spark declarative pipelines on Databricks by using this Skill to automate development in Python or SQL. It streamlines pipeline creation, management, and execution for both batch and streaming data.

Can I build streaming data pipelines from Kafka using Delta Live Tables?▼

Yes, you can build streaming data pipelines from Kafka sources using Delta Live Tables. This Skill automates the end-to-end pipeline process from data ingestion to transformation and analysis.

Do I need the Databricks CLI and Spark to develop declarative pipelines?▼

Yes, you need the Databricks CLI and Spark installed to develop declarative pipelines. These dependencies are required to integrate with Databricks and process or analyze large volumes of data.

What is the best way to automate batch and streaming pipeline execution on Databricks?▼

The best way to automate batch and streaming pipeline execution on Databricks is by leveraging declarative pipelines. This Skill automates execution and integrates directly with Databricks platform capabilities.

Does this Skill support Python and SQL for Delta Live Tables development?▼

Yes, this Skill supports Python and SQL for Delta Live Tables development. It streamlines the creation of batch or streaming data pipelines using either language to define your declarative logic.

Related Skills

Explore More Software Engineering Skills→
RamVegirajuRamVegiraju
4

quickstart

Automate Databricks agent environment setup and authentication configuration.

Community
Intermediate
rappdwrappdw

synthdata-compute

Transforms xlsx, csv, or json datasets into aggregated or scored tables.

Community
Advanced
juanlamadrid20juanlamadrid20

databricks-genie

Create and query Databricks Genie Spaces using natural language.

Community
Advanced
AradotsoAradotso
5

apache-airflow-orchestration

Automate data pipeline scheduling and monitoring with Apache Airflow DAGs.

Official
Advanced