databricks-jobs

Manage Databricks Jobs with Python SDK, CLI, and Asset Bundles.

Updated Mar 10, 2026

Search Tags:#workflow #orchestration #scheduling #etl #pipelines #databricks #jobs

Ready to run

npx skills add https://github.com/robkisk/coding-agents-databricks-app --skill databricks-jobs-robkisk

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: databricks-jobs
Source: https://github.com/robkisk/coding-agents-databricks-app/tree/main/.claude/skills/databricks-jobs
Command: npx skills add https://github.com/robkisk/coding-agents-databricks-app --skill databricks-jobs-robkisk

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill streamlines the creation, management, and execution of Databricks Jobs, simplifying complex workflow orchestration.

Core Features & Use Cases

Job Creation & Management: Define and manage multi-task DAGs, triggers, and compute configurations.
Workflow Orchestration: Automate data pipelines, ML training, and reporting tasks.
Use Case: Set up a daily ETL pipeline that extracts data, transforms it, loads it into a data warehouse, and then runs validation checks, all orchestrated as a single Databricks Job.

Quick Start

Use the databricks-jobs skill to create a new job named 'My Daily ETL' that runs a notebook located at '/Workspace/Users/[email protected]/etl_notebook'.

Frequently Asked Questions about databricks-jobs

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I orchestrate multi-task ETL pipelines in Databricks?▼

To orchestrate multi-task ETL pipelines in Databricks, you can define complex multi-task DAGs, configure dependencies, and set up triggers using Python SDK, CLI, or Asset Bundles. This enables automated extraction, transformation, loading, and validation checks within a single workflow.

What is the best way to schedule and manage Databricks Jobs programmatically?▼

The best way to schedule and manage Databricks Jobs programmatically is by using the Python SDK, CLI, or Asset Bundles. You can define task dependencies, configure compute resources, and set various trigger types for robust automated workflow execution.

Can I create complex dependency graphs for data pipelines using Databricks Jobs?▼

Yes, you can create complex dependency graphs for data pipelines using Databricks Jobs. The system supports multi-task DAGs, allowing you to chain tasks like data extraction, transformation, loading, and validation checks with managed dependencies.

Does Databricks Jobs support configuring compute and triggers for ML training workflows?▼

Databricks Jobs fully supports configuring compute and triggers for ML training workflows. You can define specific compute configurations and various trigger types to automate and monitor your machine learning pipeline execution.

How do I set up a daily ETL pipeline as a Databricks Job?▼

To set up a daily ETL pipeline as a Databricks Job, you define a scheduled trigger and specify tasks such as running a notebook located in your Workspace. This orchestrates the extraction, transformation, loading, and validation steps into a single automated job.

What are the limitations of using Asset Bundles for Databricks job orchestration?▼

Asset Bundles for Databricks job orchestration are used alongside Python SDK and CLI to define task structures and compute configurations. While they facilitate robust monitoring and complex DAGs, specific limitations depend on your configured compute resources and trigger types.