skills.rest
thbeh
thbehCOMMUNITY
Explore all skills by this author →
GitHub

databricks-jobs

Automate Databricks Jobs creation, scheduling, and execution via the Databricks API.

Updated May 31, 2026
Search Tags:#orchestration#data engineering#scheduling#databricks#jobs
One-click install
npx skills add https://github.com/thbeh/coding-agents-databricks-apps --skill databricks-jobs-thbeh
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: databricks-jobs
Source: https://github.com/thbeh/coding-agents-databricks-apps/tree/main/.claude/skills/databricks-jobs
Command: npx skills add https://github.com/thbeh/coding-agents-databricks-apps --skill databricks-jobs-thbeh

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires databricks, pandas, sqlalchemy, and includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill simplifies the creation, management, and execution of Databricks Jobs, enabling users to automate data workflows, schedule tasks, and monitor job health without manual intervention.

Core Features & Use Cases

  • Job Creation: Build and execute jobs that run notebooks, Python scripts, SQL queries, and more.
  • Scheduling: Schedule jobs to run on a cron-based schedule or trigger based on file arrival or table updates.
  • Notifications: Set up email or webhook notifications to alert on job status changes.
  • Health Monitoring: Monitor job health with metrics and health rules to quickly identify and address issues.
  • Use Case: Automate a daily ETL pipeline that extracts, transforms, and loads data into a data warehouse, ensuring consistent data availability and reliability.

Quick Start

Use the databricks-jobs skill to create a new job that runs a Python script in the workspace.

Frequently Asked Questions about databricks-jobs

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I schedule and orchestrate Databricks jobs for a daily ETL pipeline?▼

To schedule and orchestrate Databricks jobs, you can use cron-based triggers or event-based triggers like file arrivals. This automation ensures consistent data availability and reliable execution of your ETL workflows without manual intervention.

Can I run Python scripts and SQL queries in scheduled Databricks jobs?▼

Yes, Databricks job orchestration supports running various task types including Python scripts, SQL queries, and notebooks. You can build and execute these diverse workloads within a single automated job pipeline.

What is the best way to monitor Databricks job health and get failure alerts?▼

Monitoring Databricks job health involves setting up metrics and health rules to quickly identify execution issues. You can configure email or webhook notifications to automatically alert relevant teams about job status changes or failures.

Do I need the databricks-sdk and API to automate job creation and execution?▼

Yes, automating the creation and management of Databricks jobs requires the Databricks API and relevant libraries like databricks-sdk. These dependencies facilitate programmatic interaction with your Databricks environment for job execution.

How does triggering Databricks jobs based on table updates work?▼

Triggering Databricks jobs based on table updates allows the orchestration system to detect data changes and automatically initiate downstream tasks. This event-based scheduling ensures your data pipelines run immediately when source data is modified.

What are the limitations of using Databricks orchestration for data engineering workflows?▼

Databricks orchestration is ideal for data engineering and data science workflows but requires dependencies like databricks, pandas, and sqlalchemy. Users must configure the Databricks API properly to avoid manual intervention and ensure consistent job execution.

Related Skills

Explore More Software Engineering Skills→
alessandro9110alessandro9110
4

databricks-python-sdk

Automate Databricks environments using the Python SDK, Connect, CLI, and REST API.

Community
Advanced
jaredcroxtonjaredcroxton

teams-meeting-pipeline

Operate Microsoft Teams meeting summary pipeline jobs via Hermes CLI.

Community
Advanced
BLSQBLSQ

sql-tables

Read and write SQL tables from Python using SQLAlchemy and pandas.

Official
Advanced
leary-pokenleary-poken
1

databricks-jobs

Automates Databricks job lifecycle including creation, scheduling, and monitoring via CLI.

Community
Advanced