databricks-jobs

Define tasks with unique task_key values and link them using depends_on to form a DAG. You can configure this via the Python SDK with Task objects, the CLI with JSON, or Asset Bundles YAML, and control execution with run_if conditions like ALL_SUCCESS.

Add a schedule block with a quartz_cron_expression such as "0 0 8 * * ?" for daily at 8 AM, plus a timezone_id and pause_status set to UNPAUSED. This works identically in DABs YAML, the Python SDK CronSchedule class, and CLI JSON.

Databricks Jobs supports cron schedules, periodic interval triggers, file arrival triggers for cloud storage paths, table update triggers for Unity Catalog tables, and continuous mode for always-running jobs. A job can combine multiple trigger types evaluated independently.

The most common cause is pause_status not set to UNPAUSED, or an invalid timezone_id. For file arrival triggers, verify the storage path permissions, and for table update triggers, confirm the tables are in Unity Catalog with proper SELECT grants.

Yes, notebook and Python tasks run on serverless compute when you omit cluster configuration. For custom dependencies, define an environments block with spec.client set to "4" and a dependencies list, then reference it via environment_key on the task.

Use for_each_task with an inputs array, either static JSON or dynamically passed from an upstream task via dbutils.jobs.taskValues.set. The nested task accesses the current item through the {{input}} parameter, and concurrency controls parallel iterations.