long-running-server

Long-running background tasks exceeding standard HTTP timeouts can be executed asynchronously using background execution patterns. This allows operations to run independently while avoiding HTTP timeout failures.

The best way to handle asynchronous tasks taking more than 120 seconds is using background execution with persistent storage. This supports complex data analysis and model training by storing results in PostgreSQL after completion.

Databricks Lakebase supports persistent storage for async task results by utilizing Lakebase PostgreSQL. This configuration ensures that results from long-running background executions are reliably persisted and retrievable.

Background execution on Databricks supports three request patterns: standard, background plus poll, and background plus stream. These patterns handle asynchronous operations by managing how task results are retrieved after execution.

You need the databricks-ai-bridge agent-server dependency to run long-running server tasks. Proper Lakebase configuration is also required to enable the asynchronous execution and PostgreSQL result persistence.