databricks-python-sdk

Guides Databricks development using the Python SDK, Databricks Connect, CLI, and REST API.

Updated Mar 24, 2026
One-click install
npx skills add https://github.com/AarushiShah/coding-agents-databricks-apps --skill databricks-python-sdk-aarushishah
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: databricks-python-sdk
Source: https://github.com/AarushiShah/coding-agents-databricks-apps/tree/main/.claude/skills/databricks-python-sdk
Command: npx skills add https://github.com/AarushiShah/coding-agents-databricks-apps --skill databricks-python-sdk-aarushishah

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires databricks-sdk, databricks-connect.

What problem does it solve? Developers working with Databricks often struggle to find correct SDK method signatures, authentication patterns, and API usage across clusters, jobs, SQL warehouses, Unity Catalog, and model serving. This Skill provides verified code patterns and documentation links so you can write correct Databricks SDK code without guessing parameters or hunting through docs. ## Core Features & Use Cases - Complete API Coverage: Working code examples for clusters, jobs, SQL statement execution, warehouses, Unity Catalog (catalogs, schemas, tables, volumes), files, serving endpoints, vector search, pipelines, secrets, and DBUtils. - Authentication Patterns: Environment variables, named profiles, explicit tokens, Azure service principals, and account-level clients. - Async Safety Guidance: Critical patterns for wrapping the synchronous SDK with asyncio.to_thread() in FastAPI and other async applications. - Use Case: You need to build a FastAPI service that queries a SQL warehouse and lists Unity Catalog tables. This Skill gives you the exact WorkspaceClient patterns, statement execution code, and async wrapping required to avoid blocking the event loop. ## Quick Start Ask the agent to write Python code using the Databricks SDK to list all clusters and run a SQL query on a warehouse.

Frequently Asked Questions about databricks-python-sdk

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I authenticate with the Databricks Python SDK?

Set the DATABRICKS_HOST and DATABRICKS_TOKEN environment variables, then call WorkspaceClient() with no arguments. You can also use a named profile from ~/.databrickscfg via WorkspaceClient(profile="MY_PROFILE") or pass host and token explicitly.

How do I run a SQL query with the Databricks SDK?

Use w.statement_execution.execute_statement with a warehouse_id, your SQL statement, and a wait_timeout like "30s". Check response.status.state for StatementState.SUCCEEDED, then read rows from response.result.data_array.

Does the Databricks SDK work with FastAPI or asyncio?

The Databricks SDK is fully synchronous and blocks the event loop in async applications. Wrap every SDK call with asyncio.to_thread(), for example await asyncio.to_thread(lambda: list(w.clusters.list())), to keep FastAPI endpoints responsive.

When should I use the REST API instead of SDK methods?

Prefer SDK methods when available. Use w.api_client.do(method, path, body) only for new API endpoints not yet in the SDK, complex operations where the SDK abstraction is problematic, or debugging raw API responses.

How do I run Spark code locally against Databricks?

Install databricks-connect and use DatabricksSession.builder.getOrCreate(), which auto-detects the DEFAULT profile from ~/.databrickscfg. Do not set .master("local[*]") as it breaks Databricks Connect.

How do I wait for long-running operations like cluster creation?

Use the *_and_wait variants such as w.clusters.create_and_wait with a timeout, or call .result() on the Wait object returned by create. You can also use wait_get_cluster_running with a callback for progress updates.