skills.rest
rywerth-dbx
rywerth-dbxCOMMUNITY
Explore all skills by this author →
GitHub

databricks-connect

Connect local Python code to a Databricks workspace via Databricks Connect.

Updated Feb 7, 2026
Search Tags:#debugging#environment-variables#pyspark#local-development#databricks#databricks-connect
One-click install
npx skills add https://github.com/rywerth-dbx/claude-databricks-starter --skill databricks-connect
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: databricks-connect
Source: https://github.com/rywerth-dbx/claude-databricks-starter/tree/main/.claude/skills/databricks-connect
Command: npx skills add https://github.com/rywerth-dbx/claude-databricks-starter --skill databricks-connect

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Enable local development and debugging of Databricks PySpark code by guiding safe setup and usage of Databricks Connect, avoiding hardcoded credentials and confusion around serverless vs cluster compute.

Core Features & Use Cases

  • Guides configuring a DatabricksSession via Databricks Connect for local-to-remote execution.
  • Covers serverless vs cluster compute decisions, environment-based authentication, and common connectivity issues.
  • Provides best practices for developing and debugging PySpark workflows against a Databricks workspace.

Quick Start

Run a local PySpark workflow using Databricks Connect by calling DatabricksSession.builder.getOrCreate() to connect to your Databricks workspace.

Frequently Asked Questions about databricks-connect

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run PySpark code locally against a remote Databricks workspace?▼

Databricks Connect enables local PySpark development by linking your Python environment to a remote workspace using DatabricksSession.builder.getOrCreate(), allowing you to execute code on Databricks compute.

What environment variables are needed for Databricks Connect local development?▼

Databricks Connect local development requires environment variables like DATABRICKS_CONFIG_PROFILE for authentication, plus either DATABRICKS_SERVERLESS_COMPUTE_ID or DATABRICKS_CLUSTER_ID to select your compute target.

Should I use serverless or cluster compute when debugging PySpark locally?▼

Selecting serverless vs cluster compute for local PySpark debugging depends on your project requirements, configured via DATABRICKS_SERVERLESS_COMPUTE_ID or DATABRICKS_CLUSTER_ID environment variables respectively.

How do I avoid hardcoding credentials when setting up a DatabricksSession?▼

Avoid hardcoding credentials in DatabricksSession by using environment-based configuration, specifically setting the DATABRICKS_CONFIG_PROFILE environment variable to manage authentication profiles safely.

Why is my local PySpark workflow not connecting to the Databricks workspace?▼

Local PySpark connectivity issues to the Databricks workspace typically arise from missing environment variables like DATABRICKS_CONFIG_PROFILE or invalid DATABRICKS_SERVERLESS_COMPUTE_ID and DATABRICKS_CLUSTER_ID settings.

Related Skills

Explore More Software Engineering Skills→
ScottHMcKeanScottHMcKean

databricks-config

Configure Databricks authentication profiles for Connect, CLI, and SDK.

Community
Intermediate
chf3198chf3198

network-platform-resources

Inventory network-accessible compute platforms and manage SSH/Tailscale connections.

Community
Intermediate
jorgealvesjorgealves
2

python-data-pipeline-designer

Design and validate Python ETL pipelines using Pandas, Dask, or PySpark.

Community
Intermediate
rbutinarrbutinar

databricks-run

Execute Databricks Notebooks and Jobs on clusters via command line.

Community
Advanced