skills.rest
lenlla
lenllaCOMMUNITY
Explore all skills by this author →
GitHub

run-on-databricks

Execute PySpark code on Databricks clusters and validate outputs.

Updated Aug 27, 2026
Search Tags:#validation#pyspark#cluster-management#data-processing#databricks#unity-catalog#artifact-download
One-click install
npx skills add https://github.com/lenlla/one_shot_build --skill run-on-databricks
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: run-on-databricks
Source: https://github.com/lenlla/one_shot_build/tree/main/skills/run-on-databricks
Command: npx skills add https://github.com/lenlla/one_shot_build --skill run-on-databricks

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Enable execution of PySpark workloads on Databricks when local Docker or development environments cannot handle dataset size, access requirements, or production validation needs. The Skill reduces friction around cluster lifecycle, file transfer, and environment configuration so teams can validate, benchmark, and run integration tests at scale.

Core Features & Use Cases

  • Cluster lifecycle management: check cluster status and start clusters using MCP tooling such as cluster_status and start_cluster.
  • Deterministic execution: run PySpark code via the execute_code MCP tool and upload required files ahead of execution.
  • Validation and artifact handling: compare Databricks outputs against local test expectations and download generated artifacts back to the repository.
  • Configuration and security: reads Databricks settings from kyros-agent-workflow/.harnessrc and uses a token supplied via an environment variable while warning to never hardcode credentials.
  • Use cases: full-scale ETL on large datasets, integration tests against real data, production validation before PR, Unity Catalog access, and performance benchmarking.

Quick Start

Check the cluster status, start the cluster if necessary, upload any required files, execute your PySpark code on Databricks, validate outputs against local expectations, and download artifacts for review.

Frequently Asked Questions about run-on-databricks

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run PySpark code on Databricks for large-scale data processing?▼

To run PySpark on Databricks, the Skill checks cluster status, starts clusters via MCP tooling, uploads required files, and executes code using the execute_code MCP tool for scalable data processing and validation.

Do I need a local Docker environment to validate PySpark ETL workloads?▼

No, you do not need local Docker. This Skill executes PySpark directly on Databricks clusters, enabling full-scale ETL runs, integration tests, and performance benchmarking when local development environments are insufficient.

How do I manage cluster lifecycle and file uploads when running integration tests on Databricks?▼

Cluster lifecycle and file uploads are managed through MCP tools like cluster_status and start_cluster. The Skill uploads required files ahead of execution and downloads generated artifacts back to your repository for review.

Can I use this Skill to access Unity Catalog and set schemas in my Spark session?▼

Yes, you can access Unity Catalog and set the catalog and schema directly within the Spark session. This allows production validation and integration testing against real data on Databricks clusters.

How do I configure Databricks credentials and settings securely for production validation?▼

Databricks settings are read from the kyros-agent-workflow/.harnessrc file, and authentication uses a token supplied via an environment variable. The Skill explicitly warns against hardcoding credentials in your code.

What are the limitations of running performance benchmarks on Databricks clusters?▼

Performance benchmarks depend on cluster availability and lifecycle management. You must ensure clusters are started before execution, and outputs are validated against local expectations after downloading generated artifacts.

Related Skills

Explore More Data & Analytics Skills→
databrickslabsdatabrickslabs
38

databricks-config

Automate secure Databricks authentication profile setup for Connect, CLI, and SDK.

Official
Intermediate
i9wa4i9wa4
11

jupyter-notebook

Execute Jupyter notebooks on local Python kernels and Databricks clusters.

Community
Advanced
ThomazRossitoThomazRossito
31

databricks-execution-compute

Execute Spark, Python, SQL, Scala, and R code on Databricks compute resources.

Community
Advanced
kagentikagenti
292

kind

Manage Kind clusters for local Kagenti development and testing.

Official
Intermediate