skills.rest
d-padmanabhan
d-padmanabhanCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

databricks

Design and standardize Databricks governance workflows for workspace bootstrap, Unity Catalog migration, and DLT pipelines.

15|2|Updated Dec 13, 2025
Search Tags:#terraform#databricks#unity catalog#delta live tables#cluster policies#governance audit#delta time travel
One-click install
npx skills add https://github.com/d-padmanabhan/agent-engineering-handbook --skill databricks-d-padmanabhan
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: databricks
Source: https://github.com/d-padmanabhan/agent-engineering-handbook/tree/main/skills/databricks
Command: npx skills add https://github.com/d-padmanabhan/agent-engineering-handbook --skill databricks-d-padmanabhan

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill provides a production-grade Databricks playbook for standing up governed workspaces, designing Delta Live Tables pipelines, enforcing cluster policies, and performing Unity Catalog migration with recovery-ready guardrails.

Core Features & Use Cases

  • Workspace bootstrap: secure workspace setup with SSO/SCIM, metastore binding, storage credentials, group access, and cost dashboards.
  • Unity Catalog migration: plan and execute migrations from Hive metastore (or unmanaged states) using external locations, cutover runbooks, and rollback.
  • DLT pipeline design & governance: declarative DLT workflows with expectations/quality gates, lineage, CI/CD deployment patterns, and alerting on failures.
  • Cluster policy rollout & cost/performance tuning: enforce runtime pinning, tags, isolation mode, job-vs-interactive compute separation, Photon/cost levers, and monitoring.
  • Audit and recovery: governance audits using system tables and recovery via Delta time travel (RESTORE) with VACUUM-aware constraints.

Quick Start

Ask the AI to generate a Databricks workspace bootstrap plan (Terraform modules, Unity Catalog metastore/catalog layout, cluster policies, and cost dashboard requirements) for your target cloud account.

Frequently Asked Questions about databricks

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I set up a Databricks workspace with Terraform and Unity Catalog?▼

Setting up a Databricks workspace with Terraform involves generating a bootstrap plan that configures SSO/SCIM, binds metastores, establishes storage credentials, and applies group access for secure infrastructure.

What is the best way to migrate from Hive metastore to Unity Catalog?▼

Migrating from Hive metastore to Unity Catalog requires planning external locations, executing a cutover runbook, and maintaining rollback capabilities to ensure a secure and recoverable transition of your data assets.

How do Delta Live Tables expectations work for pipeline data quality?▼

Delta Live Tables expectations act as data quality gates within declarative pipelines, enabling validation rule enforcement, lineage tracking, and failure alerting during DLT workflow operation.

Can I enforce cluster policies to separate job and interactive compute?▼

You can enforce cluster policies to separate job and interactive compute by applying runtime pinning, tags, and isolation modes, creating guardrails for effective cost and performance tuning.

How do I use Delta time travel to RESTORE data after a failed migration?▼

Using Delta time travel to RESTORE data involves leveraging system tables for governance audits and applying RESTORE procedures with VACUUM-aware constraints to recover data after migration failures.

When should I not use VACUUM during a Delta time travel recovery?▼

You should avoid running VACUUM when you need to retain older data versions for Delta time travel recovery, as VACUUM permanently removes files beyond the retention threshold, breaking RESTORE capabilities.

Related Skills

Explore More Data & Analytics Skills→
cockroachdbcockroachdb
3

configuring-sso-and-scim

Configure SSO and SCIM integration for CockroachDB across Cloud Console, DB Console, and SQL/Cluster.

Official
Advanced
JiliarJiliar
1

cicd-pipeline

Design and deploy minimal CI/CD pipelines with GitHub Actions templates.

Community
Advanced
Arry8Arry8
1

mesh-policy-enforcer

Enforce per-device policy controls across multi-device OpenClaw orchestrations.

Community
Advanced
kk0gakk0ga

terraform-search-import

Discover cloud resources and generate Terraform import blocks.

Community
Intermediate