skills.rest
personamanagmentlayer
personamanagmentlayerOFFICIAL
Explore all skills by this author →
WebsiteGitHub

databricks-expert

Provide Databricks expertise on Spark, Delta Lake, MLflow, and Unity Catalog.

41|9|Updated Jan 13, 2026
Search Tags:#data engineering#spark#pyspark#lakehouse#mlflow#databricks#delta lake
One-click install
npx skills add https://github.com/personamanagmentlayer/pcl --skill databricks-expert-personamanagmentlayer
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: databricks-expert
Source: https://github.com/personamanagmentlayer/pcl/tree/main/stdlib/data/databricks-expert
Command: npx skills add https://github.com/personamanagmentlayer/pcl --skill databricks-expert-personamanagmentlayer

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill provides expert-level guidance and code examples for leveraging the full power of the Databricks platform, enabling efficient data engineering, machine learning, and analytics.

Core Features & Use Cases

  • Databricks & Spark Expertise: Deep knowledge of cluster management, Delta Lake, PySpark transformations, and SQL.
  • MLflow Integration: Streamlines experiment tracking, model registry, and deployment.
  • Workflow Orchestration: Demonstrates Databricks Jobs and notebook utilities for building robust pipelines.
  • Unity Catalog: Covers catalog, schema, and access control management for data governance.
  • Use Case: Optimize a large Delta Lake table for faster queries, set up an MLflow experiment to track model training, or build a multi-task Databricks job for a daily ETL process.

Quick Start

Use the databricks-expert skill to create a new Delta Lake table named 'customer_data' partitioned by 'date' from a JSON source located at '/mnt/raw/customer_data'.

Frequently Asked Questions about databricks-expert

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I optimize a Delta Lake table for faster query performance?▼

To optimize Delta Lake tables for faster queries, apply partitioning strategies on date or categorical columns and compact small files. This skill provides PySpark and Spark SQL examples to configure partitions, manage table maintenance, and improve query performance.

What is the best way to track machine learning models with MLflow in Databricks?▼

Track ML models with MLflow by logging parameters, metrics, and artifacts during PySpark training runs. This skill covers experiment tracking, model registry management, and deployment workflows to streamline your ML lifecycle on the Databricks platform.

How do I manage data governance and access control with Unity Catalog?▼

Manage data governance with Unity Catalog by defining catalogs, schemas, and tables to apply granular access controls. This skill provides configuration steps for securing data assets and managing permissions across your Databricks lakehouse environment.

Can I build a multi-task ETL pipeline using Databricks Jobs?▼

Yes, you can build multi-task ETL pipelines using Databricks Jobs and notebook utilities for orchestration. This skill demonstrates configuring job dependencies, scheduling daily data processing tasks, and structuring robust data engineering workflows.

Does this skill cover structured streaming and time travel in Delta Lake?▼

Yes, this skill addresses advanced use cases including structured streaming for real-time data processing and time travel for querying previous table versions. It provides implementation details for leveraging Delta Lake capabilities within the Databricks platform.

Related Skills

Explore More Data & Analytics Skills→
PaldomPaldom
11

databricks-synthetic-data-gen

Generate synthetic data with Spark and Faker into Unity Catalog volumes.

Community
Advanced
m19cm19c

databricks-model-serving

Deploy and query MLflow models and AI agents on Databricks Model Serving endpoints.

Community
Advanced
PatrickGallucciPatrickGallucci
16

fabric-spark-compute-perf-remediate

Diagnose and remediate performance issues across Fabric Spark compute and Delta Lake tables.

Community
Advanced
robertlupo1997robertlupo1997

data-engineer

Designs scalable data pipelines and manages them across hybrid IT and OT environments.

Community
Advanced