skills.rest
JosiahSiegel
JosiahSiegelCOMMUNITY
Explore all skills by this author →
GitHub

databricks-2025

Orchestrate Databricks jobs in Azure Data Factory with Managed Identity authentication.

51|10|Updated Oct 22, 2025
Search Tags:#orchestration#managed identity#2025#databricks#connectors#azure data factory
One-click install
npx skills add https://github.com/JosiahSiegel/claude-plugin-marketplace --skill databricks-2025
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: databricks-2025
Source: https://github.com/JosiahSiegel/claude-plugin-marketplace/tree/main/plugins/adf-master/skills/databricks-2025
Command: npx skills add https://github.com/JosiahSiegel/claude-plugin-marketplace --skill databricks-2025

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill addresses the challenge of efficiently orchestrating Databricks jobs within Azure Data Factory, ensuring adoption of the latest recommended practices and connectors for 2025.

Core Features & Use Cases

  • Databricks Job Activity: Guides users to migrate from legacy activities to the recommended DatabricksJob activity for modern Databricks orchestration.
  • 2025 Connector Updates: Details enhancements and migration paths for connectors like ServiceNow V2, PostgreSQL, and Microsoft Fabric Warehouse.
  • Managed Identity Best Practices: Emphasizes the mandatory use of Managed Identities for secure and passwordless authentication.
  • Use Case: A data engineer needs to update an existing ADF pipeline to use the new Databricks Job Activity and ensure it uses Managed Identity for authentication to Azure Blob Storage.

Quick Start

Update an Azure Data Factory pipeline to use the Databricks Job Activity with Managed Identity authentication for Azure Blob Storage.

Frequently Asked Questions about databricks-2025

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I orchestrate Databricks jobs in Azure Data Factory using the recommended 2025 practices?▼

To orchestrate Databricks jobs in Azure Data Factory, use the recommended DatabricksJob activity instead of legacy activities. This approach leverages modern connectors and Managed Identities for secure, passwordless authentication within your ADF pipelines.

What is the best way to set up Managed Identity authentication for Databricks linked services in ADF?▼

Managed Identity authentication for Databricks linked services in ADF requires configuring the linked service to use the system or user-assigned identity. This passwordless method is mandatory for securely connecting Databricks to resources like Azure Blob Storage.

How do I migrate Azure Data Factory pipelines to the new Databricks Job Activity?▼

Migrating Azure Data Factory pipelines to the Databricks Job Activity involves updating your pipeline configuration to replace legacy activities. This ensures modern orchestration, compatibility with Spark 3.3 data flows, and integration with DevOps best practices.

What are the 2025 connector updates for Azure Data Factory when working with Databricks?▼

The 2025 connector updates for Azure Data Factory include enhancements and migration paths for ServiceNow V2, PostgreSQL, and Microsoft Fabric Warehouse. These updates facilitate streamlined data ingestion and orchestration within Databricks pipelines.

Does Azure Data Factory support ServiceNow V2 and Microsoft Fabric Warehouse connectors for Databricks orchestration?▼

Azure Data Factory supports orchestration with updated ServiceNow V2, PostgreSQL, and Microsoft Fabric Warehouse connectors. The 2025 updates provide specific migration paths and enhancements to integrate these sources with Databricks jobs.

Why should I use Managed Identities instead of passwords for Databricks authentication in Azure Data Factory?▼

Managed Identities should be used instead of passwords for Databricks authentication in Azure Data Factory to enforce secure, passwordless access. This best practice eliminates credential exposure when connecting ADF pipelines to Azure Blob Storage and other resources.

Related Skills

Explore More Software Engineering Skills→
pauldotyupauldotyu

aks-workload-identity

Configure Microsoft Entra Workload ID on AKS clusters for passwordless pod authentication.

Community
Advanced
thbehthbeh

databricks-spark-declarative-pipelines

Automate Spark data pipeline development and execution on Databricks.

Community
Advanced
bmarcurellabmarcurella

connectors-data

Connect agents to enterprise data sources via OpenAPI, MCP, and A2A protocols.

Community
Advanced
microsoftmicrosoft
2.9k

azure-ai-ml-py

Manage Azure ML resources programmatically with MLClient and DefaultAzureCredential.

Official
Advanced