skills.rest
paulasilvatech
paulasilvatechOFFICIAL
Explore all skills by this author →
GitHub

hdinsight-migration

Migrates Azure HDInsight Spark and Hive workloads to Microsoft Fabric Lakehouse and Pipelines.

Updated Aug 19, 2026
Search Tags:#migration#Spark#hive#delta lake#onelake#microsoft fabric#hdinsight
One-click install
npx skills add https://github.com/paulasilvatech/Fabric-Agentic-SDLC --skill hdinsight-migration-paulasilvatech
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: hdinsight-migration
Source: https://github.com/paulasilvatech/Fabric-Agentic-SDLC/tree/main/.github/skills/hdinsight-migration
Command: npx skills add https://github.com/paulasilvatech/Fabric-Agentic-SDLC --skill hdinsight-migration-paulasilvatech

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve? Retiring an Azure HDInsight cluster requires rewriting legacy Spark code, Hive DDL, storage paths, and Oozie workflows for Microsoft Fabric, and doing this manually is error-prone and slow. ## Core Features & Use Cases - Spark API Modernization: Replaces legacy HiveContext, SQLContext, and standalone SparkContext constructors with Fabric's pre-instantiated SparkSession. - Storage Path Conversion: Converts WASB and ABFS storage paths to OneLake abfss URLs via Shortcuts, and transforms Hive DDL (STORED AS ORC, external tables) into Delta Lake schemas inside a Fabric Lakehouse. - Orchestration Migration: Maps Oozie workflow actions (spark, hive, shell, sqoop, coordinator) to Fabric Pipeline activities and schedule triggers, and introduces notebookutils for file and credential operations. - Use Case: When decommissioning an HDInsight cluster, use this Skill to convert a Hive ORC table definition into a Delta table in a Lakehouse schema and rewrite the Oozie coordinator that scheduled it as a Fabric Pipeline trigger. ## Quick Start Migrate my HDInsight Spark notebook that uses HiveContext and wasb:// storage paths to run on Microsoft Fabric.

Frequently Asked Questions about hdinsight-migration

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I migrate an HDInsight Spark cluster to Microsoft Fabric?▼

HDInsight migration to Fabric replaces the persistent cluster with on-demand Fabric Spark pools, moves Hive tables to Delta Lake in a Lakehouse, and converts Oozie workflows to Fabric Data Pipelines. Legacy HiveContext and SparkContext code is replaced with the pre-instantiated spark session.

How to convert WASB paths to OneLake in Spark code?▼

WASB paths are not supported in Fabric and must be replaced with OneLake abfss URLs or Lakehouse-relative Files/ paths. Create a OneLake Shortcut to the existing ADLS Gen2 or Blob container, then reference data via the shortcut path instead of storage account keys.

Can Fabric notebooks access HDFS storage directly?▼

Fabric cannot read or shortcut hdfs:// paths directly. The Fabric Pipeline HDFS connector supports Anonymous authentication only, so Kerberos-secured sources must be exported or bridged to supported storage like ADLS Gen2 before ingestion into OneLake.

How do I convert Hive DDL with STORED AS ORC to Delta Lake?▼

Hive DDL using STORED AS ORC or PARQUET is rewritten as Delta tables using CREATE TABLE ... USING DELTA inside a Lakehouse schema. External tables are read from their source format via a shortcut and written with df.write.format("delta").saveAsTable().

What replaces Oozie workflows in Microsoft Fabric?▼

Oozie workflow actions map to Fabric Data Pipeline activities: spark actions become Notebook or Spark Job Definition activities, sqoop becomes Copy Data, and time-based coordinators become pipeline schedule triggers. Shell and Java actions must be refactored since Fabric has no direct shell execution.

Why does SparkContext() fail in a Fabric notebook?▼

Fabric notebooks pre-instantiate the spark SparkSession and sc SparkContext, so calling SparkContext() or SparkSession.builder.getOrCreate() conflicts with the existing session and raises errors. Remove all legacy context constructors and use spark directly.

Related Skills

Explore More Data & Analytics Skills→
patrikboroschpatrikborosch
14

hdinsight-migration

Migrate HDInsight Spark, Hive, and Oozie workloads to Microsoft Fabric.

Community
Advanced
weiflycc-cmdweiflycc-cmd

spark-operations-cli

Diagnose failed Spark jobs and Livy sessions in Microsoft Fabric via read-only CLI.

Community
Advanced
tomztomz

azure-hdinsight-migration-interactive-query-to-fabric-spark-or-dw

Migrate HDInsight Interactive Query workloads to Microsoft Fabric with schema and data compatibility.

Community
Advanced
microsoftmicrosoft
1.1k

e2e-fabric-cost-estimation

Estimates Microsoft Fabric capacity cost by profiling workloads and fetching live Azure pricing.

Official
Advanced