skills.rest
ThomazRossito
ThomazRossitoCOMMUNITY
Explore all skills by this author →
Website𝕏@thomazrossitoGitHub

hadoop-to-databricks

Migrates Hadoop ecosystems to Databricks through an eight-phase playbook with deterministic DDL generators.

4|1|Updated May 22, 2026
Search Tags:#migration#hive#databricks#Unity Catalog#hadoop#delta lake#oozie
One-click install
npx skills add https://github.com/ThomazRossito/ai-data-agents --skill hadoop-to-databricks-thomazrossito
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: hadoop-to-databricks
Source: https://github.com/ThomazRossito/ai-data-agents/tree/main/plugins/ai-data-agents/skills/hadoop-to-databricks
Command: npx skills add https://github.com/ThomazRossito/ai-data-agents --skill hadoop-to-databricks-thomazrossito

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) components.

What problem does it solve? Migrating a Hadoop ecosystem (Hive, HDFS, Oozie, Sqoop, Ranger/Kerberos) to Databricks is error-prone when done by hand: DDL conversion, reconciliation, orchestration conversion, and cutover each carry risks of silent data loss or broken governance. This Skill provides an operational playbook that enforces discovery-first analysis, a human approval gate, and deterministic generators instead of hand-written SQL. ## Core Features & Use Cases - Manual Discovery Playbook: Step-by-step Beeline, HDFS, YARN, Oozie, and Ranger commands to inventory Hive schemas, volumes, jobs, and security policies, since no dedicated Hive MCP exists. - Deterministic Conversion: Runs scripts/hive_generate.py to convert raw SHOW CREATE TABLE output into Delta Lake DDL and scripts/reconcile_generate.py to produce two-phase reconciliation SQL, with gates that fail loudly on errors. - Full Migration Lifecycle: Covers complexity scoring and wave planning, Oozie to Lakeflow Jobs conversion, Ranger/Kerberos to Unity Catalog mapping, CDC strategy, and a cutover/rollback runbook with numeric rollback thresholds. - Use Case: A data platform team decommissioning a CDH cluster uses this Skill to inventory 200 Hive tables, generate an approved migration SPEC, convert DDL deterministically, reconcile snapshots before enabling CDC, and execute a phased cutover with documented rollback criteria. ## Quick Start Ask the agent to start discovery for migrating your Hive databases to Databricks by providing the output of SHOW CREATE TABLE for your tables.

Frequently Asked Questions about hadoop-to-databricks

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I migrate Hive tables to Databricks Delta Lake?▼

Collect SHOW CREATE TABLE output via Beeline into a single .sql file, then run scripts/hive_generate.py to produce Delta DDL with correct type mapping, CLUSTER BY from partition and bucket columns, and SerDe/LOCATION clauses removed. Never hand-write the DDL.

How to convert Oozie workflows to Databricks jobs?▼

Map Oozie workflows to Lakeflow multi-task Jobs, coordinators to scheduled Jobs, and bundles to multiple Jobs via DABs. Hive actions become SQL tasks, shell actions become notebook tasks, and fork/join becomes parallel tasks with depends_on. Note Oozie CRON has 5 fields while Quartz has 6, so prepend a 0 for seconds.

Can Pig Latin or MapReduce jobs be automatically converted to Spark?▼

No, Pig Latin, MapReduce Java, and HBase workloads require complete rewrites, not mechanical conversion. Lakebridge does not transpile them. Pig operations map conceptually to PySpark or Spark SQL, and MapReduce mappers/reducers map to select/groupBy patterns, but each needs redesign effort.

Does Databricks Catalog Federation work with on-premises HDFS?▼

No, Hive Metastore Federation does not support genuinely on-premises HDFS or remote Thrift metastores. For on-prem HDFS data, use DistCp to copy files to cloud storage (ABFSS, S3A, or GS), then ingest with COPY INTO or Auto Loader.

How do I validate data after Hadoop to Databricks migration?▼

Run scripts/reconcile_generate.py with the generated spec to produce source and target reconciliation SQL, then follow the two-phase rule: approve a full snapshot reconciliation before enabling CDC, then reconcile only the delta. Fix the source-side quoting from brackets to backticks for HiveQL.

What are the limitations of the Hive DDL generator?▼

The generator always targets catalog.gold schema regardless of the source database, so adjust it if using Bronze/Silver/Gold layering. Its reconciliation spec also lacks a Hive source dialect, emitting bracket quoting that is invalid in HiveQL and must be manually changed to backticks.

Related Skills

Explore More Data & Analytics Skills→
microsoftmicrosoft
952

hdinsight-migration

Migrate Azure HDInsight Spark and Hive workloads to Microsoft Fabric.

Official
Advanced
m19cm19c

databricks-spark-declarative-pipelines

Automates creation, configuration, and management of Databricks Lakeflow Spark Declarative Pipelines.

Community
Advanced
vivekganavivekgana
4

access-management

Automate RBAC/ABAC access management with Python and SQL templates for Unity Catalog.

Community
Intermediate
RamVegirajuRamVegiraju
4

databricks-config

Configure Databricks profiles in ~/.databrickscfg and authenticate via CLI.

Community
Advanced