skills.rest
ricardolui
ricardoluiCOMMUNITY
Explore all skills by this author →
GitHub

blip-pubsub-eventhub-migration

Migrate Azure Event Hubs ingestion pipelines to Google Cloud Pub/Sub Import Topics and BigQuery Iceberg tables.

Updated Jul 7, 2026
Search Tags:#migration#terraform#bigquery#iceberg#oidc#pubsub#azure event hubs
One-click install
npx skills add https://github.com/ricardolui/gcp-custom-agent-skills --skill blip-pubsub-eventhub-migration-ricardolui
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: blip-pubsub-eventhub-migration
Source: https://github.com/ricardolui/gcp-custom-agent-skills/tree/main/blip-pubsub-eventhub-migration
Command: npx skills add https://github.com/ricardolui/gcp-custom-agent-skills --skill blip-pubsub-eventhub-migration-ricardolui

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve? Migrating Azure Event Hubs streaming ingestion to Google Cloud traditionally requires a costly Dataflow fleet of GCE workers. This Skill guides the replacement of that fleet with Pub/Sub Import Topics and BigQuery direct subscriptions writing to BigLake Apache Iceberg tables, cutting ingestion costs by roughly 95% while preserving legacy metadata column contracts. ## Core Features & Use Cases - OIDC Workload Identity Federation: Configure Azure Entra ID App Registrations, federated credentials, and GCP IAM bindings so Pub/Sub can pull from Event Hubs without static SAS connection strings. - SMT JavaScript UDF Transforms: Inject metadata columns (_meta_source_signature, _meta_namespace, _meta_enqueued_time, _meta_partition_id, _meta_sequence_number) so BigQuery rows match legacy Dataflow output contracts. - Terraform Modules & DLQ Architecture: Provision import topics, BigQuery subscriptions with use_table_schema, and a unified regional dead letter queue with correct service agent IAM. - Use Case: A data engineer needs to move 102 Event Hubs pipelines off a 258-VM Dataflow fleet in southamerica-east1; this Skill walks through the 6-phase lifecycle from Azure RBAC to parity audit and cutover. ## Quick Start Ask the assistant to provision a Pub/Sub Import Topic and BigQuery Iceberg subscription for a specific Azure Event Hub namespace using the 6-phase migration lifecycle.

Frequently Asked Questions about blip-pubsub-eventhub-migration

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I migrate Azure Event Hubs to Google Cloud Pub/Sub?▼

Create an Azure Entra ID App Registration with an OIDC federated credential pointing to a GCP service account, assign the Azure Event Hubs Data Receiver role, then provision a Pub/Sub topic with ingestion_data_source_settings for azure_event_hubs via Terraform. Add a BigQuery subscription with use_table_schema enabled to write directly into Iceberg tables.

How do I configure OIDC workload identity between Azure Event Hubs and Pub/Sub?▼

Register an app in Entra ID, add a federated credential with issuer https://accounts.google.com and the GCP service account's unique numeric ID as subject, then grant the Pub/Sub service agent roles/iam.serviceAccountTokenCreator on that GCP service account. Static SAS connection strings are not supported.

Why are my BigQuery metadata columns NULL after Pub/Sub ingestion?▼

NULL metadata columns occur when no JavaScript UDF Single Message Transform is attached, since BigQuery subscriptions write only the data column by default. They also appear when gcloud CLI updates reset useTableSchema to false; always pass --use-table-schema and --drop-unknown-fields on updates.

Can I truncate a BigQuery managed Iceberg table?▼

No, BigQuery Iceberg tables do not support TRUNCATE TABLE, and DELETE fails on rows in the streaming buffer. Use CREATE OR REPLACE TABLE with the same WITH CONNECTION and table_format = 'ICEBERG' options to cleanly reset the table and its storage manifest.

Why does my Pub/Sub import topic show FAILED_PRECONDITION?▼

FAILED_PRECONDITION usually means the Azure App Registration lacks the Azure Event Hubs Data Receiver role, or the Pub/Sub service agent lacks roles/iam.serviceAccountTokenCreator on the GCP ingestion service account. After fixing IAM, run gcloud pubsub topics update with a label change to force re-evaluation.

When should I not use Pub/Sub import topics for Event Hubs migration?▼

Do not use this approach for Confluent Cloud Kafka topics, batch relational database CDC ingestion from SQL Server or PostgreSQL, or when refactoring Composer Airflow DAGs for Dataform. Those scenarios require different ingestion tooling and skills.

Related Skills

Explore More Software Engineering Skills→
gke-labsgke-labs
42

gke-cost-analysis

Analyze GKE infrastructure costs from BigQuery billing export datasets.

Official
Intermediate
developer-advocacy-dremiodeveloper-advocacy-dremio

Dremio Lakehouse Modeling

Design and optimize Dremio lakehouse data models with medallion architecture and Iceberg tables.

Official
Intermediate
sitharaj88sitharaj88
1

gcp-pubsub

Generate Google Cloud Pub/Sub topic and subscription configurations with gcloud CLI and Terraform HCL.

Community
Advanced
Samuelca6399Samuelca6399

real-time-streaming

Design and operate Kafka and Flink pipelines with exactly-once processing.

Community
Advanced