What problem does it solve? Migrating Databricks PySpark Delta Lake medallion streaming pipelines to Google Cloud is error-prone when done as a literal 1-to-1 conversion, leading to unnecessary cost, latency, and architectural overhead. This Skill provides architectural mappings, code translation patterns, and cost models for moving to Apache Beam on Dataflow with BigQuery. ## Core Features & Use Cases - Architecture Mapping: Translates Bronze/Silver/Gold Delta Lake layers into native GCP equivalents using KafkaIO, BigQuery Storage Write API, and Dataform/dbt, eliminating the GCS staging sync stream entirely. - Code Translation Patterns: Provides PySpark-to-Beam conversions for Kafka Avro deserialization, stateful deduplication with State & Timers, and side-input dimension enrichment joins. - Cost & Performance Modeling: Supplies formulas comparing Databricks 24/7 cluster plus DBU costs against Dataflow serverless autoscaling, plus latency comparisons between micro-batch and record-by-record streaming. - Use Case: A data engineer needs to migrate a six-stream Databricks payment pipeline to GCP. Use this Skill to map each Delta stream to its Beam/BigQuery equivalent, translate the MERGE deduplication logic into Beam stateful transforms, and build the cost comparison for stakeholders. ## Quick Start Ask the assistant to translate your PySpark Delta Lake streaming pipeline into an Apache Beam Dataflow pipeline with a cost comparison against your current Databricks setup.
