What problem does it solve? Deploying low-latency Apache Beam streaming pipelines on Google Cloud Dataflow involves subtle failure modes: JDK incompatibility with Managed Kafka OAuth, GCS staging hangs, Python version mismatches, BigQuery Storage Write API serialization errors, and worker OOM under heavy throughput. This Skill provides field-proven configurations and resolutions for all of them. ## Core Features & Use Cases - Three-Flag Performance Tuning: Configure worker machine type, state caching, and harness threads to bypass the Python GIL and prevent OOM errors under high throughput. - Managed Kafka Ingestion: Set up SASL/OAUTHBEARER authentication with the JDK 21 harness override and correct Java expansion service classpath dependencies. - SCD Type 1 Enrichment: Implement slowly changing dimensions via dual-source ingestion, GlobalWindows triggers, and AsMultimap side inputs for sub-second CDC-based enrichment. - Reliable BigQuery Writes: Use the Storage Write API with strict local schema loading and a defensive type-conformity DoFn to prevent serialization crashes. - Use Case: A data engineer building a payments pipeline that consumes transactions from Google Managed Kafka, enriches them with merchant dimension data via Pub/Sub CDC streams, and writes conformed records to BigQuery can follow this guide to avoid the six most common production failures. ## Quick Start Ask the assistant to help you configure and launch a Dataflow streaming pipeline that reads from Google Managed Kafka and writes to BigQuery using the Storage Write API.
