What problem does it solve? Building reliable real-time data pipelines requires deep knowledge of checkpointing, watermarks, state stores, and merge semantics. This Skill provides production-tested patterns for Spark Structured Streaming on Databricks, covering Kafka ingestion, stream joins, multi-sink writes, and Delta MERGE operations so you avoid common pitfalls like data loss, state explosion, and duplicate records. ## Core Features & Use Cases - Kafka Streaming Patterns: Ingest Kafka topics into Delta bronze tables, build Kafka-to-Kafka pipelines, enable Real-Time Mode for sub-second latency, and route invalid records to dead letter queues. - Stateful Operations & Checkpoints: Configure watermarks, RocksDB state stores, and persistent checkpoint locations in Unity Catalog volumes with recovery procedures for lost or corrupted checkpoints. - Stream Joins & Multi-Sink Writes: Correlate events across streams with time-bounded joins, enrich streams with Delta dimension tables, and fan out one stream to bronze/silver/gold tables using ForEachBatch. - Merge Optimization: Implement upserts with Liquid Clustering, Deletion Vectors, and parallel merges to multiple tables. - Use Case: You need to match payment events to order events arriving on separate Kafka topics within a 10-minute window, then write matched results to a Delta table with exactly-once guarantees. ## Quick Start Ask the agent to create a Spark Structured Streaming pipeline that reads from a Kafka topic and writes to a Delta table with checkpointing and a 30-second trigger.
