databricks-spark-structured-streaming

Spark Structured Streaming pipelines can ingest data from Kafka to Delta Lake by configuring a read stream from Kafka sources, applying watermarks, and writing to Delta with persistent checkpoint storage and idempotent txnVersion writes.

Late data and stateful operations in Spark Structured Streaming are handled by configuring watermarks and optionally enabling the RocksDB state store to manage large state efficiently while dropping or retaining data past the watermark threshold.

Exactly-once semantics for Delta Lake writes are ensured by configuring idempotent writes using txnVersion and txnAppId, combined with unique persistent checkpoint locations per stream to prevent duplicate data upon restarts.

Databricks Structured Streaming supports both stream-stream and stream-static joins, allowing you to enrich incoming Kafka data with Delta dimension tables while applying watermarks to manage state and avoid indefinite buffering.

Trigger and cost tuning for Spark Structured Streaming on Databricks involves configuring processingTime or Real-Time Mode (RTM) triggers to balance latency and resource costs for production workloads.

A unique checkpoint location is required for each Spark Structured Streaming query to maintain fault tolerance and exactly-once guarantees, preventing state corruption and ensuring correct recovery after failures.