pyspark-structured-streaming

PySpark Structured Streaming pipelines read Kafka JSON events by defining a streaming read stream, applying transformations, and using foreachBatch to write micro-batch outputs to Delta Lake with a durable checkpoint location for fault tolerance.

Event-time watermarking in PySpark tracks the maximum event time observed and automatically drops late data older than the threshold, bounding state size during stateful aggregations across tumbling, sliding, or session windows.

The best way to deduplicate streaming events is using PySpark's built-in deduplication keyed by a natural key, combined with foreachBatch to execute custom upsert logic into sinks like Delta Lake or Iceberg while maintaining fault tolerance through checkpointing.

Yes, stream-stream joins in PySpark Structured Streaming require defining event-time watermarks on both input streams to bound state and manage late data, ensuring the engine correctly matches events within the defined time constraints.

You need to enable the RocksDB state store to improve performance and memory management for PySpark streaming jobs with large stateful aggregations, preventing out-of-memory errors and maintaining operational reliability during complex stateful transformations.

Yes, PySpark Structured Streaming supports writing processed data to Apache Iceberg by utilizing the foreachBatch output sink, which allows you to perform custom upserts and multi-sink logic for production-ready fault-tolerant outputs.