kafka-streams-topology

Design a Kafka Streams topology by defining KStream, KTable, and GlobalKTable abstractions, then apply join operations to combine data. Use KStream for event data, KTable for reference tables, and GlobalKTable for broadcast joins; specify join windows and co-partition topics as needed to enrich or correlate events.

KStream represents an unbounded stream of events with no state; KTable is a changelog stream representing the latest value per key, updated incrementally; GlobalKTable replicates an entire table to all instances for broadcast joins without co-partitioning. Choose based on whether you need event semantics, keyed state, or global reference lookups.

Enable exactly-once semantics (EOS) by setting processing.guarantee to exactly_once_v2 in configuration. This ensures each record is processed and state is updated atomically, preventing duplicates even during failures. Requires compatible source and sink connectors and increases latency slightly.

Kafka Streams supports tumbling windows (fixed, non-overlapping intervals), hopping windows (fixed, overlapping intervals), and session windows (activity-based, close when idle). Choose tumbling for regular time-based reports, hopping for sliding analysis, and session windows for user-session analytics or burst detection.

Yes. Use filter() to drop records matching conditions, map() to transform values, flatMap() to emit zero or more records per input, and branch() to split streams by predicates into separate subtopologies. These stateless operations compose for flexible event routing and transformation.

Co-partition topics before joins, use appropriate window retention, configure state-store cleanup policies, enable EOS for critical pipelines, handle null values explicitly, and test topology logic independently. Design for failure recovery, monitor lag and throughput, and document state-store semantics for maintainability.