streaming-pipeline

Define latency and throughput requirements first, then choose Kafka as the message queue and Flink as the stream processor. Design producers with partitioning keys, build consumers with manual offset commits, add windowed aggregations, and enable checkpointing for exactly-once semantics.

Kafka is the durable message queue, not a processor. Flink offers true event-time processing with sophisticated state management, while Spark Structured Streaming uses micro-batches and integrates well with the Spark ecosystem. Choose based on latency needs and existing stack.

Exactly-once combines Flink checkpointing with Kafka transactional producers. Flink periodically checkpoints state, and on failure restarts from the last checkpoint while Kafka transactions ensure outputs are written only once, preventing duplicates and data loss.

Event time uses the timestamp when the event actually occurred, handling out-of-order and late events correctly via watermarks. Processing time uses when the system processes the event, which is simpler but inaccurate for delayed data. Event time is recommended for production pipelines.

Growing consumer lag means consumers cannot keep up with producer throughput. Common causes include slow downstream sinks, insufficient consumer parallelism relative to partition count, or expensive per-event processing. Monitor lag and scale consumers or optimize the processing logic.

Use a dead letter queue for poison-pill events that fail validation or deserialization, so invalid events do not block the entire pipeline. Route them to a separate topic for inspection while transient errors should be retried instead.