What problem does it solve?
This Skill helps review and harden streaming data pipelines so delivery semantics, state handling, and failure recovery remain correct under retries, duplicates, late events, and downstream outages.
Core Features & Use Cases
- Delivery semantics decisioning: Choose appropriate at-most-once, at-least-once, or exactly-once strategies based on side effects and downstream requirements.
- Exactly-once design patterns: Apply Kafka transactions (read_committed, transactional offset commits) and Kafka Streams EOS settings (exactly_once_v2).
- At-least-once with idempotency: Implement idempotent consumer writes using upserts/conditional updates to tolerate replays safely.
- DLQ and poison message handling: Design DLQ routing for Kafka Connect and custom handlers, including reprocessing runbooks.
- State and recovery guardrails: Configure Flink checkpoints/safe externalized checkpointing and checkpoint timing constraints.
- Operational stability and observability: Reduce rebalance storm risk (cooperative-sticky/static membership) and set end-to-end latency monitoring and alerting with a reliability checklist.
Quick Start
Ask the agent to review your Kafka/Flink/Spark streaming pipeline design for reliability gaps and produce a corrected plan covering delivery semantics, DLQ handling, checkpointing, rebalance stability, and end-to-end latency SLO monitoring.
