kafka-streams-programming

Generate a Gradle project with the Avro plugin, place schemas in src/main/avro, and configure SpecificAvroSerde with your Schema Registry URL. The skill produces App.java, a testable TopologyBuilder, TopologyTestDriver tests, and scripts to create topics and run the app.

Use KTable when both sides are co-partitioned and the join key matches the table key. Use GlobalKTable when the lookup table is small, the join key differs from the record key, or co-partitioning is impractical — it replicates the full table to every instance.

Rebalancing loops usually occur when processing or state restoration exceeds max.poll.interval.ms, causing consumer eviction and repeated rebalances. Increase max.poll.interval.ms, reduce max.poll.records, add standby replicas, and use persistent volumes to avoid full state restoration.

Yes, but WarpStream requires client overrides: larger batch sizes, higher linger, larger fetch sizes, and zone-aware routing via client.id with a ws_az suffix. Exactly-once semantics reduce throughput significantly, so at-least-once with downstream deduplication is preferred.

This error occurs when records were produced without Schema Registry, such as with a plain console producer, so the expected magic byte and schema ID header are missing. Re-produce the data using schema-aware producers like kafka-avro-console-producer.

Avoid exactly_once_v2 when downstream consumers can deduplicate, since at-least-once is simpler and faster. EOS adds transaction overhead, enforces producer constraints, and on WarpStream significantly reduces throughput due to limited in-flight request concurrency.