skills.rest
ivanshamaev
ivanshamaevCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

apache-kafka

Configure Apache Kafka topics, producers, consumers, connectors, and monitoring.

14|1|Updated May 5, 2026
Search Tags:#kafka#topic design#exactly-once#schema registry#kafka connect#consumer groups#producer configuration
One-click install
npx skills add https://github.com/ivanshamaev/de-agent-skills --skill apache-kafka-ivanshamaev
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: apache-kafka
Source: https://github.com/ivanshamaev/de-agent-skills/tree/main/skills/apache_kafka
Command: npx skills add https://github.com/ivanshamaev/de-agent-skills --skill apache-kafka-ivanshamaev

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill helps you design and operate Apache Kafka message pipelines safely, efficiently, and with the right delivery guarantees.

Core Features & Use Cases

  • Kafka topic design: partition counts, replication factor, durability knobs (e.g., ISR, min.insync.replicas), retention, and naming conventions.
  • Producer/consumer configuration: batching, compression, acks, idempotence, commit strategies, rebalance behavior, and offset management.
  • Exactly-once semantics & schema governance: transactional producers/consumers and Schema Registry compatibility planning.
  • Kafka Connect & DLQ operations: source/sink connector configurations, error handling with DLQs, and REST-based lifecycle management.
  • Operational monitoring: consumer lag, CLI management, and key operational metrics for troubleshooting.

Quick Start

Tell the agent: Design the Kafka topic and producer/consumer settings for an orders event stream that needs at-least-once durability, manual commits, and Schema Registry compatibility planning.

Frequently Asked Questions about apache-kafka

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I configure Kafka producer acks and idempotence for exactly-once delivery?▼

Kafka exactly-once delivery requires configuring transactional producers with idempotence enabled, proper `acks` settings, and isolation levels for consumers to ensure messages are processed without duplication or loss.

What's the best way to design Kafka topics for partitioned workloads with retention and replication?▼

Kafka topic design requires setting partition counts for parallelism, replication factor for fault tolerance, `min.insync.replicas` for durability, and retention policies to manage data lifecycle across partitioned workloads.

How do I handle Kafka Connect connector errors with a Dead Letter Queue?▼

Kafka Connect error handling routes failed records to a Dead Letter Queue (DLQ) topic, allowing the main pipeline to continue processing while isolating poison pills for later troubleshooting and reprocessing.

Why does Kafka consumer lag occur and how can I monitor it?▼

Kafka consumer lag occurs when consumers cannot keep up with producer rates, and you can monitor it using CLI tools and operational metrics to track offset differences and rebalance behavior across consumer groups.

Can I use Schema Registry to manage Kafka topic compatibility?▼

Schema Registry manages Kafka topic compatibility by enforcing schema evolution rules, ensuring producers and consumers can safely evolve data formats without breaking downstream processing pipelines.

Do I need to configure Kafka consumer commit strategy for at-least-once durability?▼

Kafka at-least-once durability requires manual consumer commit strategies to ensure offsets are only committed after successful message processing, preventing data loss during consumer rebalances or failures.

Related Skills

Explore More Software Engineering Skills→
heyAyushhheyAyushh
2

kafka-kraft-firecracker

Provision Apache Kafka in KRaft mode inside Firecracker microVMs.

Community
Advanced
doanchienthangdevdoanchienthangdev
4

Stream Processing

Process unbounded data streams with Kafka Streams, Apache Flink, and ksqlDB.

Community
Advanced
involvexinvolvex

database-migrations-migration-observability

Monitor database migrations with CDC pipelines, metrics, and alerts.

Community
Advanced
shafibabarshafibabar
1

go-amqp-publisher

Publish AMQP messages in Go with publisher confirms and mandatory routing.

Community
Advanced