skills.rest
erwinv2k-TKG
erwinv2k-TKGCOMMUNITY
Explore all skills by this author →
GitHub

kafka-streaming

Inspect Kafka brokers, topics, consumer groups, and consumer lag for incident investigation.

Updated Mar 26, 2026
Search Tags:#streaming#diagnostics#kafka#sre#incident investigation#consumer groups#consumer lag
One-click install
npx skills add https://github.com/erwinv2k-TKG/AgentesVSC --skill kafka-streaming-erwinv2k-tkg
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: kafka-streaming
Source: https://github.com/erwinv2k-TKG/AgentesVSC/tree/main/packs/incidentfox/sre-agent/.claude/skills/streaming-kafka
Command: npx skills add https://github.com/erwinv2k-TKG/AgentesVSC --skill kafka-streaming-erwinv2k-tkg

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires confluent-kafka, and includes scripts (resource) components.

What problem does it solve? Investigating Kafka issues during an incident usually requires juggling CLI tools and remembering admin commands. This Skill provides ready-to-run Python scripts that report broker health, topic configuration, consumer group membership, and consumer lag in structured JSON, so you can diagnose streaming pipeline problems quickly. ## Core Features & Use Cases - Broker and Cluster Health Checks: Retrieve cluster ID, controller, and broker inventory as the mandatory first step of any investigation. - Topic Inspection: List topics and describe partitions, replication, configs, and per-partition offsets, including under-replicated partition detection. - Consumer Lag Analysis: Compute per-partition lag with automatic health classification (healthy, minor_lag, lagging, severely_lagging). - Use Case: During an outage, run the broker info script, list consumer groups, then check lag for the affected group to determine whether consumers are stuck or the cluster itself is unhealthy. ## Quick Start Ask the agent to check the consumer lag for a specific Kafka consumer group and report whether it is healthy.

Frequently Asked Questions about kafka-streaming

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I check Kafka consumer lag from Python?▼

Use the get_consumer_lag.py script with a consumer group ID to compare committed offsets against partition high watermarks. It returns per-partition lag, total lag, and a health classification from healthy to severely_lagging.

How to list Kafka topics and consumer groups with confluent-kafka?▼

The list_topics.py and list_consumer_groups.py scripts use the confluent-kafka AdminClient to enumerate topics with partition details and all consumer groups with their states, outputting results as JSON.

Does this Kafka tooling support SASL and SSL authentication?▼

Yes, the shared kafka_client module supports PLAINTEXT, SSL, SASL_SSL, and SASL_PLAINTEXT protocols configured via environment variables. Credentials are injected transparently by a proxy layer, so scripts run without manual secret handling.

Why does describe_topic report under-replicated partitions?▼

A partition is under-replicated when its in-sync replica count is lower than its configured replica count, usually indicating a broker is down or lagging. The script flags these partitions so you can identify availability risks.

What Kafka consumer lag is considered a problem?▼

Total lag of zero is healthy, under 1,000 is minor_lag, under 100,000 is lagging, and 100,000 or more is severely_lagging. These thresholds help prioritize whether a consumer group needs immediate intervention.

Related Skills

Explore More Software Engineering Skills→
wangzitian0wangzitian0

secrets-management

Manage environment variables and secrets across deployment environments using Vault, Pydantic settings, and .env.example files.

Community
Advanced
incidentfoxincidentfox
656

kafka-streaming

Inspect Kafka brokers, topics, consumer groups, and consumer lag via Python scripts.

Official
Intermediate
snoodleboot-iosnoodleboot-io

incident-timeline-creation

Reconstructs incident timelines from server logs, metrics, and chat history.

Official
Intermediate
armoin2018armoin2018
4

ailey-data-kafka

Automate Kafka workflows for real-time data pipelines across Confluent Cloud, Platform, Apache Kafka, and AWS MSK.

Community
Advanced