managing-amazon-msk

Operate Amazon MSK Provisioned clusters with Standard and Express brokers.

Updated Sep 8, 2026
One-click install
npx skills add https://github.com/dennisvink/yolomancer --skill managing-amazon-msk-dennisvink
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: managing-amazon-msk
Source: https://github.com/dennisvink/yolomancer/tree/main/skills/aws/specialized-skills/analytics-skills/managing-amazon-msk
Command: npx skills add https://github.com/dennisvink/yolomancer --skill managing-amazon-msk-dennisvink

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve? Operating Amazon MSK Provisioned clusters requires distinguishing between Standard and Express broker types, which behave differently across sizing, storage, maintenance, and monitoring — and generic Kafka knowledge frequently conflates them, leading to wrong recommendations. ## Core Features & Use Cases - Performance and Lag Troubleshooting: Diagnose high CPU, latency, traffic shaping, consumer lag, and rebalance storms with broker-type-specific guidance. - Sizing and Cost Estimation: Run the bundled msk_sizing.py script to compute broker counts, instance choices, and monthly costs for Standard vs Express clusters. - Storage, Monitoring, and Maintenance: Manage EBS expansion and tiered storage on Standard, monitor StorageUsed on Express, configure CloudWatch alarms, and handle patching, upgrades, and rolling restarts. - Streaming Delivery: Set up Streaming Tables to S3 Tables (Iceberg) and Data Delivery to general-purpose S3 buckets on Express brokers, including IAM policies and channel management. - Use Case: A user asks why consumer lag is growing on their MSK cluster; the skill identifies the broker type, walks through lag diagnosis, and recommends client configuration and alarm fixes. ## Quick Start Ask the assistant to diagnose high consumer lag on your Amazon MSK cluster and recommend the right broker size and CloudWatch alarms.

Frequently Asked Questions about managing-amazon-msk

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I choose between MSK Standard and Express brokers?

Express brokers are the default recommendation for most workloads: they scale faster, rebalance partitions quicker, have no maintenance windows, and bill storage per GB retained. Standard brokers use customer-managed EBS and suit cases needing provisioned throughput or tiered storage.

How do I troubleshoot consumer lag on Amazon MSK?

Check MaxOffsetLag and EstimatedMaxTimeLag in CloudWatch at DEFAULT monitoring level, then review consumer settings like session.timeout.ms, max.poll.interval.ms, and partition assignment strategy. The skill's consumer lag reference walks through rebalance storms and stuck consumer groups.

Can I use MSK Streaming Tables or Data Delivery on Standard brokers?

No, Streaming Tables and Data Delivery for general-purpose S3 buckets are only available on Express brokers. On Standard or Serverless clusters, use Amazon Data Firehose, a Kafka Connect S3 sink, or Managed Service for Apache Flink instead.

Does MSK support custom domain names on KRaft clusters?

Yes, via the custom.advertised.listeners cluster configuration property, which works on both ZooKeeper and KRaft mode Provisioned clusters. The older per-broker kafka-configs.sh override does not work on KRaft, and MSK Serverless does not support custom broker configurations.

Why does UnderReplicatedPartitions spike during MSK maintenance?

During patching, MSK performs rolling broker restarts one at a time, so partitions on the offline broker temporarily stop replicating. This is expected on Standard brokers and resolves as the broker catches up; Express brokers do not emit this metric.

What are the limitations of MSK Data Delivery to S3?

Data Delivery works only on Express brokers, delivers no backfill (only records produced after enabling), and has bounded non-adjustable freshness. Only DataFreshnessInSeconds is updatable after channel creation; other changes require deleting and recreating the channel.