infra-kafka-cost-optimizer

What problem does it solve?

Kafka clusters often overpay for disk, bandwidth, and operational overhead due to oversized retention, unnecessary partitions, idle consumer metadata, and inefficient compression settings.

Core Features & Use Cases

• Tiered storage for cold logs (local.retention.ms vs retention.ms): Offload older segments from broker disks to S3/GCS to reduce storage spend while keeping long total retention.
• Topic retention right-sizing and audit: Identify over-retained topics and estimate cost impact by combining retention configuration with topic size signals.
• Partition and consumer group hygiene: Detect over-partitioning, reduce partition waste (with safe migration notes), and remove idle consumer groups to lower metadata overhead.
• Compression and cost modeling: Estimate savings from producer/broker compression (e.g., lz4) and compare self-managed vs managed Kafka economics (including cross-region considerations).
• Broker sizing guidance: Use practical checks to tune broker instances based on observed workload characteristics and disk-vs-compute constraints.

Quick Start

Ask your AI to audit your Kafka cluster’s topic retention settings, detect over-retained topics and over-partitioned usage, and propose a tiered storage + retention adjustment plan with an estimated monthly savings using the compression calculator.