sysdesign-top-k-count-min-sketch

You can track top-K items in a high-cardinality stream using a count-min sketch to bound memory while maintaining acceptable approximate results. This approach enforces explicit error and probability parameters to size the data structure correctly.

A count-min sketch works for real-time dashboards by using multiple hash functions to estimate event frequencies in constant memory. It prescribes window sealing and checkpointing to serve approximate top-K results from completed time windows.

For streaming top-K workloads, a Kappa architecture uses a single stream processing path with periodic checkpointing, while Lambda combines batch and speed layers. The design evaluates both to choose the appropriate checkpointing strategy.

Yes, you can use approximate top-K results for high-throughput workloads with many unique keys. This approach is specifically designed for contexts where approximate results are acceptable and clear checkpointing is required.

Serving from sealed windows handles late events and recovery by closing windows before serving results. The design prescribes a clear recovery path using periodic checkpointing to ensure streaming-to-serving consistency.