investigating-metric-anomalies

Start by characterizing the anomaly to get its direction, magnitude, onset time, and top-moving label values. Then drill into the suspicious label values with filtered metric queries and correlate with logs and traces around the onset time to find the cause.

Characterize the lag metric to find which service is falling behind, then check that service's throughput counter over the same window. Zero throughput during the gap indicates the consumer was down, and logs around the onset typically reveal restarts or crashes.

Counters are only meaningful as rate or increase because they reset on process restart. Gauges should be read as averages, and histograms should be queried with histogram_quantile to inspect tail latency rather than trusting a flat average.

A gap in the series means the emitter stopped reporting, not that the value dropped to zero. When a label value vanishes from top movers, treat it as a dead emitter and pivot immediately to that service's logs.

Yes, by setting baselineFrom and baselineTo to the same window on a previous day. This is important for daily-pattern metrics, which often look anomalous when compared only against the immediately preceding hours.