observability

Start read-only: inspect the alert's exact label set, query history, for and keep_firing_for durations, and evaluation intervals, then correlate with deploys and dependency signals. Test any rule change against history and document the detection trade-off before applying it.

Create a narrow, time-boxed silence with exact matchers, a named owner, an incident or change reference, and an expiry. Confirm it will not hide other services or severities, and never use broad wildcard silences or indefinite mutes.

No, promtool check rules only validates rule file syntax and structure. It does not verify query correctness, alert routing, cardinality, data availability, or the operational effects of reloading a live Prometheus server.

Silences mute notifications for matching alerts within a time window, inhibition suppresses alerts when another alert is firing, and routing determines which receiver gets a notification. Record which mechanism is operating before changing any of them.

Check target and scrape health, rule evaluation state, route and receiver configuration, alert limits, and active silences or inhibition matchers. A missing series or failed scrape can make dashboards look healthy while detection is broken.

Never raise a threshold, delete an alert, or disable notifications as a diagnostic shortcut during an active incident. Preserve the detection signal, gather evidence first, and escalate if no safe narrow change can be defined.