alerting-rules

Define alert rules in a Prometheus rules file with an expr condition, a for duration to filter transient blips, severity labels, and annotations linking to runbooks. Alert on symptoms like error rate above 5% for 5 minutes rather than internal causes.

Configure Alertmanager with a routing tree that matches severity labels to receivers. Send critical alerts to a PagerDuty receiver with a service key, warnings to a Slack channel, and info alerts to email, using group_wait and repeat_interval to control batching.

Burn rate alerting fires when the error budget is consumed faster than the SLO allows, such as 14.4x the allowed rate for a 99.9% target. It detects customer-impacting issues faster than static thresholds by measuring budget consumption over a time window.

Add a for duration so alerts only fire on sustained conditions, use percentiles like p95 instead of averages, and set inhibit rules so downstream alerts are suppressed when a parent alert like cluster down is firing. Track false positive rates and tune thresholds quarterly.

Yes, use amtool to create silences matching alertname or service labels with a duration and comment, such as silencing ServiceDown for two hours during planned maintenance. Silences can be queried and expired early when work completes.

Write alert rule tests with input series values and expected alerts at given evaluation times, then run them with promtool test rules. This verifies thresholds fire correctly in staging before deploying to production.