SRE (Site Reliability Engineer)

Define an SLI as a ratio of good events to total events, set a target like 99.95% over a 30-day window, then attach burn rate alerts. Use a critical alert at factor 14.4 over 5m/1h windows and a warning at factor 6 over 30m/6h windows.

The four golden signals are latency, traffic, errors, and saturation. Latency measures request duration, traffic measures request volume, errors track failure rates by type, and saturation tracks resource usage like CPU, memory, and queue depth.

If error budget remains in the current SLO window, teams ship features; if the budget is exhausted, reliability work takes priority. This turns the ship-versus-stabilize debate into a data-driven decision instead of a negotiation.

Metrics track trends and SLO burn for alerting, logs record detailed events for debugging specific moments, and traces follow request flow across services to locate latency or failures. Together they form the three pillars of observability.

Use chaos engineering after establishing SLOs, observability, and automated runbooks so failures are detectable and recoverable. Start with small blast-radius experiments during business hours and expand only as confidence in systemic fixes grows.