eng-incident-response

Incident response for production outages is managed through a five-phase workflow: Detect, Investigate, Mitigate, Resolve, and Postmortem. This process includes severity classification, hypothesis-driven investigation, and standardized artifact creation to restore service and document findings.

A blameless postmortem is an audit-ready artifact created after resolving an incident, focusing on root cause analysis without assigning personal blame. It is needed within 48 hours of resolving P0–P3 production outages or degraded services to document evidence and prevent recurrence.

Hypothesis-driven diagnostics generate and prioritize multiple hypotheses for an incident, run targeted tests to validate them, and converge on a root cause using evidence. This structured investigation process is applied during the Investigate phase of incident management.

Yes, this incident management workflow applies to engineering teams handling P0–P3 incidents across cloud and product systems. It requires access to PRODUCTS.md for tech-stack context and local docs/ write access to store incident records and postmortem files.

MCP cloud logging is optional for incident response, used to record a one-line incident summary in cloud memory and send notifications. The core workflow functions with local docs/ write access for postmortem files and does not strictly require MCP.

The best way to structure on-call workflows is using a multi-phase process with decision gates and severity classification. This approach guides teams from detection through mitigation and resolution, producing standardized incident records and postmortems for audit readiness.