principle-resiliency

Prevent cascading failure by designing for partial failure and controlled degradation instead of total outages. Isolate resource pools per dependency using bulkheads, enforce timeout-driven I/O boundaries, and define clear failure domains to reason about blast radius.

Graceful degradation keeps services useful during partial outages by allowing a system to fail-soft and maintain limited functionality instead of crashing completely. It involves designing fallback ladders and choosing appropriate fail-fast versus fail-soft strategies when dependencies return intermittent errors.

Design bulkheads by isolating resource pools such as connection pools, semaphores, thread pools, and queues per dependency. This fault isolation technique prevents one slow or failing dependency from exhausting shared resources and starving other critical system components.

Health checks cause restart storms when liveness and readiness probes are not properly configured for upstream outages. Misleading probes can incorrectly trigger service restarts or probe storms, so you must differentiate liveness versus readiness behavior and design safe fallback modes.

The best way to define failure domains is to identify which components fail together to avoid unnamed blast radii. Map cascading failure paths across your service architecture, isolate resources per dependency, and enforce safe deadline-driven I/O boundaries for production readiness.

Use fail-fast strategies when you need immediate feedback to prevent resource starvation, and use fail-soft strategies when maintaining partial service availability is critical. Correct failure handling requires choosing appropriately per dependency to ensure graceful degradation and controlled system behavior.