chaos-engineering

Use k6 fault injection to make a percentage of dependency calls return 503 errors, then assert the circuit breaker opens and the system returns graceful 503 responses with Retry-After headers instead of 500s. Thresholds like http_req_failed rate under 15% verify the breaker absorbs excess failures.

Load testing verifies steady-state performance under expected traffic, while chaos engineering injects failures like dependency errors, timeouts, and latency spikes to verify the system degrades gracefully and recovers. Both are complementary: load tests check NFR targets, chaos tests check behavior when those targets are breached.

Use Toxiproxy for network-level faults that k6 cannot simulate at the HTTP layer, such as TCP connection resets to a database or sustained latency on all connections. k6 handles HTTP error rates and request-level latency; Toxiproxy proxies real connections and adds toxics like latency or connection drops.

No. Chaos tests require a real deployed environment and a controlled blast radius, so they run post-staging-deploy rather than on PRs. The CI job triggers after the staging deployment is healthy and typically only on the main branch.

Skip it for services without explicit resilience NFRs, simple CRUD services with no external dependencies, or systems without high availability SLOs. Without circuit breakers, retries, or degradation requirements in the design, chaos tests add pipeline cost without verifying anything meaningful.

Chaos engineering injects faults into a live staging deployment and observes one outcome, suiting graceful-degradation verification. Deterministic simulation testing runs real code in-process behind a seeded random source, so failures replay exactly; use it for concurrent or distributed bugs needing a precise multi-fault interleaving.