chaos-engineering

Install the Litmus Chaos operator with kubectl, then apply a ChaosEngine manifest targeting your deployment by label. Experiments like pod-delete or pod-network-latency run for a configured duration while HTTP probes continuously verify application health.

Chaos Toolkit is a general-purpose framework that runs YAML-defined experiments against any infrastructure including AWS. Litmus Chaos is Kubernetes-native, using custom resources and operators to inject failures into pods and containers.

Start in staging to validate experiment design, then run in production with a limited blast radius such as a single instance or 1% of traffic. Production testing is necessary because real traffic patterns, dependencies, and data volumes cannot be replicated in staging.

Set blast radius limits restricting affected instances to a small percentage of total capacity, and implement automatic rollback that halts the experiment when error rates double or latency exceeds defined thresholds. Always test rollback procedures before running experiments.

A steady-state hypothesis defines normal system behavior through measurable metrics like p95 latency under 200ms and error rate under 0.1%. Experiments verify the system maintains this state during failure injection rather than testing random destruction.