chaos

To inject controlled faults for resilience testing, apply a canonical 12-fault suite including timeouts, errors, and malformed tool output within strictly isolated sandbox environments. This validates that systems recover gracefully while maintaining safe blast-radius controls and separation from production.

Validating resilience ladder transitions requires fault classes such as model-unavailable, model-slow, partial-response, tool-errors, tool-output-malformed, rate-limit-burst, context-overflow, memory-corrupt, clock-skew, audit-log-write-fail, and vault-unavailable. These specific fault injections exercise recovery procedures across diverse failure conditions.

Document chaos engineering hypotheses and observations by creating plan ADRs to capture expected outcomes before execution and result ADRs to record actual observations afterward. This ADR-driven process ensures structured planning, evidence collection, and follow-up actions for your fault injection experiments.

You can safely run chaos testing by using a sandbox-first approach with strict blast-radius controls. This methodology ensures complete separation from production environments and provides safe rollback capabilities, allowing you to inject faults and validate resilience without risking live system stability.

The best way to execute a sandboxed chaos experiment is to plan a representative project test, apply the 12-fault suite under strict isolation, observe outcomes across timeouts and errors, and document both a plan ADR and result ADR to capture hypotheses, observations, and follow-ups.

You should not run fault injection in production because uncontrolled faults like context-overflow, memory-corrupt, or vault-unavailable can cause irreversible data loss or system failure. The sandbox-first approach with strict blast-radius controls and safe rollback is required to prevent unintended damage.