Prompt file imported from cdalsoniii/brightpath-coder (.cursor/commands/chaos-experiment-design.md). Copyright stays with the author.

Chaos Experiment Design

Overview

Design a chaos experiment with a clear hypothesis, steady-state definition, and blast radius control to validate system resilience under failure conditions.

Steps

1. Define steady-state hypothesis

   • Document what "normal" system behavior looks like using metrics (latency, error rate, throughput)
   • Establish baseline measurements for key SLIs (availability, latency, throughput)
   • Define acceptable ranges for each metric during normal operation
   • Identify critical user journeys that must remain functional

2. Identify the variable to inject

   • Select failure type: network partition, service crash, resource exhaustion, dependency failure, or data corruption
   • Determine injection point: specific service, network segment, infrastructure component
   • Plan injection method: manual, automated tool (Chaos Monkey, Litmus, Gremlin), or infrastructure-level
   • Specify injection duration and intensity (gradual vs immediate)

3. Set blast radius limits and abort conditions

   • Define maximum scope: single service, availability zone, or percentage of traffic
   • Establish abort triggers: error rate exceeds threshold, critical SLI violated, cascading failures detected
   • Set time limits: maximum experiment duration, auto-abort after X minutes
   • Define rollback procedures: how to immediately restore normal conditions

4. Plan monitoring and observation during experiment

   • Configure dashboards to track key metrics in real-time
   • Set up alerts for abort conditions and critical thresholds
   • Enable distributed tracing to observe request flows during failure
   • Document observation points: what to watch, when to check, who monitors

5. Document expected vs actual results

   • Record hypothesis: what behavior is expected when failure is injected
   • Capture actual observations: metrics, logs, traces, user impact
   • Compare expected vs actual: did system behave as predicted?
   • Document gaps: unexpected behaviors, unhandled failure modes, missing mitigations
   • Create action items: improvements needed based on findings

Checklist

• Steady-state hypothesis documented with measurable SLIs and baseline metrics
• Failure injection variable clearly defined with specific injection point and method
• Blast radius limits set with clear abort conditions and rollback procedures
• Monitoring dashboards and alerts configured before experiment starts
• Hypothesis documented with expected behavior and failure modes to validate
• Results documented comparing expected vs actual with identified gaps
• Action items created for improvements based on experiment findings

References

• Rule: .cursor/rules/063-chaos-engineering.mdc
• Rule: .cursor/rules/resilience-fault-tolerance-standards.mdc
• Plan: .cursor/plans/resilience_chaos_engineering.plan.md
• Skill: .cursor/skills/chaos-engineering/SKILL.md