Chaos Engineering and Resilience Testing

Skill metadata

Name chaos-engineering-resilience-testing · Level senior · Tags chaos-engineering resilience fault-injection gameday sre reliability aws-fis

"Chaos engineering practice: steady-state hypotheses tied to SLIs, blast-radius containment, pre-agreed abort criteria and automated rollback, fault injection with AWS FIS, Chaos Mesh or LitmusChaos, and GameDay facilitation that produces fixes. Use when a failover or redundancy claim has never actually been tested, when planning a GameDay or resilience exercise, or when deciding whether it is safe to inject failure into production and how to bound it."

Source: skills/sre-slo-sla-observability/chaos-engineering-resilience-testing/SKILL.md

When to Use This Skill

Triggers — load this skill when:

• A redundancy or failover claim exists on paper and has never been exercised
• A GameDay, resilience exercise, or failure-injection experiment is being planned
• Injecting failure into production is proposed and the blast radius must be bounded
• Incidents keep revealing dependencies nobody knew were hard requirements

Route elsewhere when:

• Restore, backup, and region-loss recovery drills -> backup-and-disaster-recovery
• Finding a throughput or latency ceiling under load -> performance-load-testing
• Designing the redundancy being tested -> scalability-high-availability-patterns
• Defining the SLIs the hypothesis measures against -> sli-slo-error-budget-design
• Running the incident an experiment escalates into -> incident-management-and-postmortem

1. An experiment without a hypothesis is an outage

Every experiment is written down before anything is injected, in this shape:

experiment: checkout-survives-single-az-loss
steady_state:
  # Measured, not asserted. These are existing SLIs with existing dashboards.
  - metric: checkout_success_rate      # good events / valid events
    expected: ">= 99.5% over 5m"
  - metric: checkout_p99_latency_ms
    expected: "<= 800"
hypothesis: >
  Losing every checkout pod in eu-west-1a keeps success rate above 99.5% and p99 under 800ms,
  because two other zones hold sufficient warm capacity and the LB ejects failing targets
  within 30 seconds.
method:
  - terminate all checkout pods in eu-west-1a
blast_radius:
  scope: checkout service, one AZ, staging first then production
  traffic: 100% (this is a redundancy claim; a sample would not test it)
  duration: 10 minutes maximum
abort_criteria:            # ANY of these ends the experiment immediately
  - checkout_success_rate < 99.0% for 60s
  - checkout_p99_latency_ms > 2000 for 60s
  - error budget consumption for the period exceeds 5%
  - any customer-reported incident, related or not
rollback: scale the AZ back up; automated, tested, under 60 seconds
owner: "@payments-team"
observers: ["@sre-oncall", "@payments-oncall"]

If the steady state cannot be measured, stop. You will not be able to tell a successful experiment from an outage, and the exercise becomes an argument about screenshots.

2. Bound the blast radius deliberately

Dimension	First run	Escalation path
Environment	Staging with production-shaped traffic	Production off-peak
Scope	One instance, one pod, one dependency	One AZ, then one region
Traffic share	1% via header-routed cohort	100% for redundancy claims
Duration	60 seconds	Up to the abort window
Recovery	Manual, human at the keyboard	Automated and timed

Two rules that keep this safe: the abort must not depend on the thing being broken — a kill switch that reads a config service you just partitioned is not a kill switch — and the experiment runs with a human watching, until its automated abort has been proven by deliberately tripping it.

3. Fault injection with a managed service

AWS FIS is the low-effort start on AWS: it holds the stop condition as a first-class object bound to a CloudWatch alarm, so the abort criteria live in the experiment rather than in a runbook.

{
  "description": "checkout-survives-single-az-loss",
  "targets": {
    "checkoutTasksAZ1": {
      "resourceType": "aws:ecs:task",
      "selectionMode": "ALL",
      "resourceTags": { "service": "checkout" },
      "filters": [
        { "path": "AvailabilityZone", "values": ["eu-west-1a"] }
      ]
    }
  },
  "actions": {
    "stopTasks": {
      "actionId": "aws:ecs:stop-task",
      "targets": { "Tasks": "checkoutTasksAZ1" }
    }
  },
  "stopConditions": [
    {
      "source": "aws:cloudwatch:alarm",
      "value": "arn:aws:cloudwatch:eu-west-1:111122223333:alarm/checkout-slo-burn-fast"
    }
  ],
  "roleArn": "arn:aws:iam::111122223333:role/fis-checkout-experiment",
  "tags": { "experiment": "checkout-az-loss", "owner": "payments-team" }
}

The FIS role is scoped by tag to the checkout service only. A chaos tool with broad permissions is itself the largest single-blast-radius risk in the platform.

Chaos Mesh and LitmusChaos are the in-cluster equivalents when the fault is Kubernetes-shaped — network latency, DNS failure, packet loss, pod kill:

apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata:
  name: payments-dependency-latency
  namespace: staging
spec:
  action: delay
  mode: all
  selector:
    namespaces: [staging]
    labelSelectors: { app: checkout }
  direction: to
  target:
    mode: all
    selector:
      namespaces: [staging]
      labelSelectors: { app: payments-gateway }
  delay:
    latency: "400ms"
    jitter: "100ms"
  duration: "5m" # always set a duration: the cluster self-heals if the operator dies

Latency injection finds more real defects than termination does. Killing a pod tests the scheduler; adding 400ms to a dependency tests every timeout, retry budget, connection pool, and circuit breaker you have — which is where the actual bugs live.

4. The failure modes worth injecting, in order of yield

Fault	What it usually exposes
Dependency latency (+300–500ms)	Missing timeouts, unbounded retries, pool exhaustion, retry storms
Dependency returns errors	Fallbacks that were never exercised; cache stampede on recovery
DNS failure	Clients that resolve once at startup and never again
Single pod or instance kill	Whether readiness and connection draining actually work
AZ loss	Warm capacity assumptions and cross-AZ retry cost
Certificate or token expiry	The renewal path nobody has run since setup
Clock skew	Token validation, distributed locks, TTL logic
Full disk on one node	Log rotation, spool handling, eviction behaviour

5. A GameDay that produces fixes, not slides

1. Two weeks before — pick one hypothesis, write the experiment, name the owner and observers.
2. One week before — dry-run in staging; confirm the abort criteria fire when tripped on purpose.
3. Day of — announce in the incident channel, note the start time, inject, watch the SLI.
4. During — one scribe records timestamps: injection, first alert, first human action, recovery.
5. Immediately after — the gap between injection and first alert is the most valuable number produced. It is the detection debt you did not previously know you had.
6. Within a week — file the fixes. An experiment whose findings do not become tickets was entertainment.

Success is not "nothing broke". Success is a validated hypothesis or a discovered defect. An experiment that reveals a broken assumption before a customer does is the highest-value outcome.

6. Best practices and anti-patterns

Do:

• Tie the hypothesis to an existing SLI so the verdict is arithmetic, not opinion.
• Test the abort path first, deliberately, before relying on it.
• Announce experiments and label the telemetry with the experiment ID, so an unrelated real incident is not misdiagnosed as the experiment — and vice versa.
• Run experiments when the error budget can afford them, and stop when it cannot.
• Re-run the experiment after the fix. An unverified fix is a hypothesis.
• Automate the ones that pass into a recurring, low-blast-radius schedule so resilience does not regress silently.

Do not:

• Inject failure in production first. Earn it in staging, with the abort proven.
• Run without pre-agreed abort criteria. Deciding mid-experiment whether things are bad enough to stop is how an experiment becomes an incident.
• Give the chaos tool broad IAM or cluster-admin. Scope it by tag or label to its targets.
• Skip the duration or TTL on an injected fault; the operator can die holding the fault open.
• Measure only whether the system recovered. Also measure whether anyone was told — silent recovery hides an alerting gap that will matter when recovery does not happen.
• Call it chaos engineering when it is unannounced breakage. Without a hypothesis, a bounded blast radius, and an abort criterion, it is an outage with a nicer name.