Chaos engineering without production changes

Definition: inject faults into a model

In Cloud World Model, chaos is a method inside the simulation, not the product category. You inject controlled failures (for example instance loss, zone outage, database stress, or network latency) into a virtual model and step it to estimate resilience and remediation signals — simulated chaos for early hypothesis testing before you deploy or change infrastructure.

Start with a falsifiable failure hypothesis

• Name the suspected weak point, such as an app instance, zone, database, or network path.
• Choose the traffic level that makes the question meaningful and hold other inputs steady.
• Select metrics that answer the question: latency percentiles, errors, goodput, per-resource CPU, connection use, health, or estimated cost.
• Write down model coverage and assumptions that may affect how you interpret the output.

State what you expect to happen before running the experiment. For example: if one app server becomes unavailable while traffic is at its expected peak, will the remaining servers and database still serve the offered load without unacceptable errors? Define the target resource, traffic condition, signals to inspect, and what would count as evidence against your expectation. A useful hypothesis names the architecture behavior to learn about; it does not claim that a virtual result predicts production exactly. Keep the question specific enough that the result could change your next design or validation decision.

Build a topology and establish baseline traffic

Create a virtual topology that represents the path under test: entry point, application resources, data stores, and their connections. Match resource types and sizing to what the model supports, and set scaling behavior deliberately. If the goal is N-1 capacity, use a fixed app fleet so the experiment reveals what the surviving servers can carry rather than allowing an autoscaler to change the fleet during the comparison. Apply baseline traffic and advance simulation steps until the metrics provide a baseline for comparison. Record the offered traffic alongside the outcome; latency without workload context is easy to misread. Keep the topology and workload consistent across experiment states so observed differences have a clear cause.

Copy-and-run AWS replay (virtual resources only)

This authenticated script creates one CWM simulation with a virtual ALB, three fixed m5.large app resources, and a db.r5.large MySQL database. It then records, in order, baseline traffic, a targeted reversible app outage, peak traffic while that app is down, baseline traffic with the app still down, and recovery. Set CWM_API_KEY to a World Model API key before running it. The script calls only CWM simulation endpoints; it does not use AWS credentials or create or change AWS resources. Each printed metrics object is the model's current estimate for that run, not a production measurement. Seed 420 makes the stochastic inputs repeatable for the same model version and configuration.

set -euo pipefail

if [ -z "${CWM_API_KEY:-}" ]; then
  echo "Set CWM_API_KEY to your World Model API key first." >&2
  exit 1
fi

BASE_URL="https://www.cloudworldmodel.ai/api"

api_post() {
  curl --fail-with-body -sS -X POST "$BASE_URL$1" \
    -H "Authorization: Bearer $CWM_API_KEY" \
    -H "Content-Type: application/json" \
    --data "$2"
}

api_get() {
  curl --fail-with-body -sS "$BASE_URL$1" \
    -H "Authorization: Bearer $CWM_API_KEY"
}

step_and_report() {
  label="$1"
  result=$(api_post "/simulations/$SIM_ID/step" '{}')
  echo "$label"
  printf '%s\n' "$result" | jq '{
    offeredRps: .simulation.traffic,
    latencyP50Ms: .metrics.latencyP50,
    latencyP95Ms: .metrics.latencyP95,
    errorRate: .metrics.errorRate,
    goodputRps: .metrics.throughput,
    estimatedCostUsdPerHour: .metrics.costPerHour,
    resources: [.simulation.resources[]? |
      select(.type == "compute" or .type == "database") |
      {name, status, cpuPct: .cpuUsage, routedRps}]
  }'
}

created=$(api_post "/simulations" '{
  "name": "AWS chaos replay (virtual)",
  "traffic": 300,
  "seed": 420,
  "appWeight": "typical",
  "resources": [
    {"id":"alb","name":"ALB","type":"network","provider":"aws",
      "location":{"regionKey":"us-east-2"},
      "characteristics":{"serviceFamily":"elb"}},
    {"id":"app-1","name":"App 1","type":"compute","provider":"aws",
      "location":{"regionKey":"us-east-2"},
      "characteristics":{"size":"m5.large","instanceCount":1}},
    {"id":"app-2","name":"App 2","type":"compute","provider":"aws",
      "location":{"regionKey":"us-east-2"},
      "characteristics":{"size":"m5.large","instanceCount":1}},
    {"id":"app-3","name":"App 3","type":"compute","provider":"aws",
      "location":{"regionKey":"us-east-2"},
      "characteristics":{"size":"m5.large","instanceCount":1}},
    {"id":"db","name":"MySQL database","type":"database","provider":"aws",
      "location":{"regionKey":"us-east-2"},
      "characteristics":{"size":"db.r5.large","serviceFamily":"rds",
        "workloadDatabaseEngine":"mysql","multiAz":false}}
  ],
  "connections": [
    {"sourceId":"alb","targetId":"app-1"},
    {"sourceId":"alb","targetId":"app-2"},
    {"sourceId":"alb","targetId":"app-3"},
    {"sourceId":"app-1","targetId":"db"},
    {"sourceId":"app-2","targetId":"db"},
    {"sourceId":"app-3","targetId":"db"}
  ]
}')
SIM_ID=$(printf '%s\n' "$created" | jq -er '.id')
echo "Virtual simulation: $SIM_ID"

# 1. Healthy fleet at baseline.
step_and_report "1. Baseline: 300 RPS, all three app resources healthy"

# 2. Persist a reversible failure on app-3, then compare at the same load.
api_post "/simulations/$SIM_ID/failures" '{
  "name":"Targeted app instance outage",
  "type":"instance_down",
  "targetResourceId":"app-3",
  "startTime":0
}' >/dev/null
step_and_report "2. Targeted failure: app-3 down, traffic remains at 300 RPS"

# 3. Raise load with the same target still unavailable.
api_post "/simulations/$SIM_ID/inject-traffic" '{"targetRps":450}' >/dev/null
step_and_report "3. Peak load: app-3 still down, traffic set to 450 RPS"

# 4. Restore baseline traffic without recovering the app yet.
api_post "/simulations/$SIM_ID/inject-traffic" '{"targetRps":300}' >/dev/null
step_and_report "4. Traffic reduced: app-3 still down, traffic back to 300 RPS"

# 5. Start recovery and step until app-3 is healthy, reporting each result.
api_post "/simulations/$SIM_ID/recover-resource" '{"resourceId":"app-3"}' >/dev/null
for attempt in 1 2 3 4 5 6 7 8; do
  step_and_report "5. Recovery step $attempt"
  app_status=$(api_get "/simulations/$SIM_ID" | jq -er '.resources[] | select(.id == "app-3") | .status')
  if [ "$app_status" = "healthy" ]; then
    echo "Recovery complete: app-3 is healthy."
    break
  fi
done
if [ "$app_status" != "healthy" ]; then
  echo "app-3 is not healthy yet; inspect the simulation and continue stepping." >&2
fi

Inject an instance or zone failure

• Keep baseline traffic unchanged immediately after the failure so its effect can be separated from a load change.
• Inspect health and per-resource metrics, not only one aggregate resilience or error score.
• Save before and after snapshots or metric history so the causal comparison is explicit.

Target the resource that matches the hypothesis instead of accepting a random target. An instance failure asks how much traffic the surviving application fleet can absorb; a zone outage asks a broader placement and routing question and is meaningful only when the topology represents zones. Apply the failure in the virtual simulation and advance it. Inspect which resource became unavailable, whether traffic shifted as expected, and whether the failed component or a downstream dependency became the limiting factor. A failure that does not change the modeled path may indicate a test-design issue, not proof of resilience.

Add peak traffic while the resource remains down

Once the failure-only state is understood, raise traffic to the peak condition while keeping the resource unavailable. Combining reduced capacity with increased demand is more informative than checking an average-load failure and a healthy-fleet traffic spike independently. Track offered RPS and goodput separately: goodput is the successfully served portion of demand, while errors describe requests the model did not serve successfully. Check the application fleet and database together; spare application CPU does not prove that a database has CPU or connection headroom. Compare the resulting error rate with its component causes where available. CPU-overload and database-pressure explanations can both contribute, so do not treat one component diagnosis as the whole result. Record estimated cost at the same state, but keep cost distinct from performance and reliability conclusions.

This is a controlled what-if comparison, not a forecast of an exact real incident. The model's inputs, represented resource behaviors, and workload assumptions determine what can be inferred. If a metric is absent or a component is not represented, mark that gap rather than filling it with an assumption presented as a result.

Worked example: replayable modeled AWS sequence

The script above is a virtual reference scenario, not a measured production result, an observed customer deployment, an AWS load test, or an independent validation of a production system. It creates an AWS Application Load Balancer, three fixed m5.large app resources, and one db.r5.large MySQL database with typical app weight. The table describes what to inspect at each state; the script's returned values are estimates for that run and configuration.

The script's step responses are the results to use when replaying against the current model version and settings. The fixed seed makes the run repeatable with the same model version and configuration, but a later model revision may return different latency, CPU, cost, or recovery timing. These are modeled estimates; compare them with production measurements only as separate evidence, never as if one were the other.

Reduce traffic, recover the resource, and compare states

After recording the peak-under-failure state, reduce traffic to the original baseline while leaving the failure in place. Step again and check whether errors and latency return to the baseline pattern with the reduced fleet. Then initiate recovery for the failed resource and continue stepping through its recovery lifecycle; issuing a recovery action does not mean the server is immediately healthy or serving traffic. Verify resource health and routing as it rejoins, and note when the full modeled architecture is healthy. Compare five explicit snapshots: before failure, during failure at baseline and peak traffic, after traffic reduction, and during/after recovery. This makes the effect of each intervention legible instead of conflating failure, load, and recovery.

Use the replay script above to observe the current modeled state at every phase. Its returned values and recovery duration depend on the model version and configuration. They do not establish how quickly an actual load balancer detects a real instance loss or how an actual application behaves during recovery.

Interpret the result as a capacity question

Use the replay to test N-1 capacity at peak traffic, not only average load. If the returned peak-under-failure state shows a database limit or increased errors while baseline and failure-only states remain healthy, the modeled result suggests a capacity question to investigate. A possible design question is whether the fleet, database, traffic controls, or architecture should change, but this single modeled example does not prove which production change is correct. Run comparable variants by changing one factor at a time, preserve the same workload, and identify which outputs are estimates and which evidence is independently measured. Use the result to prioritize a question, not to declare a production system reliable or unreliable.

Check current accuracy methodologyRead simulation fidelity and assumptions

How CWM compares with AWS FIS and Gremlin

CWM, AWS Fault Injection Service (FIS), and Gremlin can support different stages of a resilience program. CWM applies failure actions to a virtual model, which is useful for exploring hypotheses and architecture alternatives before deployment or before changing infrastructure. AWS FIS documents experiments that execute actions against real AWS resources, under experiment controls and safeguards configured by the operator. Gremlin documents reliability experiments against systems an organization operates. Live tools can expose dependencies, operational behavior, and telemetry that a model does not include; they also require real-world scope, authorization, safeguards, and abort planning. These approaches are complementary rather than interchangeable: use simulation to refine a question, then choose an appropriately controlled live experiment when production evidence is needed.

Gremlin documentationCWM accuracy methodology

Limits, evidence, and responsible next steps

A modeled outcome is an estimate conditional on the topology, workload, configuration, pricing inputs, and behavior that the simulator represents. It is not a measured production result, an invoice, an SLO, an availability guarantee, or evidence that a deployment will recover the same way. A model can omit application-specific behavior, hidden dependencies, real network conditions, provider control-plane timing, and operational response. Coverage and assumptions matter as much as the displayed metric; consult published accuracy and fidelity material before relying on any result.

Use the simulation to produce a more specific question for later validation. Before a real experiment, confirm ownership and authorization, choose a bounded blast radius, define abort conditions, notify responsible operators, and select production indicators that directly test the hypothesis. Record actual observations separately from modeled values, compare them without relabeling either, and revise the model or assumptions when they disagree. Neither a simulated pass nor one guarded experiment establishes universal production reliability.

Review accuracy evidenceRead the glossaryExplore pre-provision simulation

Evidence: simulation is not a production experiment

AWS Fault Injection Service documents experiments that execute actions against real AWS resources. That is materially different from CWM's simulated chaos jobs. The CWM fidelity and accuracy pages document what is modeled and where measured validation is limited. Do not report a modeled resilience score as measured availability or claim a simulated pass demonstrates production recovery.

For an example of CWM's publicly described evaluation methodology, read the owned AWS benchmark. It does not establish production-fault equivalence.

Worked API example: start and inspect a simulated chaos job

Create a simulation first using the walkthrough, set SIM_ID from its response, and list available scenario IDs before selecting one. The run body uses the chaosRunRequestSchema fields simulationId, scenarioId and duration; the request creates an asynchronous job, not an immediate final resilience measurement. Save the returned job ID and query its results when complete.

curl -sS https://www.cloudworldmodel.ai/api/chaos/scenarios \
  -H "Authorization: Bearer $CWM_API_KEY"
# Select a scenarioId from the response, then:
curl -sS -X POST https://www.cloudworldmodel.ai/api/chaos/run \
  -H "Authorization: Bearer $CWM_API_KEY" \
  -H "Content-Type: application/json" \
  -d "{\"simulationId\":\"$SIM_ID\",\"scenarioId\":\"$SCENARIO_ID\",\"duration\":300}"
# Save the returned job.id as JOB_ID; poll status before requesting results.
curl -sS "https://www.cloudworldmodel.ai/api/chaos/jobs/$JOB_ID/results" \
  -H "Authorization: Bearer $CWM_API_KEY"

Connect through the MCP quick start

From hypothesis to real-world validation

Use modeled failure outcomes to identify likely weak points, then design a guarded real-world experiment with an owner, blast-radius limit, abort criteria and measured service indicators. Compare observed results to model predictions, update the model where they diverge and communicate the distinction between the two types of evidence. Simulated chaos does not replace production fault injection such as AWS Fault Injection Service (FIS) or Gremlin-style experiments on real systems; use those when you need live validation with safeguards. See fidelity and /accuracy for what the model covers.

/accuracy

Frequently asked questions