We stopped guessing what “accurate” meant

Cloud World Model is a simulator. It predicts how a cloud setup would behave—cost, speed, and errors—without asking you to provision the real thing.

To earn trust, we publish an accuracy page: side-by-side numbers for what we simulate and what we use as a real-world reference. The benchmark is a standard three-tier AWS application: an Application Load Balancer, two mid-size application servers, and one database.

The number we used to call accuracy

For a long time, the page showed about 95% overall. One scenario looked especially bad. Under a sudden traffic spike called “Burst,” the old reference said that 2% of requests would fail, while our simulator said about 10%.

Those 2% and 10% figures are now retired comparison values. They are part of the history of why we changed the benchmark, not current owned measurements.

The reference was not a test we ran

That 2% did not come from our own load test. It came from reading AWS documentation about how many database connections a database size usually allows. Useful background, but not the same as running this application, sending it traffic, and watching the resulting metrics.

Meanwhile, the page sounded as if every value had been carefully measured. That was too strong a claim. We could have tuned the simulator until Burst looked better, or we could measure the real setup and let the page follow the data.

We measured.

What “measure” meant for us

We built a small public program called cwm-bench. It runs the same canonical architecture used by the accuracy page: an AWS Application Load Balancer routing to 2 × m5.large instances and a db.r5.large MySQL Single-AZ database in us-east-2. The campaign results and the exact source revision are checked in so someone else can inspect what we did.

Each workload includes a 5-minute warmup and a 15-minute steady window. We wrote the rules down before looking at the score:

• Fit the model on ordinary traffic levels first: idle, normal, and peak.
• Keep the large Burst spike as a holdout that we did not train on.
• Do not chase a prettier overall percentage.
• Say clearly which numbers are owned observations, which are public price-list values, and which remain documentation-backed.

What the holdout showed

Under the canonical CRUD workload at Burst, owned measurements showed approximately 0% CRUD errors. Goodput averaged about 875 requests per second against a 1,000 RPS target.

That throughput number needs its timing attached to it. Because the result averages the 5-minute warmup and 15-minute steady window, it is about 87.5% of the target over the full run. It is not evidence that the application was secretly broken, and it is not a claim about steady-state capacity beyond this benchmark window.

The database was nowhere near its documented connection ceiling on this workload. The old “2% because the database ran out of connections” story did not match what we observed. When we intentionally forced a CPU-heavy path, the machines did saturate; they simply were not saturating on the everyday traffic mix used for this scorecard.

What changed for users

The live accuracy page now describes the provenance of each part of the comparison. The current overall score is roughly 97.4%, but the score is intentionally a mixed scorecard:

• Throughput and errors for the canonical AWS setup use owned cwm-bench observations, including the Burst holdout.
• Cost comes from the public AWS price list. It is provenance against list prices, not a claim that the benchmark measured a cloud bill.
• Latency and blended CPU remain documentation-backed on the scorecard. We do not present those fields as fully owned measurements.

The simulator now uses the owned throughput and error results for this canonical path. Burst remains a holdout check rather than a knob we turn to inflate the score. The higher percentage was a consequence of correcting the reference and labeling the evidence, not the goal.

What we are still honest about

We have not finished owning every metric. End-to-end speed and some CPU views are still estimated from documentation, and the page says so. The fit/holdout contract also keeps later-day and second-region runs as future checks; this post does not present those artifacts as already published results.

We are not asking anyone to trust a single percentage. We are asking them to trust the process: measure when we can, label what we cannot, and do not rewrite the data to flatter the product.

Read the live accuracy page →Inspect the pinned cwm-bench campaign revision →