Adding App Servers on AWS: Measured P95 at 100–300 RPS

When we published Cloud Pricing Trends, September 2026, Aleksandr Kubarskii asked a fair follow-up on Product Hunt: cost is one side of it, but what happens to latency, and when does adding app servers stop helping?

Instead of answering with model output, we measured it on real AWS infrastructure. At 300 requests per second, measured p95 latency fell from 94.7 ms with one app server, to 38.2 ms with two, to 17.9 ms with three. The third server brought p95 down to about a fifth of the one-server figure.

One difference from September: this benchmark uses the same AWS architecture as the September post, under the typical workload instead of lean, at 1, 2 and 3 app servers. Typical is a different request mix, so these latency measurements don't directly accompany September's cost results.

This post covers what we measured, what each replica costs at list price, where our model stood before this work, and what we still haven't measured.

The setup

• Stack: Application Load Balancer, 1, 2 or 3 m5.large app servers, and one db.r5.large MySQL database, in us-east-2
• Load: the "typical" workload at 100, 200 and 300 total RPS
• Measurement: p95 measured client side by k6 on a separate c6i.xlarge load generator, not at the load balancer
• Duration: each rung was a 5-minute warmup ramp plus 15 minutes steady, about 20 minutes, the same at every server count. The p95 covers the whole rung.
• Repetition: each value is the median of 3 independent reps, spread over October 10–12, 2026

The measured results

*This cell was noisy. Its three reps were 28.5, 57.6 and 40.3 ms.

At 100 RPS, server count barely matters: every configuration lands between 7.9 and 9.2 ms. The differences show up as load rises. One server goes from 8.6 ms to 94.7 ms between 100 and 300 RPS. Three servers go from 7.9 ms to 17.9 ms over the same range.

Note that app CPU on the single server was only 29.4% at 300 RPS. Latency climbed well before CPU looked busy, so CPU alone is a poor signal for when to add a replica on this workload.

P95 against a 50 ms target

To make the table easier to read, take a 50 ms p95 target. This is illustrative, not a recommendation. Your target depends on your product.

In this campaign, only one of the nine cell medians exceeds it: one server at 300 RPS (94.7 ms). One individual rep of one server at 200 RPS (57.6 ms) also exceeded it, even though that cell's median was 40.3 ms. Two servers at 300 RPS come in at 38.2 ms, and three come in at 17.9 ms.

We also have one higher-load point from an earlier cwm-bench campaign, typical-v1: two servers at 500 RPS measured p95 88.2 ms with 0.0103% errors. It's a single two-server point from an earlier code version, kept separate from the table above. At 500 RPS it missed the 50 ms target. We didn't measure one or three servers at 500 RPS, so we can't say how a third server would change that point.

What each replica costs

Latency and cost are separate questions, so here's the cost on its own. These are list-price estimates for fixed infrastructure, not bills, using 730 hours per month:

Rates are us-east-2 on-demand list prices from AWS's EC2, RDS for MySQL and Elastic Load Balancing pricing pages.

Each extra app server adds about $70/month. These figures exclude egress, ALB LCU charges, app-server EBS volumes and database storage. Comparing data transfer costs would need consistent served traffic, payload sizes and network assumptions. The September post covers how traffic moves egress in a simulation.

Put together: at 300 RPS, going from one to two servers costs about $70/month more and took measured p95 from 94.7 to 38.2 ms. Going from two to three costs another $70/month and took it from 38.2 to 17.9 ms. Whether the second step is worth it depends on your target. At 300 RPS, the two-server median was below the illustrative 50 ms target.

Where our model stood, and where it stands now

We want to be direct about this. Before this work, the engine predicted 11–18 ms p95 at 300 RPS for one to three servers; measured was 18–95 ms. The predictions barely tracked server count. The accuracy page carried a caveat saying CWM didn't yet predict how p95 changes with server count.

That's why this post uses measured data, not simulation output.

Since then, engine 1.2.21 ships a typical p95 model that passes this campaign's preregistered holdout, with a median score of 87.6. That's the median of nine p95-only cell scores, not an overall simulator accuracy. Each cell scores 100 − 100 × |measured − predicted| / measured, so 100 is exact and each 1% of error costs a point.

At 300 RPS, those cells scored 87.6, 89.7 and 68.8 on 1.2.21, compared with 19.1, 33.5 and 61.5 on the previous engine. The three-server prediction at 300 RPS still overestimates measured p95 by about 31%, so read the median score alongside the individual results. Two limits apply:

• It's validated only for this AWS topology, the typical workload, one to three m5.large app servers and 100–300 RPS. Outside that scope, treat modeled p95 as unvalidated.
• The lean model wasn't recalibrated. This campaign tested the typical workload only, and database saturation is untested.

The accuracy page now includes the typical-p95-v1 holdout card.

What's still open: the database breakpoint

Aleksandr also asked where the database becomes the bottleneck. We haven't measured that yet. In this campaign, DB CPU stayed at or below about 28% with no connection errors. This campaign did not establish a database saturation point.

A saturation benchmark that pushes this stack until the database is the limit is next. We'll publish it the same way: preregistered, measured, and with the model compared against it.

Reproduce it

The full results, per-rep data and method are in cwm-bench PR #21. The preregistration lists the scoring rules, which were fixed before the runs. To simulate the same stack yourself, describe ALB, 1–3 m5.large and db.r5.large at 100–300 RPS on the typical workload, then call Cloud World Model over MCP or REST before deploying or changing infrastructure, keeping to the validated scope above.