Most web server load tests produce a number nobody can trust. A team chooses a concurrency value, runs a script, screenshots requests per second, and calls the result capacity. That workflow looks scientific, but it often measures a warm cache, an unrealistic request loop, or the load generator's own limits.
A useful test behaves like a statistical experiment. It has a hypothesis, controlled conditions, a defined workload, repeated observations, and an outcome that can falsify an assumption. Cache state and CDN behavior deserve special attention because they can move the test boundary from the origin server to an edge layer without changing the test script at all.
Table of Contents
- What a Web Server Load Test Is Really For
- Designing the HTTP Request Model
- The Four Metrics That Actually Matter
- Caching and CDN Behavior That Flips Results
- Running the Test So Numbers Hold Up
- Reading Percentiles and Variance
- A Real Test Scenario and Final Takeaways
What a Web Server Load Test Is Really For
A web server load test isn't primarily a contest to find the largest concurrency number. It should answer a decision-making question, such as whether a release preserves latency under a representative workload, whether an origin can sustain a forecast traffic pattern, or whether a protection layer handles an expected burst without damaging legitimate requests.
That means you need to define the experiment before opening a tool. Start with the user behavior being simulated. A static asset fetch, an authenticated API request, and a state-changing checkout operation exercise different code paths, connection patterns, caches, databases, and queues. Treating them as interchangeable requests turns a useful test into a synthetic loop.
Next, document the starting conditions. Is the application process already warm? Are database connections established? Are objects present in the cache? Does a CDN serve the response, revalidate it, or forward the request to the origin? The same request model can produce very different results under cold-cache and warm-cache conditions, so the cache state is part of the test configuration, not an incidental detail. Methodology guidance also recommends separating cold-cache and warm-cache runs, recording geography, device, and network conditions, and preserving raw results for later investigation (Web Performance Suite methodology guidance).
Define the decision before the workload
A pass or fail label is too blunt on its own. Record the response-time objective, throughput expectation, acceptable error behavior, and resource boundary that matters to the service owner. The exact threshold depends on the application and its user promise, so it should come from the system's requirements rather than from a generic benchmark table.
Also decide how much fidelity the decision requires. ApacheBench is useful for a quick, repeatable request check. Its historical model is deliberately simple, repeatedly requesting the same file with configurable concurrency and either a request count or time limit. The tool began as ZeusBench V1.0, written by Adam Twiss in 1996, was later donated to the Apache Group and renamed ApacheBench, and has been included with Apache HTTP Server since version 1.0 in 1997–1998 (ApacheBench history and behavior).
That simplicity is valuable for isolating an endpoint. It isn't enough for a journey involving authentication, changing data, varied payloads, redirects, or dependency failures. Reusing one script across local, staging, and production-like environments can also hide configuration differences. A defensible result states what was controlled, what was intentionally varied, and which conclusion the evidence supports.
Designing the HTTP Request Model
The request model usually matters more than the user-count slider. Before choosing a load tool, write down what a virtual user or arrival stream sends.
Choose requests that exercise the real path
Begin with the method mix. A GET for a small static response may spend most of its time in a fast cache or web-server path. An authenticated POST can parse a body, validate a token, acquire a database connection, update state, and return a larger response. Those are different experiments.
Include the details that influence server behavior:
- HTTP methods: Separate safe reads from writes, and distinguish cacheable GET requests from dynamic operations.
- Headers and identity: Model cookies, authorization headers, content types, accepted encodings, and tenant identifiers where they affect routing or caching.
- Payload sizes: Capture representative request bodies and response sizes. A tiny synthetic payload can understate parsing, serialization, bandwidth, and memory costs.
- Connection behavior: Test persistent connections when clients use HTTP keep-alive. Test fresh sockets separately when connection establishment is part of the question.
- Pacing: Decide whether arrivals should be driven by a target request rate or generated by a fixed number of active sessions.
A cookie-less GET against a static endpoint can make a server look dramatically stronger than an authenticated POST carrying a substantial cart payload. The difference isn't a flaw in either result. It means the tests answer different questions.
Match the traffic model to the system
A closed model maintains a set of active users or connections. When one request finishes, the next begins, so latency changes can also change the arrival pattern. An open model injects arrivals at a target rate independently of response completion. It better represents traffic that continues arriving while the service slows, but it can also create a queue rapidly when capacity is exceeded.
Don't mix these models while comparing builds. A change from fixed concurrency to fixed arrival rate can look like a performance regression even when the application is unchanged. For traffic emulation, request arrivals are expected to follow an exponential distribution, and Poisson-like behavior can be checked with the coefficient of variation of inter-arrival times (research on web-traffic emulation).
| Decision | Common Default | Realistic Choice | Effect on Results |
|---|---|---|---|
| Method | Repeated GET | Production-shaped GET, POST, and other methods | Changes application code paths and dependency load |
| Identity | No cookies or one shared token | Distinct sessions or representative tenant identities | Reveals authorization, session, and cache-key behavior |
| Payload | Minimal body and response | Captured or characterized request and response sizes | Changes parsing, serialization, memory, and transfer cost |
| Connections | Fresh sockets or arbitrary reuse | Match client keep-alive behavior | Separates network setup overhead from application work |
| Pacing | Fixed concurrency without think time | Open arrivals or closed sessions chosen for the question | Changes queue formation and observed latency |
For a deeper treatment of test design and execution choices, use this HTTP load testing guide. The important discipline is to version the request model with the application. If the endpoint, cache key, authentication flow, or payload changes, the benchmark definition has changed too.
The Four Metrics That Actually Matter
A dashboard with one impressive throughput line can lie by omission. A reliable reading combines throughput, latency, error rate, and saturation, because each metric describes a different part of the service's response to pressure.
Throughput measures accepted work, commonly as requests per second, connections per second, or transferred kilobytes per second. It doesn't tell you how much work was rejected or delayed. Latency describes the time users wait, and its distribution matters more than a single mean. Error rate identifies failed handling, including 400 and 500 responses in common monitoring conventions. Saturation connects those symptoms to a constrained resource, such as CPU, a connection pool, file descriptors, worker capacity, or queue depth (web server benchmark metrics).
Read the curves as one system
Suppose throughput stops rising while latency continues climbing. That suggests the server has reached a constraint, even if the error line remains quiet. If the connection pool is nearly occupied, new work may wait in a queue before the application starts rejecting requests. A rising error percentage then confirms that the system is no longer handling the offered workload cleanly.
The reverse can also mislead. A high request rate with low apparent latency may represent cache hits terminating at an edge, a lightweight health endpoint, or a load generator that isn't producing the intended headers and bodies. Always identify where the response was served and whether the request completed the business operation represented by the test.
Practical rule: Never declare capacity from throughput alone. Find the point where latency, errors, and resource utilization begin to change together.
Response time deserves a user-centered interpretation, not just an infrastructure label. This resource on how response time impacts UX provides useful context for connecting server timing to the experience people have in a browser or application. For broader terminology, the load testing overview is a useful reference, but the operational judgment still comes from correlating application metrics with the request model.
Caching and CDN Behavior That Flips Results
Cache state can turn an origin-capacity test into an edge-cache test without announcing the change. A warm cache may serve responses before the request reaches application workers, while a cold run forces object retrieval, serialization, database access, and response transfer. Both conditions can be valid. They aren't interchangeable.
Run the same request mix against distinct configurations:
- Cold origin: Purge or bypass relevant caches, then observe origin work during fill.
- Warm origin: Repeat deterministic requests after objects are populated, and measure the steady hit path.
- CDN fronting: Send traffic through the intended edge layer and record whether requests hit the edge or reach the origin.
Cache headers define the boundary. public allows shared caching where permitted. max-age controls freshness for clients, while s-maxage can define a different shared-cache lifetime. stale-while-revalidate lets a cache serve an older object while refreshing it in the background. Each policy changes which component absorbs traffic and when the origin pays the fill cost.
Compare the saturation points
At the origin, watch CPU, application workers, database pools, file descriptors, and network egress. At the CDN, inspect edge response timing, cache-hit ratio, egress behavior, and origin fetches. During a ramp, cache misses can create a fill wave that looks like an application regression even though warm-hit performance remains stable.
| Configuration | Bottleneck Layer | Typical p95 Signal | Error Mode |
|---|---|---|---|
| Cold origin | Application, database, connection pools, or origin network | Tail latency rises as uncached work queues | Origin timeouts, dependency failures, or upstream errors |
| Warm origin | Cache process, web workers, or transfer path | Lower and steadier tail latency until a local resource saturates | Cache or server errors under sustained pressure |
| CDN fronted | Edge capacity, cache fill, or origin fetch path | Edge hits stay stable while miss latency exposes origin limits | Misses fail or origin errors propagate through the edge |
A test that bypasses the CDN can be correct for origin sizing and wrong for customer experience. A test that leaves the cache warm can validate hit-path behavior and say little about a new-key ramp. For protection testing, keep resilience controls separate from performance claims. A practical distributed denial-of-service protection guide can help frame that boundary.
Purge or isolate state between runs, and record cache-hit ratio beside latency. Without those fields, two contradictory results may both look plausible, and the team won't know which layer produced them.
Running the Test So Numbers Hold Up
A load test is a statistical experiment, not a slider exercise. Start with a falsifiable hypothesis: which request mix should meet which latency, throughput, error, and saturation objectives under specified cache and environment conditions. “The new configuration is faster” cannot guide a decision. Set pass criteria from service requirements, not from universal thresholds.
Build the run deliberately
Model production traffic from traces or an approved approximation. Remove secrets, retain meaningful method and payload variation, and document the proportions of cacheable and dynamic requests. Choose the ramp shape that matches the question:
- Step ramp: Hold each load level long enough to observe stabilization and resource queues.
- Linear ramp: Increase pressure continuously to expose the knee of the curve.
- Spike: Apply an abrupt increase when burst absorption is the risk being tested.
Short runs often measure startup behavior rather than steady state. Established benchmarking guidance recommends a warm-up and stabilization period, with meaningful runs ideally lasting 5 to 10 minutes and at least 3 minutes as a floor before trusting the results (HTTP load-testing run-length guidance). Keep duration, ramp, pacing, and cache conditions consistent across comparisons.
Repeat every configuration. Benchmark frameworks commonly repeat concurrency levels and calculate confidence intervals. Run each configuration at least three times when practical, and treat the first run as a possible warm-up discard if process, connection, or cache state differs. Never average unlike runs.
Preserve the conditions
Environment parity covers kernel behavior, TCP settings, file-descriptor limits, worker configuration, clocks, dependency versions, and load-generator placement. Record these with raw request-level results. A client that exhausts its own CPU or sockets censors the server result.
Record the following:
- Traffic seed: Exact request mix, identities, payloads, headers, and pacing settings.
- Cache state: Purge method, cache policy, hit ratio, and whether the CDN was in the path.
- Dependency behavior: Real services, controlled fixtures, or explicitly mocked calls.
- Run metadata: Build identifier, environment, geography, network conditions, timestamps, and resource limits.
- Raw output: Individual timings and errors, rather than dashboard aggregates alone.
For teams automating these checks, load testing automation is useful only if it preserves those controls. Automation can reproduce a flawed script quickly, but it cannot make the resulting evidence valid.
Reading Percentiles and Variance
The mean latency compresses the distribution into one value. That makes it easy to compare, but it can hide a slow tail created by queueing, garbage collection, lock contention, cache misses, or a small group of overloaded workers.
Percentiles expose that tail. The median, or p50, describes the middle request. p95 describes the point below which most requests fall, while p99 focuses on a smaller but operationally important tail. A service can show a healthy median while a small subset of requests waits long enough to indicate a capacity problem.
Treat a percentile as a sample
A p99 from one short run isn't a stable truth. It's an estimate drawn from that run's request population, and its reliability depends on sample size, duration, workload stability, and independence between observations. Compare repeated runs and calculate confidence intervals when the decision requires statistical defensibility. If the intervals overlap substantially, a small apparent improvement may be noise rather than a real change.
Variance adds another signal. The coefficient of variation of inter-arrival times helps identify whether an injector is producing the intended traffic shape, while latency variability can reveal bursty queues or pauses. Research on web-traffic emulation specifically recommends checking inter-arrival behavior for Poisson-like properties using that coefficient (traffic-emulation methodology).
Diagnose the shape, not just the rank
Watch which curve moves first. If p50 remains stable while p99 creeps upward, a subset of requests may be waiting behind a queue or encountering cold keys. If p50, p95, and p99 rise together, broad resource pressure is more likely. A pause-driven tail often appears as a narrow burst, while lock contention can track a particular endpoint or dependency.
Errors may remain absent while the service has already degraded for a subset of users. That's why a green error line doesn't clear a rising tail. Inspect latency by route, cache outcome, status class, worker, dependency, and geography where those dimensions are available.
A percentile is not a verdict. It's evidence whose meaning depends on the run conditions and the variation around it.
A Real Test Scenario and Final Takeaways
Consider an API behind a CDN. A casual ramp sends a narrow, cache-friendly request loop and reports 5,000 requests per second with no errors. That result describes the tested hit path. It does not show how the origin behaves when some keys miss the cache.
A defensible experiment matches the intended workload: read and write behavior, authentication headers, payload sizes, connection reuse, and cache-key variation. Ramp gradually, repeat each configuration, and separate edge hits from origin fetches. Review p50, p95, p99, error rate, cache-hit ratio, thread occupancy, connection pools, and dependency timing together.
In a scenario like this, the analysis might surface a 3% p99 latency drift and a thread-saturation event at 3,200 requests per second, both triggered by cache misses on a subset of keys. The casual run never reached that boundary, so its clean result was valid for a narrower question than the team intended.
Apply the discipline immediately
Before the next test, write down the hypothesis and pass criteria. Freeze the request model, identify the cache condition, and decide whether the test measures origin capacity, warm-hit behavior, CDN delivery, or protection behavior. Run repeated trials, retain raw observations, and report percentiles and variance rather than averages alone.
A web server load test earns trust by challenging assumptions. The useful outcome is knowing which layer saturates first, which users experience the tail, and whether the evidence supports a production decision.
RETRO//STRESS supports authorized Layer 7 HTTP/HTTPS and Layer 4 resilience testing through a web panel, REST API, and CLI, with packet-chain workflows for repeatable traffic and geographic test selection. Visit RETRO//STRESS to assess whether its controlled, automatable workflows fit cache, CDN, and web server load-testing requirements.