A green health check is not proof that failover works. It only proves that, at one sampling interval, a probe received an acceptable answer. Your load balancer can still hold dead connections, drain the wrong target, fail open to an unhealthy pool, or route fresh requests to a zone that has already lost capacity.
To learn how to test load balancer failover, treat it as a controlled fault experiment, not a checkbox in a change ticket. Define the failure, hold realistic traffic against the service, collect evidence from every layer, and measure the time between fault injection and stable recovery. If you cannot replay the test, you have an anecdote, not a resilience control.
Start with the failover contract
Before touching a backend, write down what the system is expected to do. “Traffic should fail over” is not a contract. It leaves too much room for a test that technically passes while users still see timeouts, broken sessions, or a 10-minute recovery window.
Specify the active path, standby path, failure domain, detection method, and recovery target. That may mean an active-active pool across availability zones, an active-passive pair of regional endpoints, or an L7 proxy routing around an application deployment. The mechanics differ, but every test needs an unambiguous answer to four questions:
- What failure are you injecting?
- How will the load balancer detect it?
- Which connections should survive, retry, or fail?
- What recovery time and error budget are acceptable?
Be precise about scope. A failed process, a closed listener, packet loss, a slow dependency, and a full zone outage are different events. If your health check only calls `/healthz`, killing database access may leave the target marked healthy while every meaningful request returns 500. That is not a load balancer failure. It is a health-check design failure.
Build a test that looks like production
Failover behavior under one request per second is often misleading. Connection pools stay warm, queues never build, and retry storms do not appear. Use production-shaped traffic: the same protocol mix, request paths, payload sizes, concurrency, keep-alive behavior, and geographic source distribution that matter to the service.
For L7 services, include both cheap and expensive endpoints. A cache-hit GET endpoint may recover cleanly while authenticated writes expose session affinity, token validation, or downstream saturation. For L4 services, preserve realistic connection duration and churn. A TCP load balancer may establish new flows correctly after a target loss while existing long-lived sessions hang until the client timeout fires.
Do not test only the happy request path. Include client behavior. Some callers retry immediately, some use exponential backoff, and some hold a persistent socket for hours. Those differences decide whether a brief failover becomes a minor latency spike or a self-amplifying traffic event.
A packet-level workflow is useful when the incident signature is more specific than “backend down.” Capture the traffic pattern, isolate the relevant TCP, UDP, or ICMP sequence, then replay it under authorization. Capture → chain → replay turns a hard-to-explain outage into a repeatable regression test.
How to test load balancer failover step by step
Run the test against infrastructure you own or are explicitly authorized to assess. Use a maintenance window or an isolated environment when the blast radius is uncertain. The goal is controlled evidence, not an avoidable incident.
1. Establish a clean baseline
Run steady traffic before injecting any fault. Record normal p50, p95, and p99 latency; connection success rate; HTTP status distribution; TLS handshake time where relevant; and backend request volume per target. Verify that observability is working before the failure begins.
You need timestamps from at least three places: the traffic generator, the load balancer, and the backend or application telemetry. Without synchronized time, a claimed 12-second failover can be an artifact of mismatched clocks and log delay.
2. Inject one failure at a time
Start with the simplest target-level failure. Stop the application process, block its listener, or remove it from service without changing the load balancer configuration. This validates detection and target removal.
Then increase realism. Introduce partial packet loss, elevated latency, dependency failure, connection exhaustion, or a network path partition. These tests reveal whether health checks detect degraded service before users do. They also expose the dangerous case where a target remains technically reachable but operationally useless.
For a zone or site test, remove an entire failure domain only after target-level behavior is understood. Regional failover adds DNS caching, routing convergence, certificate coverage, data consistency, and capacity questions. A regional endpoint that receives traffic but cannot absorb it has not provided continuity.
3. Keep traffic on during the transition
Do not pause the load test while you trigger failover. The transition is the test. Maintain a stable baseline load, then add a bounded burst if production incidents tend to create retries or reconnects.
Watch for the first error, the last error, and the point where latency returns to its normal envelope. Those are separate moments. A load balancer can resume successful responses quickly while tail latency remains elevated because the surviving targets are warming caches, rebuilding connections, or draining queues.
4. Validate connection semantics
Fresh connections are only half the story. Check what happens to in-flight requests, WebSockets, streaming responses, database proxies, game sessions, and other long-lived flows. Depending on protocol and architecture, some interruption is expected. The test should prove that the behavior matches the documented client contract.
Session affinity deserves special attention. If a failed target owned sticky sessions, do users get reassigned safely? If state is local, can the replacement target reconstruct it? A clean 200 response from a generic health endpoint does not answer either question.
5. Restore the failed path and observe re-entry
Recovery is another fault boundary. When the original target returns, confirm that it does not receive traffic until it is genuinely ready. Premature re-entry causes flapping: healthy, unhealthy, healthy again, while clients absorb the churn.
Test slow-start or connection draining settings if your platform supports them. A recovered target may need a staged ramp rather than an immediate equal share of traffic. Measure whether the restored pool reaches normal distribution without pushing p99 latency or error rates outside the agreed threshold.
Measure the right evidence
A useful failover result is a timeline, not a green badge. At minimum, capture fault injection time, health-check detection time, target withdrawal time, first successful request on the alternate path, final client-visible error, and stabilization time.
Pair those timestamps with request outcomes. Separate connect failures, TLS failures, upstream resets, timeouts, 4xx responses, 5xx responses, and application-level failures. Collapsing them into one availability percentage hides the mechanism you need to fix.
Also inspect capacity behavior. During failover, surviving targets carry more traffic. Confirm CPU, memory, file descriptors, connection tables, queue depth, autoscaling lag, and dependency saturation. A pool may pass a five-minute test and collapse after 20 minutes because the surviving nodes cannot sustain the new steady state.
Define pass criteria before the run. For example: unhealthy targets must be removed within 10 seconds; new requests must recover within 15 seconds; client-visible errors must remain below 0.5%; and the remaining pool must maintain p99 latency under a defined ceiling. The exact numbers depend on the service. A payment API, game relay, internal admin tool, and batch endpoint do not share the same tolerance.
Failure modes teams routinely miss
The most common mistake is testing a hard-down backend only. Real incidents are frequently gray failures: slow responses, selective packet loss, exhausted workers, dead dependencies, or a process that accepts TCP but cannot serve useful work. Health checks must be designed around the failure you need to detect, with a clear trade-off between fast detection and false eviction.
Another mistake is ignoring retries. An aggressive client retry policy can turn a small backend loss into a load spike against the surviving pool. Test with the retry behavior your SDKs, proxies, and browsers actually use. If necessary, introduce jitter, caps, circuit breaking, or request hedging limits before the next incident makes those decisions for you.
Finally, avoid treating the control plane as invisible. Configuration propagation, DNS TTLs, service discovery, route advertisements, and certificate state can all delay failover. Test the path users take, not just the path shown in the load balancer dashboard.
Turn the result into a regression test
The first run finds assumptions. The second run proves whether you fixed them. Store the traffic profile, packet chain or request script, fault procedure, thresholds, timestamps, and observed result with the service repository or runbook. Make the test callable through a token-auth API or CLI so it can run before high-risk releases and after network changes.
RETRO//STRESS is built for this kind of authorized workflow: controlled L4 and L7 traffic, packet-chain construction, PCAP-based replay, scheduled runs, and live latency and loss telemetry. The point is not more traffic. It is precise traffic with an audit trail.
Your best failover test ends with a sharper operational question: when the next dependency degrades halfway instead of dying cleanly, can your system detect the difference and move traffic before customers do?