A service can look healthy at idle, pass a basic ping check, and still shed traffic the moment real concurrency arrives. That gap is where operators ask: what causes packet loss under load? Usually, it is not one mysterious network fault. It is a finite queue, table, CPU core, link, or policy boundary reaching its limit before the rest of the stack reports an obvious failure.
Packet loss under load is a capacity signal. The hard part is locating which component made the discard decision, whether it was intentional, and whether the observed loss is actually a network problem rather than an application timeout being mislabeled as one.
Packet loss under load starts at the first constrained resource
Every packet path has buffers and limits. NIC receive rings hold packets briefly before the kernel processes them. Kernel queues hold packets before protocol handling. Firewall and NAT devices track flows in state tables. Switch ports and router interfaces use egress buffers. Applications have accept queues, worker pools, and connection limits.
Under sustained or bursty traffic, the first resource that cannot drain as quickly as it fills becomes the loss point. That resource might be at the source, on the target host, in a top-of-rack switch, inside a virtual network overlay, or at an upstream provider edge.
This distinction matters because a saturated 10 GbE link, a full Linux softnet backlog, and an exhausted conntrack table can all present as "packet loss." They require very different fixes. Increasing bandwidth will not repair a CPU-bound receive path. Raising a queue limit will not solve a firewall that cannot create new states. More retries may only amplify the overload.
The common failure domains
Interface queues, buffers, and microbursts
A link can have plenty of average headroom and still lose packets. Traffic is not always smooth. A burst from several senders can converge on one egress port in a few microseconds, exceed its buffer, and trigger tail drops before utilization graphs move enough to look alarming.
This is common in east-west traffic, storage paths, game server fleets, and load-balanced services where many clients receive a response at once. Small buffers reduce latency during normal operation but offer less burst absorption. Large buffers absorb more traffic but can create bufferbloat, where packets survive at the cost of unacceptable queueing delay.
Look for interface discards, output drops, queue depth, and latency rising ahead of loss. If loss appears in short bursts while average bandwidth remains modest, suspect congestion and buffer behavior before blaming the physical link.
NIC, driver, interrupt, and host CPU limits
Packet rate matters as much as throughput. A host may move several gigabits per second with large packets yet fail at a much lower bandwidth when traffic consists of small UDP datagrams, SYNs, or short request-response exchanges.
The receive path has work to do for every packet: DMA completion, interrupt or polling work, kernel processing, filtering, routing, socket delivery, and sometimes encryption or encapsulation. If a receive ring overruns, packets are dropped before the application sees them. If one CPU core handles a disproportionate share of interrupts or flows, the machine can lose traffic while aggregate CPU usage looks comfortably low.
Check per-queue NIC counters, RX missed errors, softnet drops, IRQ distribution, per-core utilization, and packet-per-second rates. A clean interface error count does not clear the host. The drop may be occurring higher in the kernel stack.
Virtualized infrastructure adds another layer. Virtio queues, hypervisor scheduling, vSwitch processing, noisy neighbors, and CPU steal can all delay packet handling. A VM that is adequately sized for average workload may fall behind during packet-rate spikes because it does not receive CPU time when the receive queue needs it.
Stateful devices and connection tracking exhaustion
Firewalls, NAT gateways, load balancers, and DDoS controls are not just forwarding packets. They classify traffic, allocate state, apply policy, and often inspect protocol behavior. Under connection churn, the limiting factor may be new flows per second, not bandwidth.
A full conntrack table can cause new connections to fail while established sessions continue. A SYN proxy or rate limiter may deliberately drop or challenge excess connection attempts. A load balancer may reach a backend connection cap, ephemeral port limit, or per-tenant quota. These are controlled failure modes, but they are still failures if the policy does not match expected production load.
Separate tests for established-flow throughput from tests that create new connections rapidly. If long-lived flows remain stable but new sessions fail, investigate state creation, SYN handling, NAT port allocation, and admission controls. Do not treat it as generic packet loss.
Application backpressure that looks like network loss
A server can receive packets correctly and still fail users under load. Its accept queue may fill, workers may block on a database, garbage collection may pause request handling, or an upstream dependency may slow every transaction. TCP then retransmits, connection establishment stalls, and client-side telemetry may report loss or timeout.
UDP makes this even easier to misread. If the application cannot drain its socket receive buffer, the kernel drops datagrams locally. The network delivered them to the host, but the service never consumed them. That is an application-adjacent capacity issue, not proof of a transit fault.
Correlate packet capture data with host counters and application metrics. If packets reach the interface but socket drops rise, focus on receive buffers, process scheduling, and consumer throughput. If TCP retransmissions rise without corresponding interface drops, inspect latency, application stalls, and asymmetric routing.
Why load-test shape changes the answer
A flat throughput test is useful, but it rarely reproduces the incident that woke someone up at 2:13 a.m. Real failures often depend on shape: a sudden burst, connection ramp, mixed packet sizes, fragmented requests, a specific TCP flag sequence, or a geographic path with different peering behavior.
Test one variable at a time. Establish a baseline with steady traffic, then introduce a controlled concurrency ramp. Test packet rate separately from bandwidth by varying payload size. Compare new-flow pressure against established sessions. If the service uses UDP, test realistic datagram sizes and arrival bursts rather than only smooth traffic.
The goal is not to produce the biggest graph. The goal is to identify the threshold where latency, retransmissions, queue depth, or drops begin changing together. That correlation turns a vague report into an actionable fault domain.
A disciplined way to prove the loss point
Start with synchronized timestamps. Without them, a switch counter, host metric, and client timeout can look related when they are not. Capture baseline counters across the path: source retransmits or send errors, target interface and kernel drops, state-table occupancy, load balancer metrics, and application queue depth.
Then run authorized tests against infrastructure you own or are explicitly permitted to assess. Increase pressure gradually and preserve the exact test definition. Record protocol, packet size, packet rate, concurrency, geographic source, duration, and response behavior. Repeat the same run to determine whether the threshold is stable or influenced by noisy neighbors, provider controls, or autoscaling events.
Packet capture is decisive when metrics disagree. Capture at the sender and target when possible. If packets leave the source but never arrive at the target interface, move upstream through the path. If they arrive at the target interface but do not reach the socket or application, the fault is local. If both captures show delivery but the client times out, inspect response-path congestion and application execution.
This is where capture → chain → replay workflows earn their keep. A known-bad packet sequence or connection pattern can become a regression test instead of tribal knowledge. RETRO//STRESS is built for this type of authorized validation: packet-level L4 control, L7 load, repeatable chains, and observable latency and loss rather than a slider toy that only reports requests per second.
Fix the bottleneck, then test the trade-off
The correct remediation depends on the constraint. You may spread IRQ processing across cores, adjust NIC ring or kernel backlog settings, tune socket buffers, add capacity, rebalance flows, increase state limits, alter load-balancer policy, or reduce application work per request. Each change has a cost.
Bigger queues can reduce drops while increasing tail latency. Higher conntrack limits consume memory and may expose downstream systems to more work. More aggressive rate limiting protects critical services but rejects legitimate burst traffic. Scaling out may solve worker saturation while shifting the bottleneck to a shared database or egress path.
Treat the fix as incomplete until the original test no longer fails and a neighboring failure mode has been checked. If you raised a queue, measure p99 latency. If you added firewall state capacity, test recovery after expiration and connection churn. If you scaled a service, verify that traffic distribution is actually even.
Packet loss under load is rarely random. It is the system telling you where its assumptions end. Build the test that makes that boundary visible, preserve it, and run it again before production has to teach the lesson twice.