Latency vs throughput: the tradeoff nobody names

One checkout lane versus ten

A grocery store with one checkout lane, staffed by someone extremely fast, might get each individual customer through in 30 seconds. That's low latency - the time for one specific thing to complete.

The same store with ten checkout lanes, each staffed by someone average, might process 200 customers in the time the fast single lane processes 40. That's high throughput - the total amount of work completed over time.

Notice these are answering different questions. Latency asks "how long did this one thing take." Throughput asks "how much total work got done." A system can be excellent at one and mediocre at the other, and conflating them is one of the most common mistakes in talking about performance.

Why you can't always improve both at once

Add more checkout lanes and throughput goes up - more customers served per hour, total. But if each new cashier is a little slower than the best one, average latency per customer can actually go up even as total throughput improves. This isn't a contradiction; it's the actual tradeoff.

The same thing happens in backend systems constantly. Batching requests together - waiting to accumulate ten database writes before sending them as one transaction - improves throughput (fewer round trips, more total operations per second) but hurts latency for the first request in the batch, which now has to wait for the other nine to show up before anything happens. Caching helps both, when it works, but adds latency on a cacheCacheA copy of data kept somewhere faster to read from than its original source, so repeated requests don't have to pay the full cost every time. Deliberately allowed to be wrong or empty - a cache miss should never be treated as an error. miss while paying off in throughput on every subsequent hit.

The relationship isn't a straight line either. Push concurrencyConcurrencyMultiple tasks in progress at once, not necessarily happening at the exact same instant - like juggling several conversations by switching between them. Different from parallelism, which means truly simultaneous. up on a fixed set of servers and throughput climbs for a while, then flattens, then falls over entirely, while latency goes from barely moving to catastrophic:

Concurrent requests10

10400

p99 latency

22ms

Throughput

450/s

Queue depth

0

Well under capacity. Requests are served as fast as they arrive, nothing waits in line, and latency is essentially just the work itself.

The lesson in that curve: a system at 60% of its peak throughput is often a far better place to live than one at 100%, because the second one has no headroom left before latency falls off a cliff.

Why averages lie about latency

A system with an average response time of 50ms sounds fast. But averages hide the shape of the distribution - if 99% of requests take 10ms and 1% take 4 seconds, the average still looks reasonable, while 1 in 100 real users is having a genuinely bad experience.

This is why performance work in practice talks about percentiles instead of averages - p50 (the median), p95, p99. "p99 latency is 800ms" means 99% of requests finished faster than that, and the slowest 1% didn't - a number that actually tells you something an average conveniently hides. At real scale, that unlucky 1% is not a rounding error; it's thousands of actual users.

Where the tradeoff shows up in system design

Load balancing algorithms have to choose: send each request to whichever server responds fastest right now (optimizing latency) or spread load evenly to maximize total capacity (optimizing throughput) - and under real traffic, those aren't always the same server.

Connection poolingConnection poolingReusing a small set of already-open database connections across many requests, instead of opening and closing a new one every time - opening a connection is far more expensive than using one that's already there. is the same tension in miniature: too few connections and requests queue up waiting for one to free up, hurting latency; too many and you overwhelm the database, hurting throughput for everyone. There's rarely a setting that maximizes both at once - usually just a point where the two are balanced well enough for what the system actually needs.

The question worth asking instead of "is it fast"

Is this a system where one person is waiting right now for one answer - checking out a shopping cart, loading a page - where latency is what actually matters? Or is this a system processing a queue of work where nobody's watching a specific item complete - batch exports, log processing - where throughput is the real measure? Naming which one you're optimizing for, before touching any code, is what turns "make it faster" into an actual engineering decision instead of a guess.