How to scale an API to 1 million requests per second

There's no single trick at this scale

A million requests per second is a number that sounds like it needs some exotic, secret technique. In practice, it's almost always the same handful of ideas covered elsewhere on this site - each one individually simple - layered on top of each other correctly, with every assumption that "there's just one of this" removed from the design entirely. The hard part isn't any single piece; it's making sure none of the pieces you already know quietly becomes the one thing that can't scale.

Start by killing the easy requests before they reach real work

At real scale, the cheapest request is the one that never reaches your application code at all. A CDN serves static assets and cacheable responses from edge locations close to the user, so a huge share of traffic never touches your origin servers in the first place. Behind that, an in-memory or Redis-backed cacheCacheA copy of data kept somewhere faster to read from than its original source, so repeated requests don't have to pay the full cost every time. Deliberately allowed to be wrong or empty - a cache miss should never be treated as an error. absorbs read-heavy traffic that doesn't need to hit a database on every request - if a piece of data is read a thousand times more often than it changes, serving it from cache instead of recomputing it every time is the single highest-leverage change available.

No server can be a single point that holds everything

Every server in the fleet has to be stateless - no server can be the only place holding a user's session, a piece of in-flight data, or any assumption that "the next request from this user comes back to me." The moment that's true, requests can be freely distributed across as many server instances as needed, behind a load balancerLoad balancerA component that sits in front of multiple servers and distributes incoming requests across them, so no single machine gets overwhelmed and a crashed instance doesn't take the whole system down., and instances can be added or removed based on real-time load without anything breaking.

The database becomes the real bottleneck, so it gets split too

Once the server layer is horizontally scalable, the database is usually the next wall you hit - a single database instance has a real ceiling on connections and write throughputThroughputThe total amount of work a system completes over a given period - requests per second, jobs processed per hour. Optimizing for throughput can sometimes make individual latency worse, and vice versa., no matter how powerful the machine is. The standard answer is shardingShardingSplitting a single large database into multiple smaller pieces, each holding a subset of the data, so no one machine has to hold or serve all of it. Usually split by some key, like user ID or region.: splitting data across many database instances by some key (often user ID or a similar natural partition), so no single database has to hold or serve all of it, combined with consistent hashingConsistent hashingA way of mapping keys to servers so that adding or removing one server only reshuffles a small fraction of keys, not nearly all of them. What makes it practical to grow or shrink a cache or shard set without a mass migration. so that adding more shards later doesn't require reshuffling nearly all the existing data.

Not every request needs an answer right now

Some work genuinely doesn't need to happen synchronously inside the request-response cycle at all - sending a notification, updating an analytics count, generating a report. Pushing that work onto a message queueMessage queueA component that lets one part of a system hand off work to another without waiting for it to finish immediately - the sender moves on right away, and a separate worker processes the job whenever it gets to it. and processing it asynchronously means the APIAPIA defined way for one piece of code to ask another to do something, without needing to know how it happens internally. Not a specific technology - a function signature, a library's exports, and a REST endpoint are all APIs. can respond fast and defer the actual work to a fleet of background workers that scale independently from the request-serving layer, and a temporary backlog in the queue degrades gracefully instead of taking down the whole API.

Rate limiting protects the system you just built

None of the above matters if one misbehaving or unexpectedly popular client can still overwhelm the system - which is exactly why rate limitingRate limitingDeliberately capping how many requests a client can make in a given time window, to keep a shared system fair and stable instead of letting one client's traffic degrade it for everyone else., enforced globally across the whole fleet rather than per-instance, is part of the design and not an afterthought bolted on later. At this scale, fairness between clients is a reliability requirement, not just a fairness one: one client's runaway traffic degrading service for everyone else is itself an outage.

The actual skill being tested

None of these ideas - caching, statelessness, sharding, async processing, rate limiting - are individually advanced. What separates a system that hits a million requests per second from one that falls over at ten thousand is whether every one of these is actually in place, correctly, everywhere it needs to be, with no quiet exception left over from an earlier, smaller version of the system.