Design a rate limiter

Protect backends from abusive or accidental overload. Limits may be per API key, IP, user ID, or endpoint. Decisions must be fast (microseconds to low milliseconds) and correct enough under concurrency.

Draw in this order so algorithm choice is grounded in traffic numbers, not vibes.

Rate limiting is a product decision wearing an infra costume. Nail the dimensions first.

Functional

Non-functional

Size the limiter like any other service — it sits on every request and can become the bottleneck.

Why this napkin matters: it's easy to put the limiter behind a slow DB, or do a non-atomic read-modify-write across gateways. At 50k QPS, both fail immediately. The math also justifies Redis and explains why sliding-window-log (store every timestamp) can be memory-expensive under abuse. Worth anticipating: hot API keys hammering one Redis slot, multi-region global quotas, and what happens when Redis latency spikes.

Compare a few algorithms and pick one. Draw the visual early.

Pick: Token bucket in Redis for API gateways (burst-friendly). Mention sliding window as the smoother alternative when exact-ish fairness matters more than burst UX.

Why token bucket is the usual MVP pick: APIs want short bursts (upload retries, page loads firing parallel calls) without letting a client sustain 2× the long-term rate. Fixed window is simpler to implement with INCR+EXPIRE but has the edge-burst bug you should name — that's the rejected-alone alternative. Sliding window log is fairest and most memory-hungry — rejected for MVP on cost, not correctness. Worth anticipating: how you refill tokens atomically, and whether burst capacity b equals the sustained rate or is intentionally higher.

Placement is as important as the algorithm. Wrong layer = either too late or too coarse.

Often you'll use both: coarse IP limits at the edge, finer per-user/API-key limits after authentication in the gateway or middleware.

Placement is a product decision: edge IP limits stop scrapers before auth burns CPU, but NAT and mobile carriers share IPs, so a per-IP limit alone punishes innocent users behind the same CGNAT block. Post-auth per-user limits are fairer and match billing tiers — the same reason Tatkal checks your account's quota at the booking gateway rather than after you've already reached payment. Defence in depth: coarse at the door, precise once you know who they are. Worth anticipating: internal service-to-service calls need separate, higher limits, and the limiter has to run synchronously on the request path — async rate limiting is close to useless for admission control.

Redis is the usual shared counter store because it's fast and supports atomic ops.

Token bucket needs to read tokens, refill based on elapsed time, and consume — three steps that must not interleave across gateways. A single Redis script gives you that atomicity in one round trip. The trade-off: the script has to stay short and CPU-light at 50k QPS; don't push business logic into Redis. Worth anticipating: Redis Cluster key hashing (keep related fields on one hash tag) and clock source (use Redis's own TIME, not each gateway's local clock).

With multiple gateway instances, local in-memory counters don't work — each instance thinks it has the full quota. That's like every booth counting turnout independently and declaring victory.

Local in-memory counters fail the distributed story because N gateways each allowing the full quota effectively multiplies the limit by N. Shared state fixes that but creates a dependency — which is exactly why the fail-open vs fail-closed decision matters. Exact multi-region quotas are genuinely hard, since a single source of truth means every request pays cross-region latency; a more honest design starts with per-region limits plus a coarser global budget.

Hot-key mitigations, concretely: a single viral API key or IP hammering one Redis key can bottleneck on that one key's CPU cost even when the cluster overall has headroom. Shard the counter itself — rl:{key}:{shard} across N shards with the request hashed to one shard, then sum for reporting — trading a slightly looser limit (N× burst in the worst case) for no single hot shard. Or keep a short-lived local approximate counter per gateway instance for that one key and only true up against Redis periodically, accepting the same over-allow trade the token-bucket cache mentioned earlier makes.

Redis failover mitigations, concretely: a primary failover (or a slow, GC-pausing replica) can turn your rate limiter into your outage if you don't plan for it. Set an aggressive client-side timeout (a few milliseconds, not the driver default) on the Lua call so a slow Redis doesn't stack up request latency; wrap the call in a circuit breaker so repeated timeouts trip to the pre-agreed fail-open/closed default instead of retrying against a dying node; and prefer Redis Sentinel or Cluster with replicas so failover is measured in seconds, not the outage window of a single box. The one thing to say out loud: whatever you pick, name the timeout number, not just the word "timeout".

Good rate limiters communicate clearly so clients can back off instead of retry-storming.

The Retry-After header does more than politeness — it's the thing that stops a retry storm. A client that gets a bare 429 with no guidance tends to retry immediately (or on a fixed short interval), and a thousand clients doing that at once turns one rate-limit event into a synchronized thundering herd against the same endpoint the moment the window resets. A concrete number tells well-behaved clients exactly when to come back, spread out. On the ops side, emit a counter for rate-limit hits tagged by key/route and a histogram of remaining-tokens at request time — the first tells you who's being throttled and whether a limit needs raising, the second gives you an early warning that a key is trending toward its cap before it starts erroring.

Two levers worth naming together because they trade against each other. Memory via TTL: every rate-limit key should carry an expiry roughly matching its window (a minute key TTLs in ~60s, a token-bucket key TTLs after it would have fully refilled) — without it, a small percentage of keys that go cold (churned API keys, one-off IPs) accumulate forever and your ~10MB napkin estimate quietly becomes unbounded. Local cache accuracy: caching a hot key's token count in each gateway's memory for even a few hundred milliseconds cuts Redis round trips dramatically, but every instance is now enforcing against a slightly stale count — the more instances, the more the effective limit can overshoot. The honest framing for an interviewer: TTL is close to free correctness, local caching is a deliberate accuracy-for-throughput trade you should only take once Redis, not the algorithm, is the bottleneck.

Distributed race

Two gateway pods INCR without Redis atomicity ⇒ over-allow. Use Redis INCR/Lua or a single rate-limit service. For multi-region, decide: global limit (hard) vs per-region budget (simpler).

Failure modes to mention — specific to this system

Redis goes slow, not down. A GC pause on a replica or a noisy-neighbor primary adds tail latency to every Lua call — the limiter is on the hot path of every request, so this shows up as your whole API getting slow, not as an error. Mitigation: a tight client-side timeout on the rate-limit call specifically (a few ms, tighter than the request's overall timeout budget) with a circuit breaker that falls back to the pre-agreed fail-open/closed default rather than blocking.

Redis fails over. A primary crash triggers a Sentinel/Cluster failover that takes seconds, during which writes to the old primary fail. Mitigation: replicas plus a client that retries once against the new primary — but cap that retry, since a naive unlimited retry during a failover is exactly the retry-storm pattern you warned the client about with Retry-After.

A hot key. One viral API key or IP concentrates load on a single Redis key/shard even when the cluster is otherwise idle. Mitigation: shard that key's counter across N sub-keys and sum for reporting, or accept a short local approximate count per gateway instance for just that key.

Clock skew. If any part of the logic reads a gateway's local wall clock instead of Redis's own TIME command, gateways with drifted clocks compute different window boundaries or refill amounts for the same key — one instance might refill tokens a few hundred ms early or late relative to another. Mitigation: always derive time inside the Lua script from Redis's clock, never the caller's.

Architecture sketches by level

Three progressive drawings — use the matching sketch in the interview and say the trade-off out loud.

Pros: Fast to draw — enforce early, return 429 + Retry-After. Cons: Per-instance counters don't agree across replicas — attackers multiply quota by N gateways.

Pros: Shared Redis + atomic script gives consistent limits; token bucket allows controlled bursts. Cons: Redis is a critical dependency; non-atomic GET/SET races; sliding-window-log burns memory under abuse.

Pros: Multi-dimension keys, HA Redis, and an explicit fail-open/closed policy show production judgment. Cons: Local approx caches trade accuracy for QPS; global multi-region quotas are hard — don't claim perfect global sync unless you design for it.

Sample answers to say out loud