Designing a rate limiter

On paper a rate limiter is one piece of middleware: count requests, return 429 Too Many Requests when a caller is over the line, otherwise forward. That's why interviewers reach for it. The naive version works on a demo. Real traffic asks harder questions almost immediately. Which identity are you actually limiting? What happens once the API runs across ten servers behind a load balancer? Can a smart caller time bursts around your window boundaries? And what does the limiter do when its own dependency starts getting slow?

The whole point is to be much faster than the service you're protecting, while staying accurate enough to actually catch abuse. The hard part isn't building one globally consistent counter. It's deciding where you really need exactness, where approximation is fine, and how to roll out policy changes without melting your customers in the process.

• Functional: enforce per-API-key, per-user, per-IP, and per-endpoint limits; return 429 with Retry-After; support plan tiers; update policies without redeploying application code.
• Non-functional: add less than 5ms p99 inside a region; protect a backend serving around 100k requests per second; tolerate regional Redis failures with explicit route-level behavior; produce enough telemetry to debug false positives and abuse spikes.

Each version below fixes the failure mode that broke the version before it. The point isn't to leap to a maximal architecture in one step. It's to make the next bottleneck visible, then handle it on purpose.

Start with the naive middleware. Every API process keeps a map of counters keyed by identity:endpoint:minute. Above the threshold you return 429, otherwise you forward.

It's fast because there are no network calls. It's also only correct for one process. Run ten API workers behind a load balancer and a client gets ten quotas instead of one. Restarts and autoscaling reset the counters. And the fixed-window algorithm lets a caller double their effective rate by sending the back half of one window plus the front half of the next.

Move the counters into Redis. Every API worker increments the same key for identity:endpoint:window, and the per-key TTL handles cleanup. That kills the worst v1 problem: load balancing stops multiplying a client's effective quota.

Swap fixed windows for token buckets, implemented as a single Redis Lua script. Each quota key holds two values: the current token count and the last refill timestamp. The script refills based on elapsed time, consumes a token if one's available, refreshes the TTL, and returns the allow/deny decision along with the metadata clients actually need (remaining, retry-after). This is also a good moment to lift the limiter out of the API workers and run it as its own service behind the API Gateway. Admission gets a stable contract, and the algorithm can evolve without redeploying every backend.

The v3 limiter is accurate inside one region, but a user in Sydney shouldn't have to cross an ocean before your API can answer them. Deploy the limiter service and Redis cluster regionally. Add an edge worker for the coarse stuff: obvious bot bursts, abusive IP ranges, anything that doesn't need precise quota state to reject. Replicate policy outward as versioned snapshots, so each region enforces locally while operators still manage rules from one place.

The final version keeps the hot path local and moves the learning loop off it. The limiter emits a compact event for every allow and deny. Dashboards plot deny rate, false-positive rate, retry-after distribution, Redis latency, and which routes are currently in a failover mode. Abuse scoring reads that same stream and pushes adaptive limits back through the policy store as new versioned rules. New rules run in shadow mode first; nothing gets enforced until the shadow data looks sane. And every route has an explicit answer for what to do when Redis goes wobbly — fail open, fail closed, or fall back to a local emergency bucket.

v5 still isn't an exact global quota system. A client willing to spray traffic across all your regions can collect a regional allowance in each one until the async abuse controls notice. For protecting an API from abuse, that's a fair tradeoff for the latency you save. For billing-grade quotas, contractual limits, or anything carving up a genuinely scarce resource, you'd want reservations, reconciliation, or a separate global quota service.

It also assumes whoever calls the limiter has already worked out who is calling. Authentication, bot detection, IP reputation, and client fingerprinting all feed the limiter, but each is a separate system with its own false positives and privacy tradeoffs.

Archie's reviews walk the design through the same order a strong interview answer follows. Make the state shared. Make the decisions atomic and fair. Move enforcement closer to users. Then add the controls that stop policy changes from hurting real traffic.