7.2 Rate Limiting: The Leaky Bucket, Measured

Intermediate

What you will learn

1. The leaky-bucket model — what fills, what leaks, what overflows.

2. limit_req_zone vs limit_req — creator and consumer (again).

3. The three-knob semantics: rate / burst / nodelay — with the measured evidence for each combination.

4. Choosing the key: IP vs API-token vs header vs composite — and the spoofing risks.

5. Production posture: status codes, Retry-After, dry-runs, and the logs that make it all defensible.

1. The model

One picture, the whole module:

                      ┌───────────────────────┐
 requests pour in ──▶ │   the bucket          │ ──▶ drained at RATE
 (bursty, unfair)     │  (per KEY)           │      (smooth, fair)
                      └──────────┬────────────┘
                                 │ overflows when fuller than BURST
                                 ▼
                            429 / 503
                          (refused, logged)

A bucket per key (usually a client identity). Requests fill it; time leaks it at exactly rate; a request arriving when the bucket holds more than burst is refused. Everything else about limit_req is this picture plus one decision: what happens to requests that are inside the burst allowance — queued ( delay, the default) or admitted instantly ( nodelay).

2. Creator and consumer

# http level — CREATE the shared state (Lesson 1.4's blackboard):
limit_req_zone  $binary_remote_addr  zone=reqs:10m  rate=5r/s;
#                └─ the key           └─ name+size  └─ the leak rate

# server/location level — CONSUME it:
limit_req       zone=reqs  burst=10  nodelay;
limit_req_status 429;               # refuse with 429, not default 503

The zone syntax decodes as: key expression, zone name, memory budget (10m ≈ 160k keys), leak rate. Why $binary_remote_addr and not $remote_addr ? The binary form is fixed 4/16 bytes vs. up to 15/45 textual — smaller keys, more keys per megabyte, faster hashing. Same identity, cheaper storage: the standard choice unless you're keying on something else.

One zone, many consumers: /rl/, /rl-strict/, and /api/ could all point at zone=reqs — they'd share buckets (affecting each other) — or at separate zones for independent policy. The example deliberately shows both: reqs shared by two locations, delayz/ peruser isolated — and watch for the classic bug where one test’s requests pollute the next test’s bucket, making a "delayed" bucket look broken. Zone sharing is a policy decision, not an accident.

3. The three knobs — measured, not asserted

Knob set 1: rate=5r/s burst=10 nodelay — the credit card

The /rl/ location. Fourteen rapid requests, real capture:

$ for i in $(seq 1 14); do
    curl -s -o /dev/null -w '%{http_code} ' -H 'Host: edge.example.test' \
         http://127.0.0.1:8700/rl/
  done
200 200 200 200 200 200 200 200 200 200 200 429 429 429

Read it like a bank statement: eleven accepted (the sustained slot + the burst-10 overdraft, admitted instantly — nodelay), then refusals until time leaks the bucket back down. This is the API default posture: generous to bursts, absolute about the ceiling.

Knob set 2: rate=5r/s (no burst) — the turnstile

Back-to-back requests, second one refused:

$ curl …/rl-strict/ → 200
$ curl …/rl-strict/ → 429        # instantly; no overdraft exists

Login endpoints, OTP sends, password resets — anywhere a second immediate attempt is inherently hostile.

Knob set 3: rate=5r/s burst=10 (NO nodelay) — the queue

The default behavior nobody tries: burst requests are held and released at rate intervals. Six rapid requests, real timing per request:

$ for i in 1..6: curl -w '%{http_code}@%{time_total}s ' …/rl-delay/
200@0.001s  200@0.194s  200@0.194s  200@0.192s  200@0.194s  200@0.193s

Every request succeeded — none refused — and the bucket spaced them ~200 ms apart (1 ÷ 5r/s). This is fairness as UX: legitimate clients feel mild latency; nobody gets an error. The cost: each queued request occupies a worker (Lesson 1.4's event loop holds the timer) — which is why huge bursts without nodelay can tie up workers, and why nodelay became the common recommendation.

Knob set 4: the key itself — ?user= buckets

limit_req_zone  $arg_user  zone=peruser:10m  rate=1r/s;
location /api/ { limit_req zone=peruser burst=2 nodelay; … }

Alice's rapid-fire sequence vs Bob's, live:

$ curl …/api/?user=alice → 200 ; again → 200 ; again → 200 ; again → 429
$ curl …/api/?user=bob   → 200          # Bob's bucket: untouched by Alice

The key is the policy. Per-IP protects infrastructure; per-token protects fairness among identified users (and survives NAT — the Module 5 ip_hash lesson's mobile problem, solved). The composite choice from Lesson 7.1 ( map choosing the key) unifies both worlds.

4. Production posture

Status code: limit_req_status 429 (Too Many Requests) — the honest code; the default 503 makes clients (and monitors!) think you're down. Pair with add_header Retry-After 5 always; in the refusing location, or (the polished version) map the zone state to a computed value.

Multiple zones stack: limit_req zone=a burst=5; limit_req zone=b burst=10; — both evaluate; the strictest wins. Classic shape: a broad per-IP zone at http-inherited level + a tight per-endpoint zone inside location = /login.

limit_req_log_level turns refusals into warnings/errors for alert pipelines without touching the access log (which records $limit_req_status = REJECTED — the Lesson 7.1 log format carries it, one grep away).

Dry-run mode: limit_req_dry_run on; (added 1.17.1 — every current branch has it) logs what would be refused without refusing it: $limit_req_status reads DELAYED_DRY_RUN / REJECTED_DRY_RUN for the would-be verdicts, one grep away — the safe way to calibrate rate / burst on production traffic. One boundary to know: limit_req_status accepts 400–599 only — never 200 — so a gentler refusal means a real status code (429 is the canonical choice; staging math — offered-rate distribution × target refusal rate ≈ (rate + burst) — is the fallback when you cannot dry-run at all).

Try it yourself

1. Reproduce all four knob-sets (§3) — including the timing capture: curl -w '%{time_total}' through /rl-delay/ is the leaky bucket, visible.

2. Count your own 429s. Hammer /rl/ 30 times and tally grep -c ' 429 ' /var/log/nginx/access.log — then find your bucket=REJECTED lines.

3. Zone-sharing experiment. Fire 12 rapid at /rl/ then immediately at /rl-strict/ — the second location's refusal is the shared bucket talking. Now point /rl-strict/ at delayz and repeat: independence restored.

4. Break the key. Rate-limit /api/ as Alice until 429, then switch to ?user=alice2 — fresh bucket, no refusal. You've just demonstrated §3's trust warning with your own hands.

Q&A

Q1. rate=10r/s burst=20 nodelay — a client sends 40 requests in one second (then stops). How many succeed? When can they try again?

Q2. Why would anyone remove nodelay deliberately?

Q3. Your API sits behind a mobile gateway where thousands of users share CGNAT IPs. Per-IP limiting is punishing them — fix the key.

Q4. Zone A (rate 5r/s) is consumed by /search/ with burst=10 nodelay; zone B (rate 1r/s) by location = /login inherited from http. A request to /login — which applies?

Takeaways

• One model: per-key bucket, filled by requests, leaked at rate, overflow past burst refused. Everything else is knobs on this picture.

• nodelay = instant overdraft (API default); no- nodelay = queue at rate (fairness UX, worker cost); no burst = turnstile (hostile endpoints). §3 measured all three.

• 1 + burst requests pass in an instant — the off-by-one to remember when sizing.

• The key is the policy: IP protects infra, identity protects fairness — and client-supplied keys are attacker-supplied buckets.

• 429 + Retry-After + $limit_req_status in logs (and dry-run on mainline) turn a blunt instrument into a defensible one.