8.4 Stale Serving, Herd Collapse, and Micro-Caching

Intermediate

What you will learn

1. proxy_cache_use_stale — serving expired content when it's the kindest thing to do.

2. proxy_cache_background_update — refresh while serving.

3. proxy_cache_lock — collapsing the thundering herd (with the 5-vs-1 receipt).

4. Micro-caching — one-second TTLs for dynamic APIs (hit ratios measured).

5. The composition: all four on one endpoint.

1. use_stale — the graceful degradation engine

By default, an expired entry is useless: the request goes to the origin, whatever its condition. use_stale overrides that default for the cases where the old copy beats the alternative:

proxy_cache_use_stale  error timeout invalid_header updating
                       http_500 http_502 http_503 http_504;

Read it as a policy sentence: "serve the expired entry when the origin errored, timed out, spoke garbage, returned a 5xx — or when an update is already in flight." The classic production posture adds http_429 (origin rate-limiting you) and, on flaky origins, http_404 (controversial — a soft-404 trade).

The /stale/ location demonstrates against an echo backend's /cache/4 (four-second TTL):

$ curl -s -o /dev/null -D - http://127.0.0.1:8800/stale/ | grep -i x-cache
X-Cache-Status: MISS          # primes the entry
$ sleep 4.5; curl -s -o /dev/null -D - http://127.0.0.1:8800/stale/ | grep -i x-cache
X-Cache-Status: STALE         # expired entry served immediately
$ curl -s -o /dev/null -D - http://127.0.0.1:8800/stale/ | grep -i x-cache
X-Cache-Status: HIT           # the background update replaced it

At T+4.5 s the entry was expired — and the client got a sub- millisecond answer anyway. The origin was never on that request's critical path. That's the resilience property: an origin outage during traffic becomes a log line, not an incident (customers see slightly-old content; alarms see the STALE ratio climb — Module 12 makes that a dashboard).

2. background_update — the invisible refresh

proxy_cache_background_update  on;

Paired with use_stale updating (and usually one proxy_cache_use_stale … updating … line): when an expired entry is served stale, NGINX fires the origin fetch as a background subrequest in the same moment. The client's second request finds a fresh entry:

T+0.0   request → entry expired → serve STALE (0.3ms) ─┐ fetch starts ─┘
T+0.1   (fetch completes, entry refreshed)
T+1.0   next request → HIT (fresh)

Poll for exactly that sequence (the tolerance loop is itself a lesson: background means asynchronous, so verification — and intuition — must poll, not assume). The staleness window collapses to one request's duration: of every N requests past an expiry, exactly one pays the stale penalty, invisibly.

The knob that tunes aggressiveness: proxy_cache_valid (or origin TTLs) sets how often the cycle runs — short TTLs mean the "one stale serving" happens more often but content stays fresher. This is the mechanism behind "1-second-fresh APIs with 1-second-old worst-case data" — micro-caching's sibling, next.

3. proxy_cache_lock — collapsing the herd

The thundering herd: a popular entry expires, fifty concurrent requests all see MISS, all fetch the origin simultaneously — the cache causes the stampede it exists to prevent. The lock fixes it:

proxy_cache_lock         on;
proxy_cache_lock_timeout 5s;    # waiters give up and fetch solo after this
proxy_cache_lock_age    10s;    # if the winner is slower than this, another may start

One request wins the lock and fetches; the rest wait, polling the cache, and are served the moment the winner stores. The deterministic experiment — five concurrent cold requests against the slow origin (2 s), with and without the lock:

# five concurrent cold requests, slow (2s) origin — with proxy_cache_lock on:
$ for i in 1 2 3 4 5; do curl -s -o /dev/null -D - http://127.0.0.1:8800/lock/ \
    | grep -i x-cache & done; wait
MISS          # exactly one — the winner fetched
HIT
HIT           # the other four waited, then were served
HIT
HIT

# the same five against /herd/ (no lock):
MISS
MISS
MISS          # all five hit the origin — thundering herd
MISS
MISS

One origin request versus five — at real scale it's one versus thousands (the day a homepage entry expires during a traffic spike). _lock_timeout bounds the pathology of a hung winner: waiters defect and fetch independently (a herd of one, then a trickle) rather than error out; _lock_age lets a second fetch start if the winner looks dead. The defaults are sane; knowing they exist is the requirement.

4. Micro-caching — dynamic content, cache semantics

The philosophy: if one second of staleness is acceptable, your origin's QPS collapses by orders of magnitude:

location /micro/ {
    proxy_pass         http://127.0.0.1:9001/;
    proxy_cache        cache;
    proxy_cache_valid  200 1s;      # the whole trick
    proxy_cache_lock   on;          # cold bursts collapse to one fetch
    add_header         X-Cache-Status $upstream_cache_status always;
}

The measurement — 30 rapid requests against a 1 s TTL:

$ for i in $(seq 30); do curl -s -o /dev/null -D - http://127.0.0.1:8800/micro/ \
    | grep -io 'x-cache-status: [a-z]*'; done | sort | uniq -c
     29 X-Cache-Status: HIT      # 1s TTL: 29 of 30 served from cache
      1 X-Cache-Status: MISS     # exactly one origin fetch

97% offload from one directive. Scale the arithmetic: a 1 000 QPS endpoint → the origin sees ~1 QPS (plus one per expiry tick per instance); a 50 ms-origin API's p99 becomes the cache's p99 (sub-millisecond). The cost is the contract you must negotiate with the product: "the data is at most 1 second old" — for dashboards, catalogs, feeds, search-typeahead, this is free money. For balances and inventory, it's a lie; don't.

5. The composition

All four techniques on one endpoint — the production shape the module has been building toward:

location /api/catalog/ {
    proxy_pass                      http://catalog/;
    proxy_cache                     cache;
    proxy_cache_valid               200 2s;
    proxy_cache_lock                on;
    proxy_cache_use_stale           error timeout invalid_header
                                    updating http_500 http_502 http_503 http_504;
    proxy_cache_background_update   on;
    proxy_cache_revalidate          on;
    add_header X-Cache-Status $upstream_cache_status always;
}

Behavior: cold bursts collapse (lock); steady state is ~100% hits with 2 s staleness; expirations are invisible (stale+update); origin errors degrade to slightly-old data instead of 502s; validators make revalidation cheap. Every line has a verified receipt somewhere in this module.

Try it yourself

1. The STALE timeline. Prime /stale/cache/4?yours, watch it expire (sleep 4.5), catch the STALE, then poll to HIT — and read the sequence in the log: one STALE, one background MISS-in-log? (Background subrequests appear in the log with the same URI — find them.)

2. Break the origin, keep the site. Point /stale/ at a dead port, reload, prime… wait — the existing entries still serve! After expiry: use_stale error keeps answering. Your cache just survived an origin outage on camera. Restore.

3. Herd, visually. Run the 5-concurrent experiment against /herd/slow and /lock/slow and diff the MISS counts — then imagine both numbers × 10 000. That's the incident you prevented.

4. Tune the micro. Set /micro/ to 5 s and re-run the 30-request burst: 29 HITs expected — now with -P30 concurrency. What's the worst status mix you can produce? (Lock keeps it at one MISS.)

Q&A

Q1. Origin is down 10 minutes; use_stale error timeout set; entries had 60 s TTLs. What do clients experience?

Q2. Why does background_update require use_stale updating?

Q3. proxy_cache_lock_timeout 5s, winner takes 8 s to fetch. Walk through the waiters' experience.

Q4. Product asks for "at-most-200ms-old data" on a 5 000 QPS endpoint. What do you offer, and what do you warn?

Takeaways

• use_stale converts origin failures and expiry moments into slightly-old answers — resilience as a policy sentence.

• background_update + updating makes refresh invisible: one stale serving per expiry, by design.

• proxy_cache_lock collapses cold/expiry stampedes to a single origin fetch — the 5-vs-1 receipt above, scalable by imagination.

• Micro-caching = short TTL + lock + stale-fallback: three lines that buy orders of magnitude on any endpoint that tolerates its staleness contract.

• The composed location (§5) is the production shape — every line earned, every behavior verified.