8.2 proxy_cache: Keys and Storage

Intermediate

What you will learn

1. proxy_cache_path anatomy — every parameter earning its keep.

2. proxy_cache_key patterns: the default, per-variant, per-cohort, and minimized keys.

3. What's physically on disk: the two-level layout, the metadata header, the lifecycle of a write.

4. keys_zone sizing and the manager/loader processes (Lesson 1.4's promise, kept).

5. Purging in open source: key versioning and friends.

1. proxy_cache_path — the storage declaration

The storage declaration, annotated:

proxy_cache_path  tmp/cache            # where bodies live (prefix-relative)
                  levels=1:2           # two-level subdirectory hashing
                  keys_zone=cache:10m  # in-memory index: name + size
                  max_size=200m        # on-disk cap (bodies)
                  inactive=10m         # eviction clock for untouched entries
                  use_temp_path=off;   # write temp files beside their home

Parameter by parameter, with the reasoning:

levels=1:2 — cache files are named by the MD5 of the key; a single directory holding a million files becomes an inode-list disaster. levels carves subdirectories from the end of the hash: a file 476af306…84edb lands at …/b/ed/476af306…. Two levels of ~1.3 MB of directory entries keep lookups O(small) — the same problem, and the same solution, as hash sharding in Lesson 5.2.

keys_zone=cache:10m — the in-memory index (key → file, freshness, size): ~8 000 entries per megabyte. This is the thing ps's cache loader/manager processes exist to maintain (Lesson 1.4's family photo). The zone is shared across workers — any worker serves any hit. Multiple caches = multiple zones with distinct names.

max_size=200m — the cap on bodies. When exceeded, the cache manager removes the least-recently-used entries (it wakes periodically — you'll see it in ps as nginx: cache manager process exactly when caching is on).

inactive=10m — entries untouched for this long are evicted regardless of remaining TTL. This is the parameter people confuse with freshness: TTL says "may serve"; inactive says "may keep". A 24-hour-TTL entry hit every 5 minutes survives forever under inactive=10m; a 24-hour-TTL entry nobody requests is gone in 10 minutes. Cache size is governed by access patterns + inactive, not by TTLs.

use_temp_path=off — responses stream into a temp file first, then rename into place (atomic: no half-written entries ever serve). off means the temp file is created inside the cache directory, so the rename is same-filesystem — the recommended setting for every modern deployment.

2. proxy_cache_key — patterns that scale

proxy_cache_key  "$scheme$proxy_host$request_uri";      # the default, spelled out

Decode: scheme (http/https must not share entries — different origins!), $proxy_host (the upstream identity — two upstreams serving the same URI stay distinct), and the full request URI with query string. Then the variations you'll actually deploy:

Per-variant (the /cookie-key/ location):

proxy_cache_key "$scheme$proxy_host$request_uri$cookie_variant";

Live proof — same URI, two cookie populations, three requests:

variant=A  →  MISS      (entry 1 stored)
variant=A  →  HIT       (entry 1 served)
variant=B  →  MISS      (entry 2 — separate identity)

The variant classes extend past cookies: Accept-Language for content-negotiated origins, API-token tiers.

Per-cohort (the split-test composition):

proxy_cache_key "$scheme$proxy_host$request_uri$edge_bucket";
#                       Lesson 7.4's split_clients variable —───┘

Minimized (the hit-ratio discipline):

# only what changes the response belongs in the key — build it from
# the path plus the params you keep, never from the raw query string:
map $arg_v $v_key {
    default   "?v=$arg_v";    # versioned asset key
    ""        "";             # no version -> no query at all
}
proxy_cache_key "$scheme$proxy_host$uri$v_key";

Every tracking parameter left out of the key merges populations that would otherwise miss — cache-busting timestamps and per-request IDs are the same disease, each unique key a fresh miss and a fresh entry. That is also why the idiom above is a whitelist (path + the params you keep), not a blacklist: a map value is one regex substitution, and no single regex can repeatedly delete interleaved utm_* params from an arbitrary query string. (Origin-side canonical redirects are the strict version of the same discipline.)

The composite-identity rule, stated once: if a request attribute changes what the origin would return, it belongs in the key; if it doesn't, keeping it out is free hit ratio. Auditing a key against that rule is the entire code review for cache configs.

3. What's physically on disk

Look inside the store — what you find teaches the model:

$ find tmp/cache -type f | head -3
tmp/cache/b/ed/476af306746b4fe5133b3c82e5f84edb
tmp/cache/b/a0/fa93b98400e879631e01542a01f88a0b
tmp/cache/b/8b/461e2b2b49db5b90c5bafdd4232d48bb

$ ls -l …/476af306…   # a cached 1 MB slice:
-rw------- 1049290    # 1 MB body + 714 bytes of metadata

Each file = metadata header + body, written together, renamed atomically. The header carries the key, the origin's headers, the freshness arithmetic, the validators — everything the loader reads at startup to rebuild the in-memory index without re-asking origins (that's the cache loader process: one pass at boot, then it sleeps). This is why "restarting NGINX empties the cache" is false — restarts preserve on-disk entries; only changing the cache path or key format invalidates them.

4. Sizing in practice

entries  ≈ (unique keys actively accessed within inactive window)
zone MB  ≈ entries / 8000          (+ headroom: 2× is cheap)
disk MB  ≈ Σ body sizes, capped by max_size (enforced by manager)

Undersized zone = index pressure and evictions that feel random; undersized disk = the manager churning aggressively. A 10 MB zone (~80k entries) is generous for a small workload on purpose — sizing mistakes are silent, so over-provision the metadata.

5. Purging, the open-source way

proxy_cache_purge (invalidate by key pattern) is NGINX Plus — the open-source binary rejects the directive at config test. The open-source equivalents, in order of practicality:

1. Key versioning (the standard):

   map $arg_v $asset_version { default v1; }   # flip to v2 on deploy
   proxy_cache_key "$scheme$proxy_host$request_uri$asset_version";

New version string → new keyspace → old entries expire via inactive naturally. Content-addressed assets (filename hashes) get this for free — the CDN habit from Lesson 3.2.

1. Whole-cache reset: stop, rm -rf tmp/cache/*, start. Blunt, rare, honest.

2. Short TTLs: design for entries whose staleness window is acceptable, and purging becomes unnecessary (micro-caching's whole philosophy — Lesson 8.4).

Try it yourself

1. Tour the store (§3): find a cached file, stat it, and diff its size against the origin's Content-Length — that delta is the metadata header.

2. Watch atomicity. Request a large uncached file and ls the cache dir during the transfer: you'll catch the temp file (the size-shifting one) before its rename.

3. Key-design drills. With /cookie-key/ as the base: add a second cookie to the key; then remove $cookie_variant entirely and watch variants A and B poison each other (both served whichever variant stored first — the bug, demonstrated).

4. Inactive vs TTL. Set inactive=5s on a scratch cache, prime an entry with /cache/60 (TTL 60s), wait 6 s without requests, request again: MISS despite 54 s of remaining freshness — the eviction clock won.

5. The purge workaround. Implement key versioning (§5.1): deploy "v2" with a reload, confirm the population flips to MISS, and the old entries age out untouched.

Q&A

Q1. Your cache serves correct data but the disk keeps filling past expectations. Which two parameters actually govern disk usage — and why isn't TTL one of them?

Q2. Why does $scheme belong in the default key?

Q3. After a planned restart, your hit rate is 100% again; after a key-format change, it's 0%. Explain both.

Q4. Design the key for: a public API where Authorization determines rate limits (not content), and responses are identical for all users.

Takeaways

• proxy_cache_path = on-disk bodies ( levels, max_size, inactive, temp-path semantics) + in-memory index ( keys_zone) + two helper processes to keep it all coherent.

• inactive evicts on access, TTL bounds serving — different clocks, both load-bearing for sizing.

• The key is the cache: default = scheme + upstream host + full URI; add per-variant separators when content differs, strip noise parameters when it doesn't.

• Files = metadata + body, written atomically; restarts preserve caches, key-format changes destroy them.

• Open-source purging is key versioning (or blunt reset, or TTL design) — Plus's proxy_cache_purge is a convenience, not a capability gap.