13.3 Runtime Re-resolution: variable proxy_pass

Advanced

What you will learn

1. The pattern: set + variable proxy_pass + resolver.

2. The receipts: a DNS flip followed in ~1.5 seconds, no reload.

3. What you trade away: group semantics — retries, weights, least_conn, quarantine (Lesson 10.5's fact, now central).

4. Failure behavior: NXDOMAIN → honest 502s; restore → self-healing.

1. The pattern

http {
    resolver  127.0.0.1:9053  valid=2s  ipv6=off;

    server {
        listen      9500;

        location / {
            set        $backend  "http://svc.example.test:9001";
            proxy_pass $backend;      # a VARIABLE: resolved at runtime
        }
    }
}

The moment proxy_pass's target contains a variable, NGINX switches machinery: the target is computed per request (variables are runtime values), and any hostname inside it is resolved by the runtime resolver — the one Lesson 13.2 armed. The name svc.example.test stops being a config-load fact and becomes a question asked every valid= seconds.

This is the pattern that ran dynamic fleets for years — because it requires nothing but 1.7.x-era features: a resolver, a variable, and discipline. (Discipline because everything Module 4 taught about proxy_pass URI semantics still applies — the variable holds the whole target, so URI parts, slashes, and args must be assembled deliberately.)

2. The receipts

The example flips svc.example.test from 127.0.0.2 (backend b-node-dns-a) to 127.0.0.3 ( b-node-dns-b) via one HTTP call to the DNS admin API, then polls the door (real capture, valid=2s):

POST /set?name=svc.example.test&ip=127.0.0.3

t=1x0.5s: b-node-dns-a     <- cached answer still in effect
t=2x0.5s: b-node-dns-a
t=3x0.5s: b-node-dns-b     <- cache expired, re-resolved: MOVED
t=4x0.5s: b-node-dns-b
   ...                      <- stays on the new address

The access log tells the same story in upstream_addr (real lines, in order):

rsn="127.0.0.2:9001"
rsn="127.0.0.2:9001"
rsn="127.0.0.3:9001"
rsn="127.0.0.3:9001"
...

And the punchline — counted over the whole window:

emerg+reload lines: 0

No reload. No restart. No dropped config. The photograph became a video, and the error log never noticed.

3. What you trade away

The variable pattern is runtime-native, but it is not an upstream group — and Lesson 10.5's warning now carries the full invoice:

For a single logical service behind one name (a VIP, a k8s Service, a Consul service DNS name), that's often fine: the discovery layer owns multiplicity, NGINX just needs "an address that works." When you need NGINX-side scheduling across peers, you need the group form — Lesson 13.4.

The usual production shape mixes both: a map picks the service name (tenant → fleet), the variable pattern carries it, and the few pools that need real balancing stay as groups with resolve.

map $host $service_fleet {
    app.example.com    "http://svc-app:8080";
    api.example.com    "http://svc-api:8081";
    default            "http://svc-default:8080";
}
server {
    location / { proxy_pass $service_fleet; }
}

4. Failure behavior, honestly

Delete the record (real capture — record gone at T0):

t=1: HTTP 200      <- valid=2s grace: cached answer still served
t=2: HTTP 200
t=3: HTTP 200
t=4: HTTP 200
t=5: HTTP 502      <- cache expired, NXDOMAIN is now the truth
t=6: HTTP 502

[error] svc.example.test could not be resolved (3: Host not found),
        client: 127.0.0.1, server: var.example.test, request: "GET /probe HTTP/1.1"

Then the antidote — restore the record and keep polling:

POST /reset        (svc.example.test -> 127.0.0.2 again)

t=1..6: HTTP 200   <- recovery was automatic, inside valid= window
final:  :9500 -> b-node-dns-a

Design notes hiding in those receipts:

• The client sees a 502 (Bad Gateway) — the honest "I have no upstream" answer, from the Module 5 vocabulary. It is better than the alternative (serving a dead cached address forever): visible failure with a logged reason beats silent necromancy.

• The grace window is valid= working as designed — a DNS blip shorter than the cache never reaches your clients. The same knob that smooths blips delays genuine-outage detection; size it knowing both.

• Recovery is automatic because nothing ever "went wrong" inside NGINX — the resolver just started getting answers again. No health-flip, no quarantine-clear, no operator.

Try it yourself

• while true; do curl -s http://127.0.0.1:9500/probe; sleep 0.5; done — the poll loop that carries the whole arc (baseline → flip → follow → outage → 502s → recovery) against :9500; the timing is the lesson.

• Do it by hand and watch the resolver think: tail -f /var/log/b-dns.log while flipping the record via the DNS admin API — you'll see the queries, then the record change, then the follow.

• Break the map shape: add a second service to $service_fleet (the map above) and prove tenant-based routing follows DNS independently.

Q&A

Q: Does EVERY request pay a DNS lookup? A: No — the answer is cached for valid= across the worker's resolver cache (shared per zone — the runtime cache the Lesson 13.4 upstreams use). The lookup lands on whichever request's need expires the cache; the rest read the cache.

Q: Why set a variable instead of inlining proxy_pass http://svc.example.test:9001 ? A: Inlined hostnames in a literal proxy_pass resolve at config load (Lesson 13.2's first context!). The variable is what forces the runtime path — even a variable that just echoes the same string.

Q: What about fastcgi_pass , memcached_pass , etc.? A: Same trick works — any *_pass that accepts variables gains runtime resolution. grpc_pass too.

Q: TLS to the upstream — SNI still correct? A: Yes — proxy_ssl_server_name on sends the name from the variable; the certificate check ( proxy_ssl_name) should be set to the service name, not the resolved IP (Module 6 habits transfer).

Takeaways

• Variable proxy_pass + resolver = the OG runtime-discovery pattern: names resolved per request (cached per valid=), zero reloads ever.

• The receipts: traffic followed a DNS flip in ~1.5 s; emerg+reload lines: 0; NXDOMAIN produced honest 502s after the grace window and automatic recovery on restore.

• You trade group semantics — no retries, weights, quarantine, keepalive. For one-name-one-fleet shapes that's fine; for real pools, Lesson 13.4.

• The map-composition ( $host → service fleet) is the production idiom: tenant-aware routing where each fleet follows its own DNS.