13.1 Static Config, Dynamic Fleet — and the Discovery Landscape

Advanced

What you will learn

1. The three lifecycles of an upstream address — config, fleet, DNS — and the mismatch that breaks static pools.

2. What reload-based scaling really costs (it's not the reload) — and the discovery contract a runtime fix must meet.

3. The discovery landscape, honestly scored: DNS, Consul (both doors), Kubernetes controllers, the Plus API — one canonical table with staleness bounds and first-failure modes.

4. How ingress-nginx turns Endpoints into upstreams — and what to verify when a controller, not you, owns the config.

5. The operational checklist: staleness bound, membership in logs, two-layer health, draining, DNS on-call.

1. Three lifecycles

An upstream address has a lifecycle, and NGINX's default one is the shortest of the three parties involved:

Everything in Module 5 assumed the slowest lifecycle on purpose — a stable pool is a fine thing when it exists. The moment your fleet autoscales, blue-greens, or runs on spot instances, the config file becomes a photograph of a crowd that has already left. The classic symptoms arrive as mysteries:

• traffic to a terminated instance: connect() failed (111: Connection refused) storms after every deploy;

• new instances never receiving traffic: pool "full" of names that were never re-read;

• scaling events stalled behind humans: "page ops to reload the edge."

2. What reload-based scaling really costs

"SIGHUP is graceful!" — Module 1 proved it: zero dropped requests, new workers, old drain out. So why is reload-per-fleet-change still wrong at scale? Because the reload is the cheap part:

1. The trigger is the expensive part. Something must notice the fleet change, regenerate config, validate it, and fire the signal — a template pipeline (consul-template, confd, sidecars) with its own failure modes, its own lag, and its own 2 a.m. pages.

2. The fleet's TTL is shorter than the pipeline's. Spot instances can live 60 seconds; a watch → template → reload cycle can take 30–60 of them. You scale through the change, never catching up.

3. Reload storms. A churn-y fleet means N reloads in a minute; each re-parses the whole config and re-shares the upstream zones. It works — until it doesn't.

4. Validation gaps. A bad template output that passes nginx -t (valid syntax, wrong world) ships silently.

None of this makes reloads wrong — config changes (routes, policies, TLS) should still ride the Module 1 signal dance. The insight is separating membership (changes with the fleet — belongs at runtime) from policy (changes with intent — belongs in config). Module 13 is the membership half.

3. The discovery contract

Any mechanism that feeds NGINX its fleet at runtime must provide:

1. A name that never changes — the service identity ( svc.example.test), decoupled from the addresses behind it.

2. Fresh addresses on a schedule — some bounded staleness (TTL or poll interval) that matches the fleet's churn rate.

3. Failure honesty — when the fleet is gone, requests must fail visibly (502s, logged) rather than silently hitting ghosts; and when it returns, recovery must be automatic.

4. No action from NGINX's config lifecycle — no reload, no restart, no template pipeline.

The DNS-plus-resolver design this module builds meets all four, and Lessons 13.2–13.4 verify each clause with receipts — including the honest-failure one (502s on NXDOMAIN, self-healing on restore).

4. The landscape, honestly scored

This module implements the first row end to end — and its patterns (variable proxy_pass, zone+ resolve) are literally the same directives you'd point at Consul's DNS interface or a corporate resolver in production. Learn it once, run it anywhere DNS answers.

The shapes compose — real shops run all three at once (DNS for third parties, templates for legacy, controllers for k8s). The craft is knowing which door each fleet uses, and making Module 12's logs answer membership questions (“which addresses served this route in the last hour?”) without caring which shape fed them.

5. Consul: both doors

HashiCorp Consul exposes its catalog two ways, and both fit NGINX:

Door A — Consul as DNS ( 127.0.0.1:8600):

resolver 127.0.0.1:8600 valid=5s ipv6=off;

upstream payments {
    zone payments 64k;
    server payments.service.consul:8080 resolve;   # Consul serves the catalog as DNS
}

Point the module's resolver pattern at Consul's DNS port and the fleet's health marks flow into the catalog (consul checks) — the A-record set shrinks/grows with them. What DNS still can't carry: weights, tags (“only the v2-tagged instances”), draining.

Door B — consul-template + reload:

upstream payments {
{{range service "payments"}}
    server {{.Address}}:{{.Port}} max_fails=3;
{{end}}
}

The template regenerates the config whenever the catalog changes and fires the Module 1 signal dance. You gain everything Consul knows (tags, weights, metadata) and pay in pipeline: watch latency, template correctness, reload cadence. Modern advice: use Door A for membership, keep templates for policy — the membership/policy split from §2, applied.

6. Kubernetes: the controller does it for you

In k8s you don't hand-write dynamic upstreams at all — an ingress controller (ingress-nginx, or a Gateway API implementation) watches the API and programs NGINX continuously:

Service -> EndpointSlices -> controller syncs -> nginx upstream (in-memory)

What the controller does on your behalf — and what you should verify when you operate one:

1. Endpoint watching at second scale (faster than any DNS TTL you'd dare set).

2. Readiness gating: only ready pods enter the upstream — k8s health that DNS can't express.

3. Config generation from Ingress resources — routes, TLS, and policies become k8s objects; NGINX config is an artifact, reviewed and versioned as such.

4. Graceful rollover on its own config reloads — the Module 1 machinery, industrialized.

The operator's job shifts accordingly: less nginx.conf, more "why did the controller generate THAT" — which is why this module's fundamentals (resolution contexts, staleness, failure honesty) matter even when you never write a resolver yourself: when routing misbehaves in k8s, the debugging path runs through exactly the facts in Lessons 13.2–13.4 (what's the resolved list? how stale? what does the error log say?).

7. The operational checklist

Whatever the shape, the same questions keep dynamic fleets honest. Pin them to your dashboard:

1. Staleness bound, written down. valid=/TTL/watch-latency — and a test that proves it (the flip test in the Try it yourself section below IS this test; steal it for your CI).

2. Membership visible in logs. $upstream_addr in every line (Module 12) — "which addresses served this hour" must be a query, not an archaeology dig.

3. Health = two layers. Discovery finds addresses (DNS has no health); max_fails/quarantine retires them (Module 5); readiness gates admission (k8s). Missing a layer = the classic "DNS says it exists but it 502s" incident.

4. Draining strategy per shape. DNS: two-record overlap or app-side grace; Plus: drain:true; k8s: preStop hooks + readiness flip. Terminating instances without a drain plan is how deploys become error-rate spikes.

5. Failure honesty test. Delete the record (or empty the Endpoints) in staging: clients must see 502s with could not be resolved in the error log, and recovery must require zero human action — the same failure-and-recovery arc as the Try it yourself section below, run against your real fleet.

6. DNS itself is on-call. In DNS-native shapes, the resolver is critical infrastructure: monitor it like one (Module 12's metrics + logs apply to the DNS tier too), and remember multiple resolver addresses are queried round-robin — rotation, not load-balancing.

Try it yourself

• nginx -t && nginx -s reload — load the Module 13 example config, then watch the zone live: curl -s http://127.0.0.1:9054/records | jq

• The whole arc lives in Lesson 13.3's receipts: curl -s http://127.0.0.1:9500/probe — then flip the record via the DNS admin API and watch it follow, fail, and recover with timestamps.

• Feel the contrast by hand: curl http://127.0.0.1:9502/x before and after flipping the record via the admin API — :9502 never moves (pinned at load).

• Once the arc above has played out, extend it once: point resolver at Consul's DNS port in any environment that has one, and watch the same flip/fail/recover arc against a real catalog.

• Simulate Door B: hand-write a tiny template loop (bash + curl :9054/records → regenerate upstream block → nginx -s reload) — then compare its flip latency to the runtime door's. The delta is the pipeline tax, measured.

Q&A

Q: Why not just keep huge pools of every possible instance? A: You'd load-balance over ghosts: quarantine (Module 5) saves you from some of it, but every dead peer is still a connect attempt, a timeout budget, and a temporarily disabled line in your logs — noise that hides the real signal.

Q: Is DNS "slow"? A: DNS propagation can be; DNS answering is microseconds locally. valid= (Lesson 13.2) is NGINX's staleness dial — you choose the tradeoff between DNS query rate and how fast you follow the fleet.

Q: Where does health checking fit? A: Nowhere in DNS itself — records say "exists," not "serves." That's why Module 5's max_fails/quarantine and this module's runtime membership are complementary: discovery finds addresses, failure handling retires them.

Q: Which shape should I default to? A: DNS-native when a DNS view of your fleet exists (k8s headless Services qualify!); templates for legacy fleets with rich metadata; controllers when you're all-in on k8s. Optimize for "fewest moving parts between fleet change and NGINX knowing."

Q: Headless Services — really just DNS? A: Really. A k8s headless Service publishes A records per ready pod; NGINX's resolver pointed at cluster DNS consumes them runtime-natively. It's the lowest-tech way to run your own ingress against k8s without a controller — and a great way to appreciate what controllers add.

Q: How do I test discovery without breaking production? A: Exactly like this module: a private DNS zone you control via API, a canary door, and scripted flip/delete/restore with assertions. The Try it yourself section above is the template — port it to your staging resolver.

Takeaways

• The mismatch is lifecycles: config holds addresses for months, fleets churn in seconds. Split membership (runtime) from policy (config + reload).

• Reload-based scaling fails on the pipeline, not the signal — and the discovery contract (stable names, bounded freshness, honest failure, automatic recovery) is what any fix must meet; all four provable, all four proven in the lessons ahead.

• The landscape composes — DNS-native, template+reload, programmed — and k8s industrializes the programmed door (readiness-gated membership, second-scale sync, config-as-artifact); know which door each fleet uses, and make Module 12's logs answer membership questions shape-independently.

• Consul offers both doors: DNS interface (this module's pattern, plus health-driven record sets) and templates (full metadata, pipeline tax). Membership via DNS, policy via templates is the durable split.

• The checklist is the deliverable: staleness bound, membership in logs, two-layer health, drain plans, failure-honesty tests, DNS on-call — six lines that separate dynamic fleets that sleep from ones that page.