14.4 Operating a Mini Ingress Controller

Advanced

What you will learn

1. The controller loop, decomposed: watch → render → validate → reload — and why each step exists.

2. The controller implementation: services.json + gen-config.sh + nginx -s reload.

3. The receipts: scale-down, weight change, new deployment — each one a manifest edit away.

4. What the real controllers add on top (and what they deliberately don't).

1. The loop, decomposed

Every ingress controller — the miniature built in this lesson and the production giants — is the same four-step cycle:

   ┌──────────────────────────────────────────────┐
   │ 1. WATCH     desired state changed?          │
   │ 2. RENDER    manifest -> nginx.conf          │
   │ 3. VALIDATE  nginx -t on the render          │
   │ 4. ROLLOUT   graceful reload (SIGHUP)        │
   └──────────────────────────────────────────────┘

Each step answers a failure mode:

• Watch bounds staleness (Module 13's contract: how fast does the fleet's truth reach NGINX?).

• Render is pure function from desired state — no side-channel edits, no "just this once" hand-patching.

• Validate is the gate that keeps a broken desired state from becoming a broken data plane (a failed render changes nothing — the running config keeps serving).

• Rollout is Module 1's graceful reload: new workers, old drain, zero drops.

The miniature's division of labor is honest about which parts are hard: rendering and validating are ~40 lines of shell; watching is the part production controllers spend their complexity budget on (API watches, leader election, change coalescing).

2. The implementation

The manifest — the "cluster state," your one source of truth:

{ "services": [
    { "name": "svc-app", "path": "/app/",
      "servers": [ {"address": "127.0.0.1:9001", "weight": 1},
                   {"address": "127.0.0.1:9002", "weight": 1} ] },
    { "name": "svc-api", "path": "/api/",
      "servers": [ {"address": "127.0.0.1:9003", "weight": 1} ] } ] }

The renderer ( gen-config.sh — a ~40-line shell script: read the manifest, emit one upstream per Service, one server line per endpoint, one location per path rule, plus the healthz surface) turns it into config and stamps the artifact contract on top:

# GENERATED by gen-config.sh from services.json — DO NOT EDIT BY HAND.
# (The config file is an artifact. The manifest is the source of truth.)
# Rendered: 2026-09-23T01:52:22Z · services: 2
...
    upstream svc-app {
        server 127.0.0.1:9001 weight=1;
        server 127.0.0.1:9002 weight=1;
    }

Then it validates the render against a scratch prefix — a fresh temp tree, so validation never depends on live state — and refuses to write on failure. nginx -s reload performs the rollout. You are the watch step: edit the manifest, run the loop. (§3's receipts are exactly this loop, replayed one manifest edit at a time.)

3. The receipts — a deployment story

Chapter 1: first render, baseline traffic (8 requests to /app/):

      5 b-node-1
      3 b-node-2        <- RR across both endpoints

Chapter 2: a pod goes NotReady. One jq edit removes 9002 from the manifest; re-render; reload. The config diff is one line gone:

 upstream svc-app {
     server 127.0.0.1:9001 weight=1;
-    server 127.0.0.1:9002 weight=1;
 }

and the traffic (4 requests after the sync):

      4 b-node-1        <- removed endpoint receives NOTHING

No config surgery, no NGINX-side failover, no quarantine — membership was the whole fix. That's Lesson 14.3's readiness-as-membership, performed by hand.

Chapter 3: weights change (HPA-shaped). Manifest edit: weight: 3 on 9001. Re-render, reload, 8 requests:

      6 b-node-1
      2 b-node-2        <- 3:1 honored by the rendered upstream

Chapter 4: a new deployment. Manifest gains svc-admin on /admin/; re-render; reload:

    /admin/ -> b-python     <- new path routes to the new service
    /app/   -> b-node-1     <- existing paths unaffected

Four chapters, zero hand-edited nginx.conf, one source of truth, every change reviewed as a manifest diff plus a config diff — the review artifact production teams live by.

4. What real controllers add (and don't)

The miniature deliberately omits, in rough order of how much they matter in production:

What they don't add — because the miniature already has it — is the discipline: validate before rollout, treat config as artifact, one source of truth. When a controller fails in production, it usually fails by violating one of these (a snippet that skips validation, a hand-patched nginx.conf on one node, a fork in the render path) — which is why this lesson exists: run the small one, recognize the big one's failure modes instantly.

Try it yourself

• bash gen-config.sh && nginx -s reload — one full iteration of the §2 loop; the receipts in §3 are this loop replayed chapter by chapter (baseline, scale-down, weights, new service — plus the deliberate breakage below).

• Do a chapter by hand: edit services.json (add an endpoint with a funny weight), bash gen-config.sh, nginx -s reload, then grep upstream -A3 /etc/nginx/nginx.conf — see your edit, rendered.

• Break the render on purpose: put a weight: banana in the manifest — the generator's nginx -t gate refuses, the running instance never notices. That refusal is the difference between a failed pipeline and an outage.

Q&A

Q: Why not render per-location include snippets instead of one file? A: Both exist in the wild: one-file renders diff cleanly as units; snippet-per-ingress scales team ownership better (and maps to conf.d/ includes, Module 1's layout). The miniature uses one file for reviewability; the contract is identical either way.

Q: What's the sync latency of a real controller? A: Typically low single-digit seconds from Endpoint change to reloaded config — faster than any DNS TTL you'd run in Module 13's pattern. The receipt shape (traffic follows the manifest) is the same; only the watch mechanism differs.

Q: Can I run this pattern WITHOUT Kubernetes? A: Yes — that's the point. Any source of truth (Consul, a database, services.json in git) + this loop = a self-service routing platform. Kubernetes just standardizes the API and the watch.

Q: Where does this leave Module 13's DNS pattern? A: As the zero-controller alternative, and as the mechanism under some controllers. Choose DNS when you want no pipeline at all; choose a controller when you want health gating, TLS automation, and policy objects. Lesson 14.5 makes the choice explicit.

Takeaways

• The controller loop is four steps — watch, render, validate, reload — and each step exists to bound a specific failure (staleness, drift, bad config, drops).

• Our 40-line implementation moves real traffic through real deployments: scale-down, weight change, and new-service receipts all came from manifest edits, never config edits.

• Config is an artifact with a DO-NOT-EDIT header; the manifest is truth; the reviewable unit is manifest-diff + config-diff.

• Production controllers industrialize the watch and the platform integration — but the discipline (validate first, one truth, artifact config) is the miniature's curriculum, and its absence is how controller incidents are born.