14.5 Choosing Your Ingress

Advanced

What you will learn

1. The candidates, honestly scored.

2. The decision procedure — four questions, in order.

3. Fleet operations: versioning, testing, and the change pipeline that keeps 200 NGINXes boring.

4. The graduation path: when your platform outgrows any single choice.

1. The candidates

Two honest notes the table can't carry:

• ingress-nginx's future: the community project's maintenance story has been turbulent; check its current status before standardizing on it in 2026 — the skills transfer regardless (it's NGINX + the Lesson 14.4 loop), but the support calculus belongs in your decision.

• Gateway API is where the ecosystem is investing: role separation (infra owns Gateways, app teams own HTTPRoutes), cross-implementation portability, and first-class fields for the policy that used to ride annotations. New multi-team platforms should evaluate it first, not last.

2. The decision procedure

Four questions, in order — stop at the first decisive answer:

1. Does your platform already dictate one? A managed k8s with a blessed ingress (EKS/GKE/AKS offerings) or a company-standard Helm chart: adopt it. Fighting the platform's choice to save one feature is a tax paid forever. (This decides most real cases.)

2. Do you need runtime edge control — drain, active health, per-server metrics? Yes → F5's Plus controller (or Plus itself behind any controller). No → OSS family.

3. Multiple teams + multiple implementations or multi-protocol routes? Yes → Gateway API shape; pick the implementation by engine preference and ops maturity. No → classic Ingress is fine.

4. Do you need Envoy-shaped things? (extensive L7 policy DSL, hot restart without connection loss, xDS ecosystem) Yes → an Envoy Gateway. No → stay NGINX — this entire series' skills apply directly.

And the meta-rule: the controller's engine matters less than the contract around it. Whichever you choose, the Lesson 14.4 discipline (manifest is truth, config is artifact, validate before rollout) is portable across every cell of that table — it's also the thing that makes changing your mind later survivable.

3. Fleet operations — keeping 200 NGINXes boring

Whether it's 200 pods of ingress or 200 VMs at edges, fleet NGINX is an operations problem before it's a configuration problem. The boring-fleet checklist:

Versioning. One blessed version, canary the upgrade (Module 7's split_clients is literally the traffic-shaping tool for canarying an infra version). Track CVEs for nginx + modules + base image together; Module 15's build discipline is what makes "rebuild and roll" a routine instead of a project.

The change pipeline. Every change rides the same path:

manifest/PR -> CI: nginx -t on render -> staged rollout -> verify
                                   |                        |
                          (Lesson 14.4's gate)   (Module 12's golden diff:
                                                 before/after histograms & ratios)

The golden diff deserves its bullet: no config change "should be invisible" — it must prove invisibility (status histograms, cache HIT-ratios, p95s unchanged) or show exactly the intended delta. Module 12's jq receipts are the same tool production SREs point at staging traffic.

Observability parity. Every instance ships the same log schema (Module 12) and the same metrics; a fleet dashboard that can't answer "which edge, which upstream, which change" within minutes is a fleet you're flying blind. The request_id join key (Lesson 12.2) crosses tiers; make sure it survives into your ingress choice's forwarded headers.

Blast-radius hygiene. Roll by anti-affinity (never all ingress pods on one node), reload after readiness, and keep worker_shutdown_timeout inside the pod's termination grace (Lesson 14.2) — a fleet-wide simultaneous reload is an outage with a boring name.

Config hygiene at scale. The same favicon rule as Module 11's if: the policy surface you expose to app teams should be small, validated, and boring. Annotations/HTTPRoute fields they can touch: allow-listed. Snippets: probably none.

4. The graduation path

Needs evolve; so should the topology, in this order:

1. Single front door (Lesson 14.1 Shape A) — one NGINX, one config, one team. Outgrown when teams queue for config merges.

2. Ingress in-cluster — routing becomes self-service; platform owns the controller, app teams own routes. This is home for most.

3. Tiered + Gateway API — multi-team, multi-protocol, or hybrid-estate shops; portability starts paying.

4. Mesh/service-tier control — when east-west needs the same policy rigor (mTLS everywhere, per-route auth), the sidecar/gateway service tier grows its own NGINX-or-Envoy fleet — and Lesson 14.1's trust-boundary questions apply again, one tier deeper.

Each step is additive, not a rewrite — because every step kept the contract: config as artifact, one source of truth, validate before rollout, observable by default. That's the real graduation criterion: the discipline, not the software.

Try it yourself

• You've built the miniature that every candidate implements — extend it once: add a render --dry-run mode to gen-config.sh that prints the diff without reloading. You have just built the PR-preview that makes fleet changes reviewable.

• Rehearse the golden diff: capture Module 12-style histograms before/after a weight change in services.json; the delta should show exactly the distribution change and nothing else.

Q&A

Q: Plus at the edge — worth a license? A: If drain, active health, or runtime membership would otherwise require a custom controller you must maintain: the license is cheaper than the engineering. If not: OSS + the 14.4 discipline covers you. Re-evaluate at every scale event.

Q: Can I run ingress-nginx AND raw NGINX tiers together? A: Yes — extremely common: ingress inside the cluster, classic NGINX tiers at the DMZ, both logging Module 12 shapes. Keep forwarded headers and PROXY protocol consistent across the boundary (Lessons 10.3, 4.2).

Q: How do I migrate controllers without an outage? A: Both controllers serving in parallel (new one shadowed — Lesson 11.4's mirror at the DNS/LB layer), golden-diff their responses, then shift real traffic by weight. Same play as any dark launch.

Q: What's the single most common fleet failure you see? A: Drift: one hand-edited instance that no pipeline knows about. The artifact header exists so that instance announces itself the moment someone opens it in an editor. Make the header obnoxious.

Takeaways

• The candidates differ on runtime control (Plus), portability/roles (Gateway API), and ecosystem (Envoy) — but the platform's blessing usually decides before features do.

• Four questions decide: platform mandate → runtime control → multi-team/multi-protocol → Envoy-shaped needs. The controller matters less than the contract around it.

• Boring fleets come from the pipeline: one blessed version, nginx -t in CI, golden diffs on every change, uniform observability, blast-radius hygiene.

• The graduation path (front door → ingress → tiered/Gateway → mesh) is additive when each stage keeps the discipline — config as artifact, one truth, validate, observe. That discipline is the deliverable of this module, and of the series' architecture half.