5.5 Active Health Checks, Plus, and the Open-Source Road

Intermediate

What you will learn

1. The precise difference between passive and active health checking — and what each catches that the other can't.

2. NGINX Plus's health_check directive and upstream API — what the money actually buys.

3. The open-source substitutes: probe traffic, log-driven quarantine, external orchestrators — with working patterns.

4. Session persistence: sticky cookie/ sticky learn (Plus) vs cookie-hash (open source) — the real trade-offs.

5. A decision framework: when Plus pays for itself.

1. Passive vs active — the precise difference

Everything you built in Lesson 5.3 is passive: NGINX infers peer health from real traffic. Three structural blind spots follow:

Active health checking closes all three: NGINX (Plus) sends probes on its own schedule, checks response semantics (status, body patterns, latency), and readmits peers the moment they prove health — with zero real traffic at risk.

2. What Plus buys: health_check and the API

# NGINX Plus syntax (health_check/match are Plus-only — not valid on open-source NGINX)
upstream api-pool {
    zone api-pool 64k;              # required: shared peer state
    server 10.0.0.1:8080;
    server 10.0.0.2:8080;
}

server {
    location /healthz-probe {       # internal endpoint the probe hits
        proxy_pass http://api-pool;
        health_check interval=2s fails=2 passes=2
                       uri=/health/ready
                       match=api_ready;    # semantic assertion
    }
}

match api_ready {                    # what "healthy" MEANS:
    status 200;
    header Content-Type = application/json;
    body ~ '"state":"ready"';        # regex on the BODY
}

Read the power in the parameters: interval (probe cadence), fails (trips to mark down — active quarantine), passes (proofs to readmit), uri (a deep health endpoint — DB, queues, warmup — not just /), and match blocks: the difference between "TCP answers" and "the service works."

The second Plus pillar is the upstream API:

curl -X POST -d '{"server":"10.0.0.3:8080"}' \
     http://localhost:8080/api/6/http/upstreams/api-pool/servers
curl http://localhost:8080/api/6/http/upstreams/api-pool/servers   # + live
# state: health, responses, fails, chosen-count per peer — JSON, no reload

Add/remove/ drain peers at runtime (drain = stop choosing, let in-flight finish — the Lesson 5.1 pattern without the reload). Autoscaling hooks directly into this; Kubernetes users get it via the Ingress Controller's own machinery (Module 14).

Plus also ships the sticky family: sticky cookie srv_id expires=1h; (set a cookie naming the chosen peer), sticky learn (learn the app's own session cookie → peer mapping from responses — zero client visibility), and sticky route (extract a routing token from the request).

3. The open-source road

None of that is magic, and most of it composes from parts you already own.

Pattern A — probe traffic (the 90 % answer)

Give the pool a health-checking route and let something probe the pool through NGINX:

location = /healthz-all {
    proxy_next_upstream error timeout http_502 http_503 http_504;
    proxy_pass http://api-pool/health/ready;   # deep endpoint
}

Your monitoring (Prometheus blackbox, uptime checkers, Kubernetes readiness probes in Module 14) hits /healthz-all every N seconds. Each probe is real traffic: it exercises proxy_next_upstream (retries past dead peers — keeping their failure counters fresh), refreshes quarantine state, and your alert becomes "N retries happened in the probe log line" — parseable from $upstream_addr's comma count, which Module 12 will formalize. Limitations, honestly: probes don't repair anything themselves (they just keep the passive machinery fed), and http_5xx retries need care (Lesson 5.3's warning).

Pattern B — log-driven orchestration

The access log's $upstream_addr/ $status is a health stream (Module 12 makes it structured JSON). A sidecar tails it, applies your own definition of "sick" (error rate, latency percentile, connect-fail count), and acts: flip a peer to down in a generated include, then nginx -t + reload (Lesson 1.5's safe ritual) — or remove the peer from your service discovery so the next config rendering omits it (Module 13 builds exactly this with Consul-Template). You've rebuilt active checking with a log pipeline and a templater — slower to act than Plus (seconds vs sub-second), often fast enough.

Pattern C — DNS as the control plane

With resolve-flagged peers + a resolver (Module 13), health decisions move into DNS: your orchestrator removes sick endpoints from the SRV/A records; NGINX follows within valid= seconds. This is the cloud-native answer — health checking isn't NGINX's job at all, it's the platform's.

Session persistence, open-source edition

upstream sessions {
    hash $cookie_srv_id consistent;      # the app (or you) sets srv_id
    server 10.0.0.1:8080;
    server 10.0.0.2:8080;
}
map $cookie_srv_id $need_srv_id {        # first-visit helper:
    ""  1;                               # no cookie yet?
    default 0;
}
server {
    location / {
        proxy_pass http://sessions;
        # app sets the cookie; or NGINX can, with a tiny njs hook (Module 15)
    }
}

The trade-offs vs Plus sticky: cookie-hash breaks mapping when the peer set changes (mitigated by consistent), and there's no learning mode — but for most session stores it's plenty. (And the Lesson 5.2 verdict applies: prefer the app's own session cookie as the hash key anyway.)

4. When Plus pays for itself

An honest decision framework, not a sales sheet:

The engineering truth: everything in Lessons 5.1–5.4 is Plus-identical — same engine, same algorithms, same keepalive pools. The delta is operational control (probes, API, stickiness) — real value, specific situations, never speed.

Try it yourself

Everything here runs on open-source NGINX, so today's practice builds the substitute:

1. Build Pattern A. Add the /healthz-all location above pointing at pool-fail (which has a dead peer), reload, and probe it 5 times:

   bash

   Copy

      for i in 1 2 3 4 5; do
        curl -s -o /dev/null -w '%{http_code} ' -H 'Host: lb.example.test' \
             http://127.0.0.1:8500/healthz-all; echo; sleep 1
      done

   Watch the access log: the probe's upstream_addr shows the dead peer being retried-past, then absent (quarantined), then back (probation) — your "active" checker, feeding the passive machinery.

2. Measure recovery time. With fail_timeout=10s, count probes until the dead peer disappears from upstream_addr (first-hit quarantine) and until it returns (probation probe). Those are the SLA numbers your monitoring would report.

3. Design exercise (no code): sketch the log-driven sidecar for your own fleet — which log fields (Module 12 preview: $upstream_addr, $upstream_status, $upstream_response_time), what rule trips "sick," and what the sidecar does (reload-with-down vs DNS removal). Write it in your notes; Module 13 will implement it for real.

Q&A

Q1. A pool peer receives zero traffic after a deploy rebalances hashing. It dies. What does passive checking do, and what would active checking do?

Q2. Your check must fail when the backend's dependency (DB) is down, even though the HTTP listener answers 200. Open source, one location + one app route — sketch it.

Q3. Why does the Plus upstream API require zone?

Q4. Compare hash $cookie_srv_id consistent (OSS) with Plus sticky learn on: client visibility, mapping durability, and failure behavior.

Takeaways

• Passive checks infer from traffic; active checks ask — catching idle-dead peers, semantic sickness, and enabling probe-proven recovery.

• Plus buys health_check+ match (semantic probes), the runtime upstream API (needs zone), and the sticky family — operations, not speed.

• The open-source road is real: probe routes feeding the retry machinery (Pattern A), log-driven quarantine via templated reloads (B), DNS-as-control-plane (C) — and cookie-hash stickiness covers most session needs.

• Choose Plus by operational requirements (autoscale dynamism, never-broken-guarantees, support), never by performance folklore.

Module 5 complete. Your pools schedule fairly, survive failures, reuse connections, and health-check themselves as well as open source allows. Next: Module 6 — the edge gets teeth: TLS termination, certificate operations, cipher hardening, mutual TLS, and access control.