5.4 Connection Reuse: Upstream keepalive Pools

Intermediate

What you will learn

1. The handshake tax — what per-request connections actually cost.

2. The keepalive directive and the two-line incantation it requires.

3. The proof: watching reused connections with ss.

4. Pool sizing, idle reaping, and the Connection: "" subtlety.

5. How keepalive interacts with retries, quarantines, and HTTP/2.

1. The handshake tax

Without upstream keepalive, every proxied request pays:

TCP:   SYN → SYN-ACK → ACK           (~0.02–0.1 ms on loopback;
                                      1–40 ms across a real network or
                                      a container overlay!)
HTTP:  request → response → close    (no amortization, ever)

Multiply by request rate: 5 000 rps across a 1 ms-RTT network = 5 seconds of pure handshake time consumed per second — plus ephemeral port churn (Module 9's ip_local_port_range becomes the wall), plus TIME_WAIT sockets accumulating on both sides. The fix is boring and beautiful: stop closing.

2. The directive and the incantation

upstream pool-ka {
    server 127.0.0.1:9001;
    keepalive 8;                      # idle connections cached per worker
}

location /ka/ {
    proxy_pass http://pool-ka/;
    proxy_http_version  1.1;          # (1) the dialect
    proxy_set_header    Connection  "";  # (2) the header
}

keepalive 8 caps the idle connections each worker caches per pool. But the directive alone is inert without the incantation — both lines, for reasons you already know from Module 4:

1. proxy_http_version 1.1 — the 1.0 default sends Connection: close semantics upstream and treats connections as request-scoped.

2. Connection "" — even on 1.1, NGINX sends an explicit Connection: close header by default (belt-and-suspenders from a more cautious era). Overriding it with the empty string removes the header entirely — the HTTP/1.1 default (persistent) applies, and after the response the connection returns to the pool instead of closing.

3. The proof — ss doesn't lie

The pool, made visible:

$ for i in 1 2 3 4 5 6; do
    curl -s -H 'Host: lb.example.test' http://127.0.0.1:8500/ka/x >/dev/null
  done

$ ss -tn | grep ':9001 ' | head -3
ESTAB  0  0  127.0.0.1:43218  127.0.0.1:9001
ESTAB  0  0  127.0.0.1:43217  127.0.0.1:9001

ESTABLISHED connections that outlive the requests — the pool, caught on camera. (A two-worker run counted 14; with one worker and sequential requests you'll typically see 1–3: one live pipe being reused, plus stragglers from earlier.) Compare the non-keepalive locations ( /rr/ etc.): after their responses complete, ss shows nothing to :9002/:9003 — every connection already closed.

The reuse is also readable in the backend: Node's HTTP server counts connections, and a burst of 6 requests through /ka/ arrives on one server-side socket (visible in the echo's connection bookkeeping), while /rr/ opens six.

4. Sizing, reaping, and subtleties

Sizing. keepalive N is per worker: total idle capacity = N × workers. A sane starting point: N ≈ (peak rps per worker × upstream RTT) — the Little's Law answer for "connections in flight that could be reused." For 1 000 rps/worker at 5 ms RTT: ~5; caches and bursty traffic reward more. Oversizing costs idle fd's (fine); undersizing costs re-handshakes (worse).

Who closes first? Both ends matter: NGINX reaps idle pool entries (it keeps at most N), and the backend has its own idle timeout (Node's default keepAliveTimeout: 5 s; many frameworks: 75 s). The effective pool lifetime is the minimum of the two — a backend with a 2 s timeout silently caps your reuse. Race-condition avoidance is built-in: NGINX validates a pooled connection (detects half-closed sockets) before reusing it, and on a dead-pipe write it falls back to a fresh connection.

Interaction with the failure machinery (Lesson 5.3):

• A pooled connection to a peer that died mid-idle: detected on reuse, request retried on a fresh connection — pool and quarantine are orthogonal layers.

• backup activation doesn't poison pools — each peer has its own cached connections; the scheduler's choice and the pool's cache are per-peer.

HTTP/2 upstreams: grpc_pass (and any http2 upstream) multiplexes streams over one connection per peer — a different, stronger form of reuse where the "pool" is the protocol itself. The keepalive curriculum here is the HTTP/1.1 upstream story; gRPC got it for free.

5. The checklist, industrialized

When latencies on proxied routes look like "network + a mystery 1–40 ms" and backends report huge connection churn:

1. keepalive N; in the upstream                    (the cache)
2. proxy_http_version 1.1;                          (the dialect)
3. proxy_set_header Connection "";                  (the header)
4. backend idle timeout > expected idle gaps        (the other end)
5. watch ss -tn | grep :<backend-port>              (the proof)

All five live in the /ka/ location and its pool — the configuration this lesson opened with is the complete production pattern, and §3's ss check is your regression test.

Try it yourself

1. Catch the pool yourself (§3): fire 6 requests at /ka/, then immediately ss -tn | grep ':9001 ' — count ESTABLISHED. Then hit /rr/ 6 times and grep :9002 — nothing persists. One command, both worlds.

2. Break the incantation. Remove the Connection "" line from /ka/, reload, rerun the burst + ss: the pool evaporates (every upstream request now carries Connection: close). Restore.

3. Sizing observation. Set keepalive 1;, hammer /ka/ with 3 concurrent curl loops, and watch ss — one idle connection per worker survives; bursts open extras that close on release. The per-worker accounting becomes visible.

4. The backend timeout ceiling. From the backend side, Node closes idle sockets after ~5 s: run the /ka/ burst, wait 10 s, ss again — NGINX's pool entries are gone (the other end reaped them). Effective pool lifetime = min(your timeout, theirs).

Q&A

Q1. keepalive 16; alone in the upstream — requests still open a fresh connection each time. Why?

Q2. Why is keepalive per worker rather than global?

Q3. A pooled connection is reused after the backend restarted. What happens?

Q4. Why does grpc_pass not need this lesson?

Takeaways

• Per-request connections are a real tax at scale: handshakes, ephemeral ports, TIME_WAIT, and (worst) cross-network RTTs per request.

• The three-part pattern: keepalive N + proxy_http_version 1.1 + Connection "" — and you can now explain why each line exists.

• Pools are per worker, capped, and validated on reuse; effective idle lifetime is bounded by the backend's own timeout.

• The proof is one command: ss -tn showing ESTABLISHED sockets that outlive requests — and it doubles as a permanent regression check.