4.4 WebSockets and Long-Lived Connections

Intermediate

What you will learn

1. The Upgrade handshake — what switches, when, and who decides.

2. The complete NGINX WebSocket configuration, every line justified.

3. The $connection_upgrade map pattern — why it exists and what breaks without it.

4. The timeout conspiracy: defaults that kill idle sessions.

5. Verifying the whole pipe, hands-on, with a real WebSocket client.

1. The handshake

A WebSocket begins life as an ordinary HTTP request with expensive ambitions:

GET /ws HTTP/1.1
Host: example.test
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==
Sec-WebSocket-Version: 13

The server either declines (plain HTTP response) or accepts:

HTTP/1.1 101 Switching Protocols
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Accept: s3pPLMBiTxaQ9kYGzzhZRbK+xOo=

The moment 101 is written, the TCP connection stops being HTTP. Both sides now speak RFC 6455 frames (opcode, length, mask, payload) — client frames always masked, server frames never. From NGINX's point of view, that's the whole trick it must support: relay bytes on a connection whose protocol changed mid-flight — which is exactly what a proxy that understands only requests-and-responses does badly by default.

2. The configuration, line by line

The working location:

# the map (http level) — from Lesson 2.4:
map $http_upgrade $connection_upgrade {
    default  upgrade;
    ''       close;
}

location /ws {
    proxy_pass http://127.0.0.1:9030;

    proxy_http_version  1.1;                     # (1)
    proxy_set_header    Upgrade    $http_upgrade;    # (2)
    proxy_set_header    Connection $connection_upgrade;  # (3)

    proxy_read_timeout  3600s;                   # (4)
    proxy_send_timeout  3600s;
}

(1) proxy_http_version 1.1 — the default 1.0 dialect has no well-defined Upgrade semantics, and NGINX's 1.0 upstream connections are request-scoped. 1.1 keeps the upstream connection open for the session's lifetime.

(2) Upgrade $http_upgrade — hop-by-hop by spec, so NGINX strips both headers when relaying (Lesson 4.2); you must re-add them for the handshake to reach the backend with the client's actual value.

(3) Connection $connection_upgrade — the subtle one. If you hardcoded Connection: Upgrade, then every request through this location — including ordinary GETs from browsers pre-connecting, health probes, and API calls sharing the location — would advertise an upgrade it isn't doing. The map derives the correct value per request: Upgrade present → upgrade; absent → close (normal, keep-alive-free completion). One variable, correct behavior for both populations.

(4) the timeouts — proxy_read_timeout (default 60s) fires when no bytes flow from the upstream; proxy_send_timeout guards the to-upstream direction. An idle-but-healthy chat or dashboard socket sends nothing for minutes — the default would reap it mid-session. Raising both (in the location, not globally!) is the fix; heartbeats from the app (pings every <timeout) are the robust fix, because they also survive NATs and LBs with their own idle policies.

And the proof, through the example gateway (the backend echoes text frames):

$ echo "hello world" | websocat ws://127.0.0.1:8400/ws
echo: hello world

A CLI like websocat does the handshake, sends your message as one masked text frame, and prints the reply — so the entire pipe (client → NGINX handshake relay → backend accept → frame relay back) is verified from one command line.

3. The failure modes you'll meet in production

"WebSocket connection failed" only through the proxy. Walk the checklist: proxy_http_version 1.1 missing (the 1.0 dialect silently fails the upgrade), the two headers not re-added (backend sees a plain GET and answers 404/426), or an intermediate device (corporate proxy, cloud LB) stripping Upgrade — the same header you just learned to preserve, being removed by someone else's Lesson 4.2.

Sessions die at exactly 60 seconds of silence. The default proxy_read_timeout. §2's fix (4).

Works from the office, dies from mobile networks. Carrier NATs reap idle UDP/TCP mappings aggressively — no NGINX timeout can help; the app must ping (frame-level heartbeats every 20–30s is the industry norm).

Upgrade succeeds, but the first frames never arrive. Usually proxy_buffering confusion — buffering applies to the response headers and body streaming; after 101 the connection is a byte pipe and buffering directives effectively stop mattering. If you see this one, check for a second proxy in the path before blaming NGINX.

4. Beyond the basics

• Timeout symmetry: send_timeout (client-side) also applies — a client that stops reading (mobile background tab) can stall a worker socket; keep client-side send_timeout sane even when you raise upstream timeouts.

• Sticky sessions: ip_hash (Module 5) or application-level reconnection is how you survive backend restarts — a dropped socket carries state you must rebuild.

• HTTP/2 on the client side is not an option for WebSockets: NGINX does not implement RFC 8441 extended CONNECT, so browsers fall back to HTTP/1.1 for the Upgrade handshake. NGINX still speaks 1.1 upstream regardless — the Upgrade dance happens between NGINX and the backend even when the client never used it.

• proxy_socket_keepalive /TCP keepalives keep NAT state warm at the TCP level when the app can't ping — a band-aid, but sometimes the only one available.

Try it yourself

1. Watch the handshake through the pipe. Use curl to attempt an upgrade and see the backend's answer relayed:

   curl -s -i --http1.1 -H 'Host: gw.example.test' \
        -H 'Upgrade: websocket' -H 'Connection: Upgrade' \
        -H 'Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==' \
        -H 'Sec-WebSocket-Version: 13' \
        http://127.0.0.1:8400/ws --max-time 2

You'll see 101 Switching Protocols — then curl gives up speaking frames (that's a WebSocket client's job, e.g. websocat).

1. Break the map. Replace Connection $connection_upgrade with Connection "Upgrade" and run the plain-GET probe:

   curl -s -H 'Host: gw.example.test' http://127.0.0.1:8400/ws

Then read the backend's log/output: it received an Upgrade advertisement on a non-upgrade request. Restore the map form.

1. Feel the timeout. Temporarily set proxy_read_timeout 1s; and open a socket that connects but sends nothing (piping a message into websocat sends it immediately, so instead use a raw connect):

   timeout 3 bash -c 'exec 3<>/dev/tcp/127.0.0.1/8400; printf "GET /ws HTTP/1.1\r\nHost: gw.example.test\r\nUpgrade: websocket\r\nConnection: Upgrade\r\nSec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==\r\nSec-WebSocket-Version: 13\r\n\r\n" >&3; head -c 200 <&3'

The connection dies ~1s after the handshake — the reaping, observed.

1. Full round trip (what to assert):

   echo "verify me" | websocat ws://127.0.0.1:8400/ws
   echo "shout" | websocat ws://127.0.0.1:8400/ws/upper

Q&A

Q1. Why must Upgrade and Connection be re-added with proxy_set_header when NGINX relays them perfectly for normal requests?

Q2. Why $connection_upgrade instead of the literal Upgrade?

Q3. Your dashboard sockets disconnect after precisely one minute of idle. Which directive, which default, and what are the two classes of fix?

Q4. After 101, can NGINX rate-limit individual messages or log them?

Takeaways

• WebSockets = an HTTP handshake ( Upgrade/ Connection/ 101) that flips the connection into RFC 6455 frames; NGINX's job is to relay the handshake then move bytes, forever.

• The canonical config is four lines with reasons: HTTP/1.1 upstream, re-added Upgrade, map-derived Connection, raised read/send timeouts in the location only.

• Mixed traffic is why the map exists — per-request derivation beats hardcoded protocol lies.

• Idle reaping is the production killer; timeouts on the NGINX side, heartbeats on the app side, and know your intermediary NATs.

• Post-101, NGINX is deliberately dumb — plan observability and limits accordingly.