9.2 Buffers and the Kernel Seam

Intermediate

What you will learn

1. The buffer inventory: request-path and response-path, in order.

2. The spill mechanics — bodies vs response chunks, with a real [warn] receipt.

3. client_header_buffer_size and large_client_header_buffers —the DoS valves.

4. The kernel's hidden buffers: socket queues, Nagle, and what tcp_nodelay/ tcp_nopush already did for you (Module 3 recap).

5. A tuning order that avoids the classic mistakes.

1. The buffer inventory

A request crosses NGINX through a chain of allocations, each with its own directive and default:

client request                    NGINX                        upstream
──────────────                    ────                        ────────
headers ────────▶ [client_header_buffer_size 1k]                │
                   └─ overflow ▶ [large_client_header_buffers 4×8k]
body ───────────▶ [client_body_buffer_size 8/16k] ─▶ (memory) ──┤
                   └─ overflow ▶ [client_body_temp_path] DISK   ▼

upstream response ◀──────────────────────────── [proxy_buffer_size 4k]
                                                [proxy_buffers 8×4k]
                   └─ overflow ▶ [proxy_max_temp_file_size] DISK
                   └─ busy window ▶ [proxy_busy_buffers_size 8k]
client response ◀────────────────  (assembled, sent, Module 3's sendfile)

Two families, two policies:

• Request side ( client_*): sized for legitimate clients. Small defaults are a feature — they cap per-connection memory and shed oversized garbage early.

• Response side ( proxy_*): sized for your backend's real responses*. Here bigger is often right — every byte held in memory is a byte your backend worker didn't wait on a slow client for (Lesson 4.6's buffering economics).

2. The spill — caught on camera

The example http context sets client_body_buffer_size 1k — tiny on purpose. Post an 8 KB body and the buffer overflows; the whole body (or its remainder) is written to a file under client_body_temp_path. The receipt (real log line):

2026/09/22 21:47:12 [warn] 9623#9623: *58 a client request body is buffered
to a temporary file /var/cache/nginx/client_temp/0000000005, client:
127.0.0.1, server: fast.example.test, request: "POST /post/ HTTP/1.1", …

That [warn] is the production signal for "uploads are hitting disk" — and a tiny body that fits produces no warn at all, proving the threshold, not just the existence, of the mechanism. Three operational notes:

• The file is unlinked when the request ends — you can't ls your way to the evidence after the fact; the log is the record. (Trying to beat the unlink with a quick ls is a race you lose.)

• Spills are normal for uploads (any body bigger than 8–16 KB spills by design) and pathological for tiny payloads (your 2 KB API POSTs should never warn → raise the buffer to match reality).

• proxy_request_buffering off disables the whole story for a location: the body streams to the upstream as it arrives (streaming APIs, huge uploads to object stores) at the cost of Module 5's retry safety (a half-sent body can't fail over).

The response-side spill ( proxy_buffers overflow → proxy_temp) is rarer and usually means: huge responses + slow clients + buffering on — the Lesson 4.6/Module 8 interplay. Its fix is almost never "more buffers"; it's proxy_max_temp_file_size 0; (refuse to disk-buffer — backpressure the backend instead) or streaming semantics.

3. Header buffers — the DoS valves

client_header_buffer_size   1k;
large_client_header_buffers 4 8k;

Normal requests' headers fit the first buffer; oversized ones (long cookies, big tokens, hostile garbage) fall to the second, capped at 4 buffers × 8 KB. Beyond the cap, NGINX answers 494 (or 400) and drops the connection — a hard, cheap limit on header-based memory amplification. The example config sets both explicitly to make the knobs visible; the defaults are these same values. Tune large_client_header_buffers up only when legitimate clients complain (some SSO/JWT stacks send 5 KB+ headers) — and remember every extra kilobyte multiplies across an attacker's connection budget.

4. The kernel's seam

Between NGINX's buffers and the wire live kernel structures you sized in Lesson 1.4's connection math and Module 3's file serving:

• Socket buffers (per connection, ~4–8 KB each side): the actual memory cost of "a connection"; tcp_nodelay/ tcp_nopush (Module 3) govern when the kernel releases bytes from them — the packet-shaping pair you already know.

• The accept queue ( somaxconn × listen backlog=): where connections wait before a worker accepts them — storms overflow it silently (SYN drops) and clients see timeouts. NGINX-side fix: more workers is not it; it's backlog= in the listen + the kernel ticket.

• Page cache (Module 3's open_file_cache interplay): "disk" reads that never leave RAM. Before blaming NGINX's buffers for slowness, check whether the bytes in question ever touched a platter.

5. The tuning order

1. DON'T touch buffers first. Measure (log timings, spill warns, fd counts).
2. Fix semantic mistakes: buffering off where streaming belongs (4.6),
   spill-free sizes for your real payload distribution.
3. Raise response buffers to match backend reality (proxy_buffers 16 8k
   for chunky APIs) — one coherent set, never one dial.
4. Header valves: only for measured, legitimate large-header clients.
5. Kernel tickets WITH numbers: somaxconn for SYN-drop receipts,
   LimitNOFILE for fd ceilings.

The classic mistake is step 0: copying a "performance config" from a blog whose author had a different workload, different kernels, and a different century. The defaults are the result of two decades of production teaching; earn every deviation with a receipt.

Try it yourself

1. Generate and grep your own spill receipt (§2):

   curl -s -o /dev/null -H 'Host: fast.example.test' -X POST \
        --data-binary "$(head -c 8192 /dev/zero | tr '\0' x)" \
        http://127.0.0.1:8900/post/
   grep "buffered to a temporary file" /var/log/nginx/error.log | tail -1

Then POST 512 bytes and confirm no new warn — the threshold, observed from both sides.

1. Header valve excursion. Send a request with a 20 KB header ( curl -H "X-Big: $(head -c 20000 /dev/zero | tr '\0' x)") → rejected at the valve. Now raise large_client_header_buffers 4 32k, reload, retry — admitted. Restore.

2. Response-side inspection. Re-run Module 8's large-file tests and watch ls /var/cache/nginx/proxy_temp/ — nothing (cache temp and proxy temp are different pools; disk-buffering only fires when buffers and your rules allow).

3. The golden diff. Add $request_time/ $upstream_response_time reading practice: hit /post/ (slow backend) and split the time — request time minus upstream time is NGINX+network's share. One log line, the whole attribution story.

Q&A

Q1. API POSTs average 32 KB; error log shows spill warns for most. Two fixes and their trade-offs?

Q2. Why can't you find the spill file an hour after the request?

Q3. A client with a 30 KB JWT header gets 400/ 494 at your edge. Which directive, what value, what risk?

Q4. $request_time = 2.1 s, $upstream_response_time = 2.0 s. Where's the problem, and which buffer family is not implicated?

Takeaways

• Two buffer families with opposite policies: request-side small by design (DoS valves, spill to disk), response-side sized to your backend (free its workers from slow clients).

• The spill signature is a [warn] log line, not a findable file; tiny-payload spills are a tuning smell, upload spills are normal.

• Header valves ( client_header_buffer_size + large_client_header_buffers) cap header amplification — widen only with measured cause.

• Below NGINX live the kernel's queues and caches — module 3's packet shapers, Lesson 1.4's connection memory, and tickets you file with numbers.

• Debugging order: attribute (log timings) → buffers (warns/valves) →kernel (receipts) → wire. Defaults are battle-tested; deviate with evidence.