3.4 Serving Files Fast: sendfile, tcp_nopush, open_file_cache

Basic

What you will learn

1. The read-write path of a naive file server, and why it's the bottleneck sendfile deletes.

2. sendfile — the zero-copy mechanism, its Linux specifics, and its genuine limits (TLS! directio!).

3. tcp_nopush and tcp_nodelay — the packet-shaping pair and the reason they coexist.

4. open_file_cache — what it caches (metadata, not bytes), its four tuning knobs, and the staleness contract you accept.

5. A tuning order of operations for a static-heavy box (what to measure before touching any of this).

1. The naive path (what you're optimizing away)

Serving one file, the textbook way, costs four crossings of the kernel-user boundary per response:

worker                     kernel
  │  read(fd, buf, 64k)      │
  ├─────────────────────────▶│  copy: page cache -> user buffer
  │◀─────────────────────────┤
  │  write(sock, buf, n)     │
  ├─────────────────────────▶│  copy: user buffer -> socket buffer
  │◀─────────────────────────┤         then NIC (or crypto)

Multiply by every chunk of every file, add the copies, and you've bought: two memory copies per byte, two context switches per chunk, and cache pollution — all for bytes that never needed to change on the way through. On a box pushing 10 Gbps of static content, that's the entire CPU budget.

2. sendfile — zero-copy to the socket

sendfile  on;     # (http/server/location; often set at http level)

One syscall — sendfile(out_sock, in_fd, offset, count) — asks the kernel to move file bytes directly from the page cache to the socket buffer. No user-space copy, no user-space buffer, one crossing instead of four:

worker                     kernel
  │  sendfile(sock, fd, n)   │
  ├─────────────────────────▶│  page cache ─▶ socket ─▶ NIC
  │◀─────────────────────────┤        (DMA does the heavy lifting)

The honest fine print:

• It only helps static bytes. Dynamic bodies (proxied responses, njs output) never pass through sendfile — they're already in user memory.

• TLS historically disabled it — bytes must be encrypted in user space, so plain sendfile can't feed the cipher. Modern Linux ( MSG_ZEROCOPY, sendfile into kTLS sockets) re-enables zero-copy TLS; NGINX uses it where available. Practical translation: on an HTTPS-only vhost the win shrinks but rarely vanishes.

• directio bypasses it. directio 4m; (Module 8's large-file friend) reads around the page cache — and NGINX automatically disables sendfile for files above the threshold. Big media boxes tune both together.

• Files must be regular files on filesystems the kernel supports. NFS historically fought with it; local disks and most modern stacks are fine.

3. The packet shapers: tcp_nopush + tcp_nodelay

The pair that looks contradictory and isn't:

tcp_nopush   on;    # with sendfile: fill packets FULLY before sending
tcp_nodelay  on;    # never delay the LAST, partial packet

TCP_NODELAY disables Nagle's algorithm — the kernel's habit of holding small writes hoping to coalesce them (latency for the sake of ancient networks). Interactive traffic wants it off… I mean on. (Naming is confusing: "nodelay ON" = "don't delay". It's the default in NGINX since 1.1.5 — stated explicitly here for teaching.)

TCP_NOPUSH (Linux) does the opposite trick for bulk streams: with sendfile, hold the socket open until segments are full, then release a burst of maximal packets. You get fewer, larger packets on the wire — fewer interrupts, better wire efficiency — and none of Nagle's latency because tcp_nodelay guarantees the final partial packet leaves immediately.

Together: bulk efficiency and tail latency — the burst fills the pipe, the tail doesn't wait. That's why every serious config sets both, and why tcp_nopush without sendfile on; is a no-op (its mechanism rides on sendfile chunks).

4. open_file_cache — metadata, not bytes

The name oversells and undersells simultaneously. What it caches, per worker, in memory:

Why that's still a big deal: every static request otherwise costs a stat() (or open) syscall tree — and on hot files, repeated. On a box serving 50k rps of cached metadata, that's the difference between syscall storm and quiet loop.

The four knobs, verbatim from the example config:

open_file_cache          max=1000 inactive=20s;  # ≤1000 entries; drop if unused 20s
open_file_cache_valid    30s;    # re-stat entries at least this often
open_file_cache_min_uses 2;      # ...but only files hit ≥2 times qualify
open_file_cache_errors   on;     # cache "doesn't exist" too (404 storms)

The staleness contract

Between revalidations, NGINX serves from cached metadata: replace a file, and Content-Length / Last-Modified can be stale for up to _valid seconds. Deleted files keep serving until the fd closes (open fds outlive unlink on Linux — usually harmless, occasionally surprising). _errors on extends the deal to negative answers: a file created within the validity window stays 404 until revalidation.

That last one is the production gotcha worth memorizing — deploys that drop new files in place can appear to "not work" for up to open_file_cache_valid seconds. CI-friendly values: _valid 5s or reloading after content drops — a reload spawns fresh workers whose per-worker open_file_cache starts empty, so the next request re-stats and picks up new files immediately (only the draining old workers keep stale entries; Module 16 measures this live with strace).

5. The tuning order of operations

These knobs are the last mile, not the first. The professional sequence for "make static serving fast":

1. Measure: ss -s, pidstat, nginx stub_status (Module 12), flame graph (Module 16)
2. Capacity: worker_connections, rlimit_nofile        (Module 9)
3. Latency: keepalive, http2                          (Modules 9)
4. Bytes on the wire: gzip/brotli                     (Module 9) ← often the biggest win
5. Only then: sendfile/tcp_nopush (defaults), open_file_cache tuning

Reordering 4 and 5 is the classic junior mistake: compressing a 90 KB JavaScript bundle to 30 KB dwarfs any syscall economy on the uncompressed original.

Try it yourself

1. Confirm the knobs are live — they're in the effective config:

   nginx -T 2>/dev/null | grep -E 'sendfile|tcp_no|open_file'

1. Watch the staleness contract. With our example's valid 30s:

   curl -s -o /dev/null -w '%{http_code}\n' -H 'Host: static.example.test' \
        http://127.0.0.1:8300/fresh-test.txt       # 404 (negative entry cached)
   echo hello > /var/www/main/fresh-test.txt
   curl -s -o /dev/null -w '%{http_code}\n' -H 'Host: static.example.test' \
        http://127.0.0.1:8300/fresh-test.txt       # still 404?! ...wait 30s... 200

You have just felt open_file_cache_errors on. (Module 16 will show the same experiment with the syscalls visible.)

1. Tune it away. Set open_file_cache_valid 2s;, reload, repeat the experiment — the 404→200 flip arrives in ~2 s. Decide which behavior your deploy pipeline can tolerate.

2. A/B sendfile honestly. With ab/ wrk if available (or a tight curl loop) measure the 5 MB payload.bin with sendfile on vs off — on loopback with small files, expect no visible difference; that's the lesson, not a failure.

Q&A

Q1. Why does sendfile not accelerate your HTTPS API gateway's JSON responses?

Q2. tcp_nopush on; alone, with sendfile off; — what changes?

Q3. After a deploy, a new CSS file 404s for exactly ~30 seconds on one worker but is instantly fine on another. Explain.

Q4. What does open_file_cache not cache, and what does instead?

Takeaways

• Naive serving pays two copies and four crossings per chunk; sendfile collapses that to one kernel-side move — for static bytes, where TLS and directio interplay honestly.

• tcp_nopush (full packets, with sendfile) + tcp_nodelay (no Nagle on the tail) = bulk efficiency without tail latency.

• open_file_cache caches metadata and negative lookups per worker — its _valid window is a staleness contract your deploy process must respect.

• Tuning order: measure → capacity → latency → compression → and only then these micro-levers. Compression beats syscall golf almost every time.