16.2 The Debug Build

Advanced

What you will learn

1. Building the microscope: --with-debug, and how to wire it in side-by-side.

2. Reading the narration: the request's whole story, decoded line by line.

3. The cost model: why 1,733,564 lines happened, and why debug_connection is the production shape.

4. The workflow: spotlight one connection, reproduce, read, revert.

1. Building the microscope

Module 15's pipeline, one flag different:

./configure --with-debug --with-threads ...   # your usual flags
make && cp objs/nginx ...

A dedicated build run (the ./configure invocation from §2, kept in a script) produces nginx-debug — same 1.30.5, same flags as the packaged stable plus the debug logging machinery, installed under its own prefix ( /usr/local/nginx-debug). Start the stable binary or the microscope by path — /usr/local/nginx-debug/sbin/nginx-debug versus the packaged one — two binaries, one fleet, no compromise:

running binary: .../nginx-debug/sbin/nginx-debug
nginx version: nginx/1.30.5   (built by gcc 14.2.0)

Why not always debug-built? The machinery costs a little always (compiled-in branches) and a LOT when enabled (everything you're about to see). Distro builds omit it for good reasons; you build it for bad days.

2. Reading the narration

With the binary in place, the switch is one directive — error_log at debug level. Then a single request to the proxied :9800 produces this, in order (real lines):

[debug] 17661#17661: *19793 http proxy header: "Content-Type: application/json"
[debug] 17661#17661: *19793 http proxy header: "Connection: keep-alive"
[debug] 17661#17661: *19793 http proxy header: "Keep-Alive: timeout=5"
[debug] 17661#17661: *19793 http proxy header: "Transfer-Encoding: chunked"
[debug] 17661#17661: *19793 http proxy header done
[debug] 17661#17661: *19793 http finalize request: 0, "/some-path?probe=1" a:1, c:1
[debug] 17661#17661: *19793 malloc: 0000558B3BA7B920:1024

Decode the format once, read it forever:

• 17661#17661 — worker pid#tid (process id and thread id; they match because a single-threaded worker's main thread has tid == pid — build with --with-threads and helper threads show different tids). Which worker served you is never a mystery again.

• *19793 — the connection id: every line about YOUR request carries it. grep '\*19793' extracts one request's complete biography from a noisy log. (Module 12's $request_id for the access log; this is the error-log sibling.)

• The vocabulary is phase-shaped: accept → read request line → parse headers → http proxy header (the exact bytes relayed upstream!) → header done → finalize → malloc/free accounting. Module 11's phase model, printed by the code that runs it.

What this buys that nothing else can: you see the exact headers NGINX sent upstream and the exact decisions it made about them — no speculation about "did my proxy_set_header survive?" The debug log is the surviving header list, in transmission order.

3. The cost model — 1,733,564 lines

The running total after a few minutes of casual probing:

(total debug lines this session: 1733564)

One point seven million. That's the floodlight: a debug level on a busy edge writes gigabytes per hour, thrashes the disk, and can measurably slow request processing. It is a staging/one-connection instrument wearing a production costume.

The production shape is the spotlight — global warn, debug only for a designated address:

events {
    worker_connections 1024;
    debug_connection 127.0.0.1;     # YOUR address gets full detail
    debug_connection 203.0.113.7;   # the reporter's, if reproducing needs them
}

debug_connection (requires the debug build!) grants debug-level detail only to matching clients. One reporter reproduces their weird interaction; the other 50k rps never notice. Combine with Lesson 12.5's per-location error_log and you can aim the microscope at one door of one vhost for one client — surgical verbosity.

4. The workflow

The disciplined loop (16.1's method, concretized):

1. Deploy the debug binary side-by-side (same version, own prefix)
2. error_log at debug + debug_connection <your-ip> (the spotlight)
3. Reproduce the symptom once — capturing *N connection ids
4. grep '\*N' error.log > the-request-biography.txt
5. Read it against the phase model; form the next hypothesis
6. Revert the log level the moment you have the receipt
   (a forgotten debug level is a slow outage you caused)

Step 6 deserves its own underline: debug-left-on is the classic self-inflicted incident. Getting the narration while running is the easy part — production's discipline is asserting it stops.

Try it yourself

• nginx -V 2>&1 | grep debug for the build; one request at debug level for the narration vocabulary; a line count on the error log for the flood.

• Read one biography: curl http://127.0.0.1:9800/anything, then grep "$(tail -1 /var/log/nginx/access.log | ...)" ... — or just tail -200 /var/log/nginx/error.log and follow one *N by eye.

• Turn on the spotlight: add debug_connection 127.0.0.1; to the events block, drop global level to notice, reload — confirm curl still narrates and other traffic doesn't.

Q&A

Q: Debug build in production — ever? A: Side-by-side with the same version and a fraction of traffic (canary door), spotlight-aimed — teams do run this for the impossible bugs. Full-floodlight debug in prod, never.

Q: Does --with-debug alone slow nginx down? A: Negligibly (compiled-in conditionals, disabled levels). The cost is enabled debug level, not the build. Build it, keep it quiet, aim it when needed.

Q: error_log debug vs rewrite_log on (Lesson 11.1)? A: Different scopes: rewrite_log narrates one phase in any build; debug level narrates everything in the debug build. Surgical vs total — choose by where the mystery lives.

Q: The narration shows my header twice / not at all — bug? A: Read the http proxy header lines as ground truth: twice usually means an inherited + an explicit proxy_set_header (the Module 4 duplicate lesson); not at all means it was cancelled by a same-name directive or never set. The log settles what no documentation argument can.

Takeaways

• The microscope is one flag to build ( --with-debug), one directive to aim ( error_log ... debug), and one directive to focus ( debug_connection — the production spotlight).

• The narration format — worker#worker, *connection-id, phase-shaped vocabulary — turns any request into a greppable biography, including the exact upstream header bytes.

• The cost is real and quantified: 1.7M lines from minutes of casual traffic. Floodlight for staging, spotlight for production, and always revert when you have the receipt.

• rewrite_log is the surgical scalpel for one phase; the debug build is the whole-body scanner. Both earn their place in the 16.1 layer walk.