16.1 The Debugging Method

Advanced

What you will learn

1. The method: symptom → hypothesis → instrument → receipt.

2. The layer map for NGINX incidents, top to bottom.

3. The toolkit, honestly scored for a locked-down host — and yours.

4. The anti-methods that waste outages.

1. The loop

Every productive debugging session you will ever run is this loop, tightened:

SYMPTOM      "edge returns 502s sometimes, ~03:00 daily"
  -> HYPOTHESIS   "upstream connection pool exhausted overnight"
     -> INSTRUMENT   count ESTABLISHED conns per worker over time
        -> RECEIPT      ss output showing 1024/1024 at 02:58
           -> (hypothesis confirmed or refuted — loop again)

The loop's engine is the refusal to act before receipt. NGINX makes this easy: every layer of its stack emits evidence (Module 12's whole thesis). The method is mostly the discipline to read before you reload.

And before any of it: reproduce, then reduce. Reproduce the failure on demand (a request, a header, a load level); then shrink it — smallest request, smallest config — until removing anything makes the bug vanish. A minimal reproducible case tells you where to point every instrument in the map below, and without one you are guessing at a moving target.

2. The layer map

An NGINX incident lives at one or more of seven layers. Know them and their instruments, and no symptom is mysterious for long:

Walk downward, not randomly. Most incidents die at layers 1–3 (a config misunderstanding, a failing upstream, a cache that stopped caching — Lesson 12.3's max-age=30data story lived at layer 2 with a layer-3 receipt). Layers 4–7 are for the "everything looks right and it's still wrong" cases — which is exactly when method matters most, because those are the ones where guessing burns days.

The 03:00 example, walked properly:

1. Layer 2: the access log shows the 502s' $upstream_status is "502, 502, 502" — retries exhausted, not no-live-upstream.

2. Layer 2 again: $upstream_connect_time is "0.000, 0.000" — connect succeeds, the app accepts and dies. Not a network story.

3. Layer 4: /proc/<worker>/fd count during the window — climbing to ceiling? Then layer 3: stub_status shows Writing pinned at worker_connections.

4. Receipt in hand: connection-pool exhaustion → the fix is Lesson 5.4's keepalive + Lesson 9.1's fd ceiling math, not a shotgun restart at 03:20.

3. The toolkit, honestly scored

A locked-down host (like many production containers) allows none of the classic kernel-attached tools:

Two attitudes to hold simultaneously: never apologize for the tools you have (the debug log + /proc path answers most NGINX questions outright), and know exactly what you're missing (a flame graph finds hot loops the sampler only hints at; a packet capture settles disputes logs can't).

4. The anti-methods

Outage archaeology is full of these; name them so they feel shameful in the moment:

• The reload-as-prayer: restarting until the symptom hides. It destroys the evidence and the timeline. (Module 1 taught you reloads are safe — not that they're free of consequences for debugging.)

• Config spraying: changing five directives at once. When it "works," you've learned nothing and shipped three unrelated mutations. One hypothesis, one change, one receipt (the Lesson 14.5 golden-diff habit, now applied to debugging).

• Tool-first thinking: "let me strace it" before reading the error log that names the failing upstream in plain text. Layers, downward.

• Metric soup: dashboards in search of a question. The hypothesis picks the instrument, always the reverse of whatever your vendor's demo implied.

And the meta-rule that ends this series where it started: the master/worker model (Lesson 1.4) is your debugging map. Every instrument in every layer is really asking "what is the master doing? what is this worker doing?" When you can answer both with receipts, no NGINX symptom survives the conversation.

Try it yourself

• The whole module is this lesson, instrumented: a build with --with-debug (16.2), /proc access (16.3), the sampler (16.5), and crash machinery (16.4) — build each piece as its lesson shows.

• Rehearse the walk: stop one example backend, watch the debug log (Lesson 16.2) narrate the failure, then fix it — timing yourself at each layer. That elapsed time, shrinking, is the skill.

Q&A

Q: Where do bug reports / upstream issues fit? A: After receipts, before hacking. A receipt-bearing report ("1.30.5, this config, this debug-log excerpt, this core") is the difference between a fix in a week and a fix never.

Q: Two hypotheses, both plausible? A: Find the cheapest instrument that distinguishes them — that's the whole art. (If they can't be distinguished by any observation, they're the same hypothesis wearing two hats.)

Q: Debugging in production without slowing it? A: That's 16.2's debug_connection — the spotlight — and 12.5's severity discipline. You already own the safe instruments; this module's tools are their escalation path.

Takeaways

• The loop is symptom → hypothesis → instrument → receipt, and no action before receipt — NGINX emits evidence at every layer, so reading beats guessing.

• Seven layers, walk downward: config semantics → request lifecycle → runtime state → process → syscalls → network → CPU. Most incidents die at 1–3.

• Score your toolkit honestly: the rootless trio (debug log, /proc, core mechanics) covers most questions on a locked-down host; know precisely what strace/gdb/perf/tcpdump would add.

• The anti-methods — reload-as-prayer, config spraying, tool-first thinking — destroy evidence and teach nothing. One hypothesis, one change, one receipt.