16.4 Core Dumps

Advanced

What you will learn

1. What a core dump IS — the process's entire memory, frozen at the crash.

2. The two NGINX directives that put cores where you'll find them (and the cwd trap that comes with them).

3. The live experiment: SEGV a worker, keep the artifact, watch the respawn, serve the next request.

4. The gdb recipe — ready for the day you have the tool.

1. What you're catching

When a process dies on a fatal signal (SEGV, ABRT, BUS...), the kernel can write its core image to disk: memory mappings, the register set, the stack — the complete crime scene. Loaded into a debugger, a core answers the question no log ever will: what was the code doing at the instant of death?

For NGINX the practical shape: a worker segfaults (a module bug, a memory-corruption cascade, a cosmic ray). The master — Module 1's promise — respawns it in milliseconds and logs the event. Clients barely notice. YOU, though, want the why, and the why is in the core.

Three things must be true for the artifact to exist:

1. The rlimit allows it — core size limits default to 0 in most environments (deliberately: cores are big).

2. The kernel writes it where findable — core_pattern (here: plain core, relative to the process cwd).

3. The cwd is writable by the process — which for a daemonized nginx is /, which it cannot write. This is the trap, and NGINX ships the escape.

2. The two directives

worker_rlimit_core  100m;     # the rlimit workers get (the size budget)
working_directory   logs;     # where cores land (must exist, must be writable)

worker_rlimit_core lifts the ceiling on the worker's core size limit — the master calls setrlimit for its children at spawn time, which is why NGINX can do this without root for the soft limit.

working_directory changes the worker's cwd so the kernel-relative core path lands somewhere you control instead of an unwritable /.

The trap we earned building this example: our first start with these directives died with —

[alert] chdir("logs") failed (2: No such file or directory)
worker process exited with code 2 and cannot be respawned

working_directory logs resolved against the daemonized master's cwd ( /), not the prefix — /logs didn't exist, and a worker that can't chdir refuses to start. The fix: launch nginx from the prefix (systemd's WorkingDirectory=), i.e. cwd = prefix, so relative paths in working_directory mean what they look like they mean. (Production configs sidestep this with absolute paths: working_directory /var/coredump; — plus mkdir in your unit file, because the directive fails hard on a missing directory.)

3. The live experiment

The whole experiment, run end to end (real receipts):

kill -SEGV 17661        # a worker, mid-flight (simulate the crash)

core artifact: 2068480 bytes
2026/09/23 02:18:22 [alert] 17433#17433: worker process 17661 exited on signal 11 (core dumped)
workers after respawn: 2
next request: HTTP 200

Four facts, in order of operational importance:

1. The core exists — 2 MB at logs/core, exactly where working_directory pointed. The crime scene is preserved.

2. The master logged it with precision — signal number, pid, and the (core dumped) suffix that tells you the artifact is out there waiting.

3. The fleet self-healed — respawn was instant; the worker count never wavered. (Lesson 1.4’s worker model — the master owns the fleet, a dead worker is a respawn away — was a rehearsal for exactly this.)

4. The service never noticed — the very next request returned 200. This is why worker crashes can hide in busy edges: the only trail is a one-line alert and a file on disk.

That combination — invisible to users, recorded for you — is the design philosophy of the whole master/worker model, and this experiment is its proof, run live.

Housekeeping matters at this point: cores are size- budgeted by worker_rlimit_core, but a crash loop writes one core per death. The master's "cannot be respawned" guard (after rapid successive failures) limits the blast, but your runbook should still rotate/collect cores — a filled disk is its own incident, and it arrives wearing the costume of the one you were already having.

4. The gdb recipe (for when you have it)

A locked-down host often has no gdb ( ptrace_scope=2, no install rights). Your debug machines will. The recipe, ready:

# on the machine with the core and the SAME binary:
gdb /path/to/nginx-debug /path/to/core

(gdb) bt                    # the crashing stack — the headline
(gdb) bt full               # ...with locals (needs debug symbols: --with-debug
                           #  builds carry them; strip nothing you want to read)
(gdb) info threads          # all threads at death
(gdb) thread apply all bt   # every stack, for thread-race autopsies
(gdb) info registers        # the register set
(gdb) x/16i $pc             # the instructions around the fault

What you're reading: the top frames name the function that faulted — an nginx core function ( ngx_http_...), a module ( ngx_http_brotli_...), or libc — and that single fact routes the bug report (nginx core vs module vs platform). bt full plus a debug-built binary is the difference between "it crashed" and a maintainer saying "oh, THAT bug — fixed in 1.30.6."

The report that gets fixes (route it upstream):

nginx version: 1.30.5 (--with-debug build)
config: [minimal reproduction]
error.log excerpt: worker process exited on signal 11 (core dumped)
gdb bt: [the stack]
core: available on request

Try it yourself

• kill -SEGV $(pgrep -P $(cat /run/nginx.pid) | head -1) — then check every fact from §3: core written (size real), respawn logged, and the very next request a 200.

• Change worker_rlimit_core to 0, reload, re-SEGV — the alert now says signal 11 without (core dumped). The budget is the switch.

• Leave no core behind in git: clean up after each debugging session, and your repos should .gitignore logs/core* from day one.

Q&A

Q: Cores on a full-disk edge — how do I ship them? A: coredump_filter (via /proc/<pid>/coredump_filter) trims what's dumped (shared, anonymous pages only is the usual compromise); systemd-coredump compresses and rotates; or rlimit small + "reproduce in staging." Pick your poison before the crash picks it for you.

Q: Does worker_rlimit_core need root? A: Raising the soft limit up to the hard one: no — the master does it for workers. Raising the hard limit above what the OS gave nginx's user: root (or systemd's LimitCORE=).

Q: The master died instead — cores for IT? A: Master crashes are kernel/ulimit/upgrade-script territory; same mechanics (core_pattern is global), but the answer is almost always in what launched it (unit file, logs, OOM killer — /var/log/kern.log's signature).

Q: No gdb on the box — analyze the core elsewhere? A: Yes — that's the play: scp the 2 MB core plus your nginx-debug binary to any gdb machine (matching OS/arch, or a container twin) and run the recipe there. The artifact is portable even when the debugger isn't present at the scene.

Takeaways

• A core is the frozen crime scene — memory, registers, stack — and NGINX's two directives ( worker_rlimit_core, working_directory) are what make it appear where you'll find it.

• Relative working_directory resolves against the daemonized cwd, not the prefix — our [alert] chdir("logs") failed receipt; launch nginx cwd'd to its prefix, or use absolute paths in production.

• The live receipts: 2 MB core written, master's signal 11 (core dumped) alert, instant respawn, next request 200 — invisible to users, recorded for you.

• The gdb recipe ( bt, bt full, thread apply all bt) turns the artifact into a routing decision — core bug vs module bug vs platform — which is what every crash report actually needs to say.