16.3 /proc Forensics

Advanced

What you will learn

1. The /proc mental model: the kernel's API surface for processes.

2. The interrogation kit: stat, limits, fd, wchan — each answering a real debugging question.

3. The receipts, gathered live from a running instance.

4. Where /proc ends (and what the missing tools would add).

1. The model

/proc is a filesystem where every PID is a directory whose files are live kernel data structures wearing file permissions. If you can cat it, you can read it — and for your own processes, you can. That single property makes it the universal rootless debugger: every container, every locked-down prod host, every environment where ptrace is off-limits.

A running instance, from nothing but the filesystem (real receipts):

master 17433, workers: 17661 17724

One pgrep -P from the pidfile — but each of those PIDs is now a directory full of answers.

2. The interrogation kit

/proc/<pid>/stat — the scheduler's view. State, CPU consumption, memory, parentage — atomically sampled:

state=S utime+stime=32 jiffies

The state field is the quick vital sign: R running on a CPU, S sleeping (waiting for an event — healthy for an epoll loop), D uninterruptible disk sleep (the one that matters: %D climbing means IO pressure or an NFS-shaped hang), Z zombie (a parent not reaping — should be impossible for nginx's master).

utime+stime in jiffies (usually 100/second) is CPU time — sample it twice, divide by elapsed, and you have utilization without any profiler. (The sampler in Lesson 16.5 is exactly this, at 200 Hz.)

/proc/<pid>/limits — the ceilings. Module 9's fd math, read directly from the enforced values:

Max open files            1024                 100000               files

Soft limit 1024, hard 100000 — and now the "worker at 1023 fds, new connections stall" failure mode (9.1) isn't a theory, it's a wc -l /proc/<pid>/fd away from confirmation.

/proc/<pid>/fd/ — what's actually held. Every open descriptor as a symlink (sockets, files, pipes, the error log):

0 1 10 11 12 2 3 4 ...
ls -l /proc/<pid>/fd | grep socket | wc -l     # the live connection count

This is the connection-pool exhaustion receipt: count sockets, compare to limits, watch it climb during the incident window. ls -l on each fd shows where it points — including, with ss's help (Module 5's keepalive receipts), which upstream they belong to.

/proc/<pid>/wchan — the wait channel. The kernel function the process is currently sleeping in. One word, enormous diagnostic weight:

do_epoll_wait

That's a healthy nginx worker's resting heartbeat — the event loop from Lesson 1.4, visible in the kernel's own vocabulary. Other values you might meet: pipe_read (waiting on a pipe — who's the other end?), futex_wait_queue_me (lock contention), poll_schedule_timeout. Combined with the sampler's frequency counts, wchan is a rootless one-line stack trace.

The honorable mentions, useful enough to know exist:

3. The receipts, assembled

A live capture run, condensed into the forensics card you can run on ANY host (real output):

--- the process tree ---
master 17433, workers: 17661 17724
--- worker fds ---
0 1 10 11 12 2 3 4 ...
--- fd ceiling ---
Max open files  1024  100000  files
--- scheduler ---
state=S utime+stime=32 jiffies
--- wait channel ---
do_epoll_wait

Read as a story: two workers exist, hold a dozen fds each, far from their 1024 soft ceiling, are sleeping in epoll — i.e., idle and healthy, having burned 0.32s of CPU total. Every production health question about an nginx process is a variation on reading this card and knowing what "wrong" looks like at each line.

4. Where /proc ends

Honesty about the boundary:

• No syscall trace: /proc shows state, not behavior over time — strace's connect(fd, ...) sequence can show a failed connection attempt that state-sampling only implies. (Mitigation: nginx's debug log narrates the equivalent application-level events — 16.2 — which for NGINX questions is usually enough.)

• No user-space stacks: wchan is kernel-only; a CPU-hog in OpenSSL can't be named without gdb/perf (16.5's sampler shows you the shape, not the symbol).

• Sampling ≠ tracing: you see snapshots. Rare events between samples vanish — raise frequency, or accept the statistical view.

The bridge when you DO get root or ptrace: strace -f -p <master+workers> -e trace=network,file -o trace.txt and cat /proc/<pid>/syscall (the live syscall, needs ptrace rights a locked-down host lacks). Keep the recipes; earn the receipts you can.

Try it yourself

• M=$(cat /run/nginx.pid); pgrep -P $M — with master and workers in hand, walk every file in the §2 kit against them; the receipts above are your expected output.

• Run the card yourself: M=$(cat /run/nginx.pid); for p in $M $(pgrep -P $M); do echo "== $p =="; cat /proc/$p/wchan; echo; done

• Watch state move: watch -n0.1 'awk "{print \$3}" /proc/<worker>/stat' while you curl in a loop — R/S flickering is the event loop breathing.

Q&A

Q: Can I read OTHER users' processes? A: Only what permissions allow (generally your own). That's the security model AND the rootless debugging contract: same-uid = full visibility.

Q: /proc/<pid>/stat field 3 vs status ? A: Same data, different packaging — stat is one machine-readable line (fast to sample), status is the human-readable spread. Samplers use stat; humans skim status.

Q: My worker shows D-state for seconds — panic? A: %D sustained = the IO-pressure signature (page faults on a slow disk, a full tmpfs, NFS). Not an nginx bug — an environment fact nginx is politely enduring. Check what the fd table and mounts say before blaming the binary.

Q: Why does ls /proc/<pid>/fd sometimes hang? A: Reading fd symlinks can briefly contend with the process opening/closing; and unresponsive NFS mounts behind an fd will make even ls wait — which is itself a diagnosis (the D-state story, experienced from the observer side).

Takeaways

• /proc is the kernel's file-permission-gated API: same-uid processes are fully interrogable — the universal rootless debugger.

• The kit: stat (state + CPU), limits (ceilings), fd/ (what's held), wchan (kernel wait channel — the one-word stack trace). Together: the health card for any nginx fleet.

• do_epoll_wait is the healthy heartbeat; sustained D-state is the IO-distress signature; fd-count vs Max open files is the exhaustion receipt.

• /proc gives state, not behavior: no syscall sequence, no user stacks. The debug log covers NGINX-level behavior; strace/gdb/perf recipes wait for a friendlier environment — and you know exactly what they'd add.