1.5 Signals, Reloads, and Zero-Downtime

Basic

What you will learn

1. The NGINX command line that matters: -t, -T, -s, -g, -p, -c.

2. The complete signal table: what each signal does, and to whom.

3. Reload internals — the six-step dance, straight from a real error log.

4. Log rotation without losing lines ( USR1).

5. Binary upgrades without dropping connections ( USR2 + WINCH).

6. The reload safety checklist that separates pros from tourists.

1. The command line that matters

nginx -t                       # TEST config: syntax + file access + temp dirs
nginx -T                       # test + DUMP the fully-expanded config
nginx -g "daemon off;"         # run with a global directive override
nginx -p /path/prefix/ -c conf/nginx.conf    # explicit prefix & config
nginx -s reload                # send a signal via the pidfile
nginx -s stop                  #   (same as: kill -TERM $(cat …/nginx.pid))

Two flags deserve respect right now:

• -t is a full dress rehearsal, not a syntax check. It parses, resolves includes, opens log files, and creates temp directories — Lesson 1.3's mkdir() failure came from exactly this thoroughness. Make it the last line of every deploy script.

• -T is your truth serum. It prints the effective configuration after all includes and inheritance — the only reliable answer to "what is this server actually running?". Lesson 2.6 turns it into a methodology.

2. The signal table

Signals go to the master. (Two exceptions noted below.)

The two signals sent to workers directly (rarely needed):

• TERM to a worker: it exits immediately — in-flight requests on it are dropped (the master will not treat it as a crash, but will still respawn per config);

• QUIT to a worker: graceful, like master-level QUIT.

3. Reload internals: the six-step dance

nginx -s reload → master receives SIGHUP. Then:

1. Master re-reads the config from disk (the -c file, all includes). If parsing/validation fails, the master logs an error and keeps the old config running — a failed reload is a no-op, not an outage. (Verify this on a running instance: add a typo, reload, watch traffic continue.)

2. Master opens new log files if their paths changed.

3. Master evaluates worker_processes and forks the new generation of workers from the new config.

4. New workers inherit the listening sockets — the same file descriptors. Not one socket closed, not one port renegotiated.

5. Master sends QUIT to old workers.

6. Old workers drain: finish in-flight requests, honor keep-alive until the client disconnects or keepalive_timeout expires, then exit.

Here is the entire dance, as it actually happened (captured from /var/log/nginx/error.log):

$ nginx -s reload

$ grep -A9 "SIGHUP" /var/log/nginx/error.log | head -10
2026/09/22 20:12:33 [notice] 2397#2397: signal 1 (SIGHUP) received from 2447, reconfiguring
2026/09/22 20:12:33 [notice] 2397#2397: reconfiguring
2026/09/22 20:12:33 [notice] 2397#2397: using the "epoll" event method
2026/09/22 20:12:33 [notice] 2397#2397: start worker processes
2026/09/22 20:12:33 [notice] 2397#2397: start worker process 2449     ← new gen
2026/09/22 20:12:33 [notice] 2397#2397: start worker process 2450     ← new gen
2026/09/22 20:12:33 [notice] 2399#2399: gracefully shutting down       ← old gen
2026/09/22 20:12:33 [notice] 2398#2398: gracefully shutting down       ← old gen
2026/09/22 20:12:33 [notice] 2399#2399: exiting
2026/09/22 20:12:33 [notice] 2398#2398: exiting

And the two assertions that prove it was zero-downtime, from the same test run:

$ M=$(cat …/nginx.pid); echo "master before: $M"
master before: 2397
$ nginx -s reload; sleep 1; echo "master after: $(cat …/nginx.pid)"
master after: 2397                          # unchanged
$ pgrep -P $M
2449 2450                                   # workers rotated (before: 2398 2399)
$ for i in $(seq 10); do curl -s -o /dev/null -w '%{http_code} ' …/; done; echo
200 200 200 200 200 200 200 200 200 200     # 10/10 across the reload window

What reload does NOT change

• Listening sockets — unchanged, inherited.

• Shared-memory zone contents — preserved when the zone directive is byte-identical (Lesson 1.4 §5). A cache or rate-limit table survives.

• The binary — reload re-reads config, not the executable. Upgrading the binary live is §5 below.

• nginx -s -based scripts that assume instant worker turnover — see the linger window above.

4. Log rotation with USR1

The classic pattern (logrotate does this for you, but its essence is two commands):

mv access.log access.log.1        # rename — nginx keeps writing to the
                                  #   same inode, now under a new name
kill -USR1 $(cat …/nginx.pid)     # master tells workers: reopen your logs

Workers open() the configured log paths again, creating fresh access.log files; the renamed files are now complete and static — ship them, compress them, burn them. Without USR1, renaming alone means NGINX keeps filling the renamed file (open file descriptors follow inodes, not names) until the next reopen — a legendary source of "why is my new log empty and my rotated file still growing".

5. Binary upgrades without dropping connections ( USR2 + WINCH)

The most cinematic NGINX trick: replace the running executable while it serves traffic.

1. Install the new binary at a NEW path (e.g. /usr/sbin/nginx.new)
   — overwriting the running executable's file is NOT allowed by the kernel
     (and on some filesystems will fail outright).

2. kill -USR2 <master pid>
   → master: renames the old pidfile to nginx.pid.oldbin
   → forks a NEW master from the NEW binary, which spawns ITS workers
   → now two full generations serve the same sockets simultaneously

3. Verify the new version behaves (health checks, canary requests).

4. kill -WINCH <old master>
   → old master's workers drain gracefully (exactly like step 6 of reload)

5. kill -QUIT <old master>
   → old master exits; the new master rewrites nginx.pid. Done.

6. The reload safety checklist

Before sending HUP in production, run this ritual:

nginx -t                      # 1. syntax + files + temp dirs — must pass
nginx -T > /tmp/new.conf      # 2. eyeball the EFFECTIVE config (includes!)
grep -c "listen" /tmp/new.conf# 3. sanity: every expected listener present
nginx -s reload               # 4. now, and only now
tail -f …/error.log           # 5. watch for: "reconfiguring" … no [emerg]
ps aux | grep [n]ginx         # 6. two generations → one generation

nginx -t first, always — steps 1, 4 and 5 belong in a wrapper that runs them for every instance and refuses to reload onto a broken config (the old config stays live). That is precisely the behavior you should demand from any deploy tooling you write.

Try it yourself

On a running instance, do the manual version of §3 to feel it:

M=$(cat /run/nginx.pid)      # note the master
pgrep -P $M                               # note the workers

1. A failed reload is a no-op. Break the config, reload, watch traffic:

   echo "this is not; valid nginx;" >> /etc/nginx/nginx.conf
   curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8100/   # 200

(The echo above appends to the live /etc/nginx/nginx.conf — remove the junk line when you're done; the next step assumes you did.)

1. A successful reload rotates workers. Fix the typo, reload, compare worker PIDs; confirm the master PID is unchanged.

2. Watch the drain window. Open a keep-alive connection ( curl -s http://127.0.0.1:8100/ http://127.0.0.1:8100/ reuses one), then reload and re-run ps — old workers visible until keep-alive expires.

3. Rotate logs by hand (§4): rename /var/log/nginx/access.log, send USR1, generate traffic, and confirm the new file receives lines.

Q&A

Q1. Your CI runs nginx -s reload and the site keeps serving — but the new feature isn't active. Name the two most likely causes.

Q2. Why does the §6 safety checklist run nginx -t first, given the master already validates on HUP?

Q3. After USR1 (log rotation), what does a worker do with an in-flight request whose response is partially written to the old log fd?

Q4. During a USR2 upgrade, what happens to connections assigned to old workers at the moment you send WINCH?

Takeaways

• The operational surface is small and complete: -t before everything, -T for truth, signals to the master, HUP for config, USR1 for logs, USR2+ WINCH for binaries.

• A reload is two worker generations sharing inherited sockets — new workers take new requests, old workers drain keep-alives, the master and all zones survive. Failed reloads are safe no-ops.

• Log rotation is a rename + USR1; inode semantics explain every "empty new log" mystery you'll ever debug.

• The zero-downtime guarantee is architectural (Lesson 1.4's socket ownership), not heroic — which is why it works every single time.

• Ritualize the safety checklist; muscle memory is what survives 3 a.m.

Module 1 complete. You now hold the complete mental model: what NGINX is, how it's built, where its files live, how its processes cooperate, and how to command them. Next: Module 2 — the configuration language itself: directives, contexts, the matching algorithm, variables, and the inheritance rule that explains 80% of "why isn't my directive working".