12.5 Conditional Logging & Error Discipline

Advanced

What you will learn

1. access_log ... if=$var — conditional logging with a map as the gatekeeper.

2. The audit-trail pattern: 4xx/5xx only, separate file, structured format.

3. error_log severity culture: what belongs at warn vs error vs notice — and when to bend it.

4. Rotation, buffering, retention: the operational lifecycle.

1. Log only what hurts

map $status $log_error_traffic {
    ~^[45]   1;      # 4xx/5xx -> log it
    default  0;      # 2xx/3xx -> skip entirely
}

server {
    listen      9402;
    access_log  logs/audit.log  golden_json  if=$log_error_traffic;
    ...
}

if=$var (1.7.0+) skips the write when the variable is empty or "0". Paired with a map on $status, you get an audit trail that contains only failures — the file your incident channel tails. The receipt (3 requests sent: 200, 500, 404 — exactly 2 lines written):

{"uri":"/fail/nope","status":500}
{"uri":"/missing/thing","status":404}

Gate on anything a map can read — the pattern generalizes:

What conditional logging deliberately does NOT do: record that it skipped. Your success-rate metrics need the denominator — so the usual production shape is two logs: the full golden log (everything, buffered, ephemeral retention) plus the conditional audit log (small, curated, long retention). The :9400 and :9402 — the two example servers from this module — are exactly those two logs.

2. error_log severity culture

Four levels do 95% of the work:

The discipline that scales:

• Production default warn . error alone hides the quarantine/retry warnings that predict tomorrow's outage; notice at scale is a firehose.

• Per-location overrides are legal — error_log inside a location — and that's how you get Module 11's rewrite_log narration for one suspicious door without re-leveling the whole instance.

• Alert on error rate, trend warn rate. A warn-storm (spiking temporarily disabled lines) is the classic leading indicator; errors are the incident itself.

• Every message NGINX emits in this state carries *N connection numbers and full context — grep by *59 and you have the whole session’s story. Teach your on-call to use them.

3. The operational lifecycle

Rotation. nginx keeps log files open; renaming the file does not redirect the writes. The canonical dance (logrotate does this for you):

mv access.log access.log.1 && nginx -s reopen    # USR1: workers open fresh files

s reopen (USR1) is safe at any traffic level — no drops, no restart (Module 1's signal table, earning its keep).

Buffering. One line: buffer=64k flush=1s buys throughput at the price of up to flush seconds of lines lost on a hard crash — the golden log takes that trade, the audit log never does, and a clean -s reopen above flushes everything, so rotation and buffering coexist; the full trade is Lesson 12.1's write mechanics.

Retention tiers (tie back to the two-logs shape):

Capacity honesty: log volume is storage and I/O — a burst-rate estimate (rps × line size) belongs next to your disk alerts. The Lesson 12.1 math (400 B × 50k rps = 20 MB/s) is the number that keeps a log storm from becoming the outage.

Try it yourself

• curl -s http://127.0.0.1:9402/fail/nope http://127.0.0.1:9402/missing/thing — then read the audit log: the 500 and the 404 land in it, and nothing else leaks (the receipt above shows the same discipline at work).

• Bend the gate: map ~^[45] → ^5 only (4xx become silent), reload, re-run the probes, read the diff in audit.log.

• Rotate by hand: mv logs/golden.log logs/golden.log.1; nginx -s reopen — watch a fresh file appear and old one stop mid-write. Then set buffer=16k flush=1s on the golden log and reload — feel the tradeoff on your own disk.

Q&A

Q: Is if=$var evaluated before or after the response exists? A: At log phase — after the response is complete. That's why it can gate on $status: the verdict exists by then. You cannot conditionally log "requests that are ABOUT to be slow."

Q: Why not just filter downstream — ship everything, query errors? A: You can, and often should. Conditional logging is for when volume or sensitivity (PII in request URIs) makes "record it at all" the wrong default. Two logs, two contracts — the shape that survives audits.

Q: error_log — can it be conditional? A: No if= equivalent; severity is the filter. Per-location error_log files are the scoping tool.

Q: syslog vs files, final answer? A: Files + tailer for resilience (disk buffers bursts), syslog for collector-owned shops (no local disk pressure). Both patterns ship the same JSON line — choose by operational topology, not fashion.

Takeaways

• access_log ... if=$map_verdict builds curated audit trails — errors-only, one-tenant-only, sampled — while the full golden log keeps the denominators.

• Severity culture: warn in production, alerts on error rate, trends on warn rate, per-location overrides for surgical verbosity.

• Rotation = rename + nginx -s reopen; buffering buys throughput with bounded loss — never on the audit log.

• Retention is tiered by log purpose: days for the full stream, months for the curated one — and log volume itself is a capacity number you own.