5.3 Failure Handling: Retries, Quarantine, and Backups

Intermediate

What you will learn

1. proxy_next_upstream — the retry policy, its defaults, and its hard safety limits.

2. max_fails / fail_timeout — passive health checking, with the double meaning of fail_timeout demystified.

3. The quarantine timeline, read from real logs.

4. backup servers — the last line, and when they're a trap.

5. The failure taxonomy revisited: 502 vs 504 vs 503 from the pool's perspective.

1. The retry policy: proxy_next_upstream

When a peer fails mid-attempt, NGINX can silently try the next one — the request survives, the client never knows. What counts as "failed" is the policy:

proxy_next_upstream    error timeout;        # the DEFAULT
proxy_next_upstream_tries   2;               # peers tried per request (0 = all)
proxy_next_upstream_timeout 60s;             # total budget incl. retries

The safety rails deserve their fame:

• Non-idempotent methods (POST!) are never retried by default. A timed-out POST may have succeeded on the backend (the response was lost, not the work) — retrying a payment is how double-charges happen. Opt in per location only when the operation is provably idempotent.

• Retries happen before any response byte exists. Once the client has received so much as a status line, switching peers is impossible — you can't un-send a response. Mid-body timeouts are terminal (504).

• _tries and _timeout bound the blast radius — without them, a 10-peer pool with all-but-one dead means every request eats 9 failed connects before its 502.

2. max_fails / fail_timeout — passive health checks

Here is the failure pool — the whole lesson in four lines:

upstream pool-fail {
    server 127.0.0.1:9998 max_fails=1 fail_timeout=10s;   # dead port
    server 127.0.0.1:9001 max_fails=1 fail_timeout=10s;   # alive
}

fail_timeout means TWO things (the single most misread parameter in NGINX):

1. the window in which max_fails failures are counted, and

2. how long the peer stays quarantined once the count trips.

With max_fails=1 fail_timeout=10s: one failure inside a 10 s window → the peer is marked temporarily disabled for the next 10 s. The defaults ( max_fails=1, fail_timeout=10s) are usually sane; the knobs let you tune sensitivity (a flaky peer you want out fast: low max_fails, long fail_timeout; a peer with hiccups you tolerate: higher max_fails).

Passive means NGINX learns only from real traffic — no probes in the background (that's Plus's active checks, Lesson 5.5). Consequences: a peer that receives no traffic can't be marked down (nobody noticed), and the first request after a death pays the failed-connect tax.

3. The quarantine timeline — real logs

What actually happens, receipt by receipt. The first request after startup:

$ curl -s -H 'Host: lb.example.test' http://127.0.0.1:8500/fail/x | jq -r .server
b-node-1

The client saw a normal 200. The access log tells the secret:

127.0.0.1 [22/Sep/2026:21:00:57 +0000] "GET /fail/x HTTP/1.1" 200
  upstream=127.0.0.1:9998, 127.0.0.1:9001 ct=-, 0.000 rt=0.000, 0.001

upstream_addr lists both peers, comma-separated: NGINX tried 9998, got connection refused (that's the error retry flag firing), and retried on 9001 — all inside one request, invisible to the client. ( ct = connect times per peer: the dead one shows -, the live one 0.000.)

And the error log's quarantine receipt:

2026/09/22 21:00:57 [warn] 5121#5121: *61 upstream server temporarily
disabled while connecting to upstream, …, upstream: "http://127.0.0.1:9998/x"

From that instant, for fail_timeout seconds, the scheduler skips the peer entirely — the second request goes straight to 9001 with no connect attempt (its upstream_addr shows a single address). After the window expires, the peer is readmitted on probation: one more failure re-quarantines it. This try-readmit-fail cycle is why fail_timeout doubles as the detection heartbeat for a dead-but-traffic-bearing peer.

4. backup — the last line of defense

upstream pool-backup {
    server 127.0.0.1:9999 max_fails=1 fail_timeout=30s;   # dead primary
    server 127.0.0.1:9002 backup;                          # never chosen…
}

$ curl -s … http://127.0.0.1:8500/backup/x | jq -r .server
b-node-2          # the backup, carrying 100% of traffic

Backup peers are excluded from all scheduling until every primary is unavailable (down, quarantined, or failing) — then they take everything, and traffic reverts when a primary recovers. Two production shapes:

• The cold spare: a smaller/cheaper instance (or a static maintenance page server!) that exists only for the outage window. NGINX-to-NGINX: server 127.0.0.1:9500 backup; where 9500 serves a "we'll be right back" page — a two-line poor-man's failover that has saved more launches than any HA framework.

• Degraded mode: the backup is a real but lesser service (cache-only node, read-only replica) — capacity drops, availability doesn't.

5. The taxonomy, completed

With pools in hand, the Lesson 4.6 table gains its third row's full meaning:

That 503 (with the classic "no live upstreams" error-log line) is the pool's honest admission — and the moment your monitoring should be screaming (Module 12 wires exactly that).

Try it yourself

1. Replay §3's timeline. Restart NGINX ( systemctl restart nginx) to reset counters, then: request /fail/x, read the access log's dual upstream_addr, read the error log's quarantine line, request again and see the single-addr line.

2. Watch the probation cycle. With fail_timeout=10s, hit /fail/x every 3 seconds for 30 seconds and chart (in your notes) which requests show two upstreams (readmission probes) vs one (quarantine).

3. Break the retry rails. Set proxy_next_upstream off; on /fail/ and feel the 502 arrive with the dead peer alone in the log. Restore. Then try non_idempotent with a POST to see why it exists.

4. Build the maintenance-page failover. Add a second server block on :8501 serving a static page, make it the backup of a pool whose primary is a dead port — full outage UX in ~10 lines.

Q&A

Q1. Why is a timed-out POST not retried even though a timed-out GET is?

Q2. max_fails=3 fail_timeout=30s on a peer that dies instantly. Walk the first ~90 seconds of its traffic.

Q3. All primaries are quarantined; no backup configured. What does the client see, and what does the error log say?

Q4. Your backup server has 2 GB RAM vs primaries' 32 GB. When is it fit for purpose, and when is it a liability?

Takeaways

• Retries are policy ( proxy_next_upstream error timeout default) with rails: no mid-response switches, no non-idempotent retries, bounded tries/time.

• fail_timeout = counting window and quarantine length; max_fails trips it. Passive checks learn only from live traffic — first request pays the tax.

• The receipts are in your logs: comma-joined upstream_addr (retry), [warn] temporarily disabled (quarantine), absent addresses (skipped).

• backup = out-of-rotation spare that inherits everything when primaries fail — and must be cold-start-proof to be trusted.

• 502 = tried and failed; 504 = tried and silent; 503 = nobody left to try. Your monitoring should distinguish all three.