incident: /var/log full from plexstreams polling a dead address

plexstreams was configured for http://192.168.31.2:32400, on the macvlan
shim subnet created in go for a Pi-hole that AdGuard replaced. Plex runs with
host networking on 192.168.10.20:32400. Each failed poll logged three PHP
warnings; 99,926 polls filled a 48 MB phplog, 99.95% of it those warnings.

Logging had been dead box-wide for six days: the last entry was written when
the filesystem filled, not when the errors stopped, so syslog for 28 Oct to
3 Nov is incomplete.

Config repointed and backed up, logs truncated rather than deleted so writers
holding them open reclaim space immediately. syslog left intact.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

docs/incidents/2026-11-03-var-log-full-plexstreams.md

@@ -0,0 +1,80 @@
+# /var/log full — plexstreams polling a dead address
+
+**Date:** 2026-11-03 (silent since 2026-10-28)
+**Severity:** P2
+**Status:** Resolved
+
+## Symptoms
+
+`/var/log` on Unraid at **100% of its 128 MB tmpfs**. Any process trying to log
+failed with `errno=28 No space left on device`. Found by accident while
+installing an unrelated script that tried to write a log file there.
+
+```
+tmpfs  128M  128M  0  100%  /var/log
+  48M  phplog
+  37M  syslog
+  31M  tailscale.log
+ 6.6M  tailscale-utils.log
+```
+
+## Timeline
+
+| Time | Event |
+|------|-------|
+| 2026-10-10 16:21 | First `phplog` entry. Plex had moved to host networking; plexstreams kept its old address |
+| 2026-10-10 → 10-28 | 99,926 failed polls, three PHP warnings each, ~every 15 s |
+| 2026-10-28 07:12 | Last entry written — **not** because errors stopped, but because the filesystem filled |
+| 2026-10-28 → 11-03 | Logging silently dead across the box for six days |
+| 2026-11-03 22:20 | Found, diagnosed, fixed |
+
+## Root Cause
+
+The **plexstreams** plugin was configured with
+`HOST="http://192.168.31.2:32400"`. That address is on the macvlan shim subnet —
+the same dead `192.168.31.0/24` range created in `/boot/config/go` for a Pi-hole
+that AdGuard replaced. Nothing answers there.
+
+Plex itself runs with **host** networking and answers on `192.168.10.20:32400`
+(HTTP 200). plexstreams was never updated when Plex moved.
+
+Each failed poll returned an empty body, so `simplexml_load_string()` failed and
+logged three warnings:
+
+```
+PHP Warning: simplexml_load_string(): Entity: line 1: parser error : Document is empty
+  in /usr/local/emhttp/plugins/plexstreams/includes/common.php on line 180
+```
+
+299,778 of 299,934 lines — **99.95% of the file** — were those three warnings.
+
+## Fix
+
+1. `HOST` and `hostbox` in `/boot/config/plugins/plexstreams/plexstreams.cfg`
+   repointed to `http://192.168.10.20:32400`, with `ALIAS-192_168_10_20="unraid"`
+   added so the UI still shows the right name. Backup at `plexstreams.cfg.bak-*`.
+2. `phplog`, `tailscale.log` and `tailscale-utils.log` **truncated**, not
+   deleted — deleting a log a process holds open frees no space until that
+   process restarts, and the writer keeps appending to an unlinked inode.
+   `syslog` was left intact as the only remaining record of the period.
+
+Result: `/var/log` 100% → 35%. Verified over 60 s (four poll cycles) that
+`phplog` stayed at 0 bytes with no new plexstreams warnings.
+
+## Open items found in the same log
+
+- **`appdata.backup` exhausted PHP's 256 MB limit 18 times**
+  (`Allowed memory size of 268435456 bytes exhausted`). Appdata backups may be
+  failing. Not investigated.
+- **`zfs.master`**: `Trying to access array offset on null` ×130 — cosmetic
+  plugin bug.
+
+## Lessons
+
+- The six-day gap is the real damage: a full `/var/log` is silent. Nothing
+  alerts on it, and `syslog` for 2026-10-28 to 11-03 is incomplete.
+- Worth a periodic check on `/var/log` usage; `go` already installs an rsyslog
+  noise filter, so log volume has been a problem here before.
+- The dead `192.168.31.0/24` shim keeps surfacing. It was also the pointless
+  route Unraid's Tailscale node used to advertise, removed the same day. The
+  shim itself is still created in `go` for a Pi-hole that no longer exists.