incident: /var/log full from plexstreams polling a dead address

plexstreams was configured for http://192.168.31.2:32400, on the macvlan
shim subnet created in go for a Pi-hole that AdGuard replaced. Plex runs with
host networking on 192.168.10.20:32400. Each failed poll logged three PHP
warnings; 99,926 polls filled a 48 MB phplog, 99.95% of it those warnings.

Logging had been dead box-wide for six days: the last entry was written when
the filesystem filled, not when the errors stopped, so syslog for 28 Oct to
3 Nov is incomplete.

Config repointed and backed up, logs truncated rather than deleted so writers
holding them open reclaim space immediately. syslog left intact.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

docs/CHANGELOG.md

@@ -13,6 +13,12 @@
 - **[SERVICE]** Exit node advertised from Unraid as a fallback to the hAP, and approved in the Tailscale admin console the same day. The tailnet now has two exit nodes: `mikrotik-tailscale` (100.74.219.35, primary) and `xtrm-unraid` (100.100.208.69, fallback). Neither has been exercised in anger from the Mac — without `--exit-node-allow-lan-access` an exit node cuts the client's own LAN access.
 - Detail, including the root-only `AllowTcpForwarding` behaviour in Unraid's stock sshd config, in `docs/09-TAILSCALE-VPN.md`.
 
+### Incident: /var/log full, plexstreams polling a dead address
+- **[INCIDENT]** `/var/log` found at 100% of its 128 MB tmpfs. The plexstreams plugin was configured for `http://192.168.31.2:32400` — the dead macvlan shim subnet created in `go` for a Pi-hole that AdGuard replaced. Plex runs with host networking and answers on `192.168.10.20:32400`. Every failed poll logged three PHP warnings from `simplexml_load_string()`; 99,926 polls between 10 Oct and 28 Oct produced 299,778 lines, 99.95% of a 48 MB `phplog`.
+- **[INCIDENT]** Logging across the box had been silently dead for six days — the last entry is 28 Oct 07:12, written when the filesystem filled, not when the errors stopped. Treat `syslog` for 2026-10-28 to 11-03 as incomplete.
+- **[SERVICE]** Fixed: `HOST`/`hostbox` repointed to `192.168.10.20:32400` with a matching alias, config backed up. Logs truncated rather than deleted (`syslog` left intact). `/var/log` 100% -> 35%, verified clean over four poll cycles. Detail in `docs/incidents/2026-11-03-var-log-full-plexstreams.md`.
+- Also seen in that log and **not** addressed: `appdata.backup` exhausting PHP's 256 MB limit 18 times, and a cosmetic `zfs.master` null-offset warning x130.
+
 ### Network: Tailscale SSH ACL set to accept
 - **[NETWORK]** Tailnet SSH policy changed from Tailscale's default `action: "check"` to `accept` for `autogroup:member` -> `autogroup:self`. Before the change the control plane minted ~12 h per-device grants and prompted for browser re-auth when they lapsed, which would stall any unattended connection. Effective policy is now a single rule: accept for all 10 tailnet nodes, no expiry, with local and remote port forwarding and agent forwarding allowed.
 - **[NETWORK]** Verified unattended with `BatchMode=yes`: a plain command over Tailscale SSH, a `-L` port forward (HTTP 302 from the Unraid web UI) and the Herdr fallback leg all ran without a prompt.

docs/incidents/2026-11-03-var-log-full-plexstreams.md

@@ -0,0 +1,80 @@
+# /var/log full — plexstreams polling a dead address
+
+**Date:** 2026-11-03 (silent since 2026-10-28)
+**Severity:** P2
+**Status:** Resolved
+
+## Symptoms
+
+`/var/log` on Unraid at **100% of its 128 MB tmpfs**. Any process trying to log
+failed with `errno=28 No space left on device`. Found by accident while
+installing an unrelated script that tried to write a log file there.
+
+```
+tmpfs  128M  128M  0  100%  /var/log
+  48M  phplog
+  37M  syslog
+  31M  tailscale.log
+ 6.6M  tailscale-utils.log
+```
+
+## Timeline
+
+| Time | Event |
+|------|-------|
+| 2026-10-10 16:21 | First `phplog` entry. Plex had moved to host networking; plexstreams kept its old address |
+| 2026-10-10 → 10-28 | 99,926 failed polls, three PHP warnings each, ~every 15 s |
+| 2026-10-28 07:12 | Last entry written — **not** because errors stopped, but because the filesystem filled |
+| 2026-10-28 → 11-03 | Logging silently dead across the box for six days |
+| 2026-11-03 22:20 | Found, diagnosed, fixed |
+
+## Root Cause
+
+The **plexstreams** plugin was configured with
+`HOST="http://192.168.31.2:32400"`. That address is on the macvlan shim subnet —
+the same dead `192.168.31.0/24` range created in `/boot/config/go` for a Pi-hole
+that AdGuard replaced. Nothing answers there.
+
+Plex itself runs with **host** networking and answers on `192.168.10.20:32400`
+(HTTP 200). plexstreams was never updated when Plex moved.
+
+Each failed poll returned an empty body, so `simplexml_load_string()` failed and
+logged three warnings:
+
+```
+PHP Warning: simplexml_load_string(): Entity: line 1: parser error : Document is empty
+  in /usr/local/emhttp/plugins/plexstreams/includes/common.php on line 180
+```
+
+299,778 of 299,934 lines — **99.95% of the file** — were those three warnings.
+
+## Fix
+
+1. `HOST` and `hostbox` in `/boot/config/plugins/plexstreams/plexstreams.cfg`
+   repointed to `http://192.168.10.20:32400`, with `ALIAS-192_168_10_20="unraid"`
+   added so the UI still shows the right name. Backup at `plexstreams.cfg.bak-*`.
+2. `phplog`, `tailscale.log` and `tailscale-utils.log` **truncated**, not
+   deleted — deleting a log a process holds open frees no space until that
+   process restarts, and the writer keeps appending to an unlinked inode.
+   `syslog` was left intact as the only remaining record of the period.
+
+Result: `/var/log` 100% → 35%. Verified over 60 s (four poll cycles) that
+`phplog` stayed at 0 bytes with no new plexstreams warnings.
+
+## Open items found in the same log
+
+- **`appdata.backup` exhausted PHP's 256 MB limit 18 times**
+  (`Allowed memory size of 268435456 bytes exhausted`). Appdata backups may be
+  failing. Not investigated.
+- **`zfs.master`**: `Trying to access array offset on null` ×130 — cosmetic
+  plugin bug.
+
+## Lessons
+
+- The six-day gap is the real damage: a full `/var/log` is silent. Nothing
+  alerts on it, and `syslog` for 2026-10-28 to 11-03 is incomplete.
+- Worth a periodic check on `/var/log` usage; `go` already installs an rsyslog
+  noise filter, so log volume has been a problem here before.
+- The dead `192.168.31.0/24` shim keeps surfacing. It was also the pointless
+  route Unraid's Tailscale node used to advertise, removed the same day. The
+  shim itself is still created in `go` for a Pi-hole that no longer exists.