diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index e97710d..5af8987 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -13,6 +13,12 @@ - **[SERVICE]** Exit node advertised from Unraid as a fallback to the hAP, and approved in the Tailscale admin console the same day. The tailnet now has two exit nodes: `mikrotik-tailscale` (100.74.219.35, primary) and `xtrm-unraid` (100.100.208.69, fallback). Neither has been exercised in anger from the Mac — without `--exit-node-allow-lan-access` an exit node cuts the client's own LAN access. - Detail, including the root-only `AllowTcpForwarding` behaviour in Unraid's stock sshd config, in `docs/09-TAILSCALE-VPN.md`. +### Incident: /var/log full, plexstreams polling a dead address +- **[INCIDENT]** `/var/log` found at 100% of its 128 MB tmpfs. The plexstreams plugin was configured for `http://192.168.31.2:32400` — the dead macvlan shim subnet created in `go` for a Pi-hole that AdGuard replaced. Plex runs with host networking and answers on `192.168.10.20:32400`. Every failed poll logged three PHP warnings from `simplexml_load_string()`; 99,926 polls between 10 Oct and 28 Oct produced 299,778 lines, 99.95% of a 48 MB `phplog`. +- **[INCIDENT]** Logging across the box had been silently dead for six days — the last entry is 28 Oct 07:12, written when the filesystem filled, not when the errors stopped. Treat `syslog` for 2026-10-28 to 11-03 as incomplete. +- **[SERVICE]** Fixed: `HOST`/`hostbox` repointed to `192.168.10.20:32400` with a matching alias, config backed up. Logs truncated rather than deleted (`syslog` left intact). `/var/log` 100% -> 35%, verified clean over four poll cycles. Detail in `docs/incidents/2026-11-03-var-log-full-plexstreams.md`. +- Also seen in that log and **not** addressed: `appdata.backup` exhausting PHP's 256 MB limit 18 times, and a cosmetic `zfs.master` null-offset warning x130. + ### Network: Tailscale SSH ACL set to accept - **[NETWORK]** Tailnet SSH policy changed from Tailscale's default `action: "check"` to `accept` for `autogroup:member` -> `autogroup:self`. Before the change the control plane minted ~12 h per-device grants and prompted for browser re-auth when they lapsed, which would stall any unattended connection. Effective policy is now a single rule: accept for all 10 tailnet nodes, no expiry, with local and remote port forwarding and agent forwarding allowed. - **[NETWORK]** Verified unattended with `BatchMode=yes`: a plain command over Tailscale SSH, a `-L` port forward (HTTP 302 from the Unraid web UI) and the Herdr fallback leg all ran without a prompt. diff --git a/docs/incidents/2026-11-03-var-log-full-plexstreams.md b/docs/incidents/2026-11-03-var-log-full-plexstreams.md new file mode 100644 index 0000000..747ad7f --- /dev/null +++ b/docs/incidents/2026-11-03-var-log-full-plexstreams.md @@ -0,0 +1,80 @@ +# /var/log full — plexstreams polling a dead address + +**Date:** 2026-11-03 (silent since 2026-10-28) +**Severity:** P2 +**Status:** Resolved + +## Symptoms + +`/var/log` on Unraid at **100% of its 128 MB tmpfs**. Any process trying to log +failed with `errno=28 No space left on device`. Found by accident while +installing an unrelated script that tried to write a log file there. + +``` +tmpfs 128M 128M 0 100% /var/log + 48M phplog + 37M syslog + 31M tailscale.log + 6.6M tailscale-utils.log +``` + +## Timeline + +| Time | Event | +|------|-------| +| 2026-10-10 16:21 | First `phplog` entry. Plex had moved to host networking; plexstreams kept its old address | +| 2026-10-10 → 10-28 | 99,926 failed polls, three PHP warnings each, ~every 15 s | +| 2026-10-28 07:12 | Last entry written — **not** because errors stopped, but because the filesystem filled | +| 2026-10-28 → 11-03 | Logging silently dead across the box for six days | +| 2026-11-03 22:20 | Found, diagnosed, fixed | + +## Root Cause + +The **plexstreams** plugin was configured with +`HOST="http://192.168.31.2:32400"`. That address is on the macvlan shim subnet — +the same dead `192.168.31.0/24` range created in `/boot/config/go` for a Pi-hole +that AdGuard replaced. Nothing answers there. + +Plex itself runs with **host** networking and answers on `192.168.10.20:32400` +(HTTP 200). plexstreams was never updated when Plex moved. + +Each failed poll returned an empty body, so `simplexml_load_string()` failed and +logged three warnings: + +``` +PHP Warning: simplexml_load_string(): Entity: line 1: parser error : Document is empty + in /usr/local/emhttp/plugins/plexstreams/includes/common.php on line 180 +``` + +299,778 of 299,934 lines — **99.95% of the file** — were those three warnings. + +## Fix + +1. `HOST` and `hostbox` in `/boot/config/plugins/plexstreams/plexstreams.cfg` + repointed to `http://192.168.10.20:32400`, with `ALIAS-192_168_10_20="unraid"` + added so the UI still shows the right name. Backup at `plexstreams.cfg.bak-*`. +2. `phplog`, `tailscale.log` and `tailscale-utils.log` **truncated**, not + deleted — deleting a log a process holds open frees no space until that + process restarts, and the writer keeps appending to an unlinked inode. + `syslog` was left intact as the only remaining record of the period. + +Result: `/var/log` 100% → 35%. Verified over 60 s (four poll cycles) that +`phplog` stayed at 0 bytes with no new plexstreams warnings. + +## Open items found in the same log + +- **`appdata.backup` exhausted PHP's 256 MB limit 18 times** + (`Allowed memory size of 268435456 bytes exhausted`). Appdata backups may be + failing. Not investigated. +- **`zfs.master`**: `Trying to access array offset on null` ×130 — cosmetic + plugin bug. + +## Lessons + +- The six-day gap is the real damage: a full `/var/log` is silent. Nothing + alerts on it, and `syslog` for 2026-10-28 to 11-03 is incomplete. +- Worth a periodic check on `/var/log` usage; `go` already installs an rsyslog + noise filter, so log volume has been a problem here before. +- The dead `192.168.31.0/24` shim keeps surfacing. It was also the pointless + route Unraid's Tailscale node used to advertise, removed the same day. The + shim itself is still created in `go` for a Pi-hole that no longer exists.