0 Fork 0

Files

T

Kaloyan DanchevandClaude Opus 5 0a115a4220 incident: /var/log full from plexstreams polling a dead address

plexstreams was configured for http://192.168.31.2:32400, on the macvlan
shim subnet created in go for a Pi-hole that AdGuard replaced. Plex runs with
host networking on 192.168.10.20:32400. Each failed poll logged three PHP
warnings; 99,926 polls filled a 48 MB phplog, 99.95% of it those warnings.

Logging had been dead box-wide for six days: the last entry was written when
the filesystem filled, not when the errors stopped, so syslog for 28 Oct to
3 Nov is incomplete.

Config repointed and backed up, logs truncated rather than deleted so writers
holding them open reclaim space immediately. syslog left intact.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

2026-11-04 00:24:43 +02:00

3.2 KiB

Raw Permalink Blame History

/var/log full — plexstreams polling a dead address

Date: 2026-11-03 (silent since 2026-10-28) Severity: P2 Status: Resolved

Symptoms

/var/log on Unraid at 100% of its 128 MB tmpfs. Any process trying to log failed with errno=28 No space left on device. Found by accident while installing an unrelated script that tried to write a log file there.

tmpfs  128M  128M  0  100%  /var/log
  48M  phplog
  37M  syslog
  31M  tailscale.log
 6.6M  tailscale-utils.log

Timeline

Time	Event
2026-10-10 16:21	First `phplog` entry. Plex had moved to host networking; plexstreams kept its old address
2026-10-10 → 10-28	99,926 failed polls, three PHP warnings each, ~every 15 s
2026-10-28 07:12	Last entry written — not because errors stopped, but because the filesystem filled
2026-10-28 → 11-03	Logging silently dead across the box for six days
2026-11-03 22:20	Found, diagnosed, fixed

Root Cause

The plexstreams plugin was configured with HOST="http://192.168.31.2:32400". That address is on the macvlan shim subnet — the same dead 192.168.31.0/24 range created in /boot/config/go for a Pi-hole that AdGuard replaced. Nothing answers there.

Plex itself runs with host networking and answers on 192.168.10.20:32400 (HTTP 200). plexstreams was never updated when Plex moved.

Each failed poll returned an empty body, so simplexml_load_string() failed and logged three warnings:

PHP Warning: simplexml_load_string(): Entity: line 1: parser error : Document is empty
  in /usr/local/emhttp/plugins/plexstreams/includes/common.php on line 180

299,778 of 299,934 lines — 99.95% of the file — were those three warnings.

Fix

HOST and hostbox in /boot/config/plugins/plexstreams/plexstreams.cfg repointed to http://192.168.10.20:32400, with ALIAS-192_168_10_20="unraid" added so the UI still shows the right name. Backup at plexstreams.cfg.bak-*.
phplog, tailscale.log and tailscale-utils.log truncated, not deleted — deleting a log a process holds open frees no space until that process restarts, and the writer keeps appending to an unlinked inode. syslog was left intact as the only remaining record of the period.

Result: /var/log 100% → 35%. Verified over 60 s (four poll cycles) that phplog stayed at 0 bytes with no new plexstreams warnings.

Open items found in the same log

appdata.backup exhausted PHP's 256 MB limit 18 times (Allowed memory size of 268435456 bytes exhausted). Appdata backups may be failing. Not investigated.
zfs.master: Trying to access array offset on null ×130 — cosmetic plugin bug.

Lessons

The six-day gap is the real damage: a full /var/log is silent. Nothing alerts on it, and syslog for 2026-10-28 to 11-03 is incomplete.
Worth a periodic check on /var/log usage; go already installs an rsyslog noise filter, so log volume has been a problem here before.
The dead 192.168.31.0/24 shim keeps surfacing. It was also the pointless route Unraid's Tailscale node used to advertise, removed the same day. The shim itself is still created in go for a Pi-hole that no longer exists.

3.2 KiB Raw Permalink Blame History Unescape Escape