Kaloyan DanchevandClaude Opus 5
0a115a4220
plexstreams was configured for http://192.168.31.2:32400, on the macvlan shim subnet created in go for a Pi-hole that AdGuard replaced. Plex runs with host networking on 192.168.10.20:32400. Each failed poll logged three PHP warnings; 99,926 polls filled a 48 MB phplog, 99.95% of it those warnings. Logging had been dead box-wide for six days: the last entry was written when the filesystem filled, not when the errors stopped, so syslog for 28 Oct to 3 Nov is incomplete. Config repointed and backed up, logs truncated rather than deleted so writers holding them open reclaim space immediately. syslog left intact. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
81 lines
3.2 KiB
Markdown
81 lines
3.2 KiB
Markdown
# /var/log full — plexstreams polling a dead address
|
||
|
||
**Date:** 2026-11-03 (silent since 2026-10-28)
|
||
**Severity:** P2
|
||
**Status:** Resolved
|
||
|
||
## Symptoms
|
||
|
||
`/var/log` on Unraid at **100% of its 128 MB tmpfs**. Any process trying to log
|
||
failed with `errno=28 No space left on device`. Found by accident while
|
||
installing an unrelated script that tried to write a log file there.
|
||
|
||
```
|
||
tmpfs 128M 128M 0 100% /var/log
|
||
48M phplog
|
||
37M syslog
|
||
31M tailscale.log
|
||
6.6M tailscale-utils.log
|
||
```
|
||
|
||
## Timeline
|
||
|
||
| Time | Event |
|
||
|------|-------|
|
||
| 2026-10-10 16:21 | First `phplog` entry. Plex had moved to host networking; plexstreams kept its old address |
|
||
| 2026-10-10 → 10-28 | 99,926 failed polls, three PHP warnings each, ~every 15 s |
|
||
| 2026-10-28 07:12 | Last entry written — **not** because errors stopped, but because the filesystem filled |
|
||
| 2026-10-28 → 11-03 | Logging silently dead across the box for six days |
|
||
| 2026-11-03 22:20 | Found, diagnosed, fixed |
|
||
|
||
## Root Cause
|
||
|
||
The **plexstreams** plugin was configured with
|
||
`HOST="http://192.168.31.2:32400"`. That address is on the macvlan shim subnet —
|
||
the same dead `192.168.31.0/24` range created in `/boot/config/go` for a Pi-hole
|
||
that AdGuard replaced. Nothing answers there.
|
||
|
||
Plex itself runs with **host** networking and answers on `192.168.10.20:32400`
|
||
(HTTP 200). plexstreams was never updated when Plex moved.
|
||
|
||
Each failed poll returned an empty body, so `simplexml_load_string()` failed and
|
||
logged three warnings:
|
||
|
||
```
|
||
PHP Warning: simplexml_load_string(): Entity: line 1: parser error : Document is empty
|
||
in /usr/local/emhttp/plugins/plexstreams/includes/common.php on line 180
|
||
```
|
||
|
||
299,778 of 299,934 lines — **99.95% of the file** — were those three warnings.
|
||
|
||
## Fix
|
||
|
||
1. `HOST` and `hostbox` in `/boot/config/plugins/plexstreams/plexstreams.cfg`
|
||
repointed to `http://192.168.10.20:32400`, with `ALIAS-192_168_10_20="unraid"`
|
||
added so the UI still shows the right name. Backup at `plexstreams.cfg.bak-*`.
|
||
2. `phplog`, `tailscale.log` and `tailscale-utils.log` **truncated**, not
|
||
deleted — deleting a log a process holds open frees no space until that
|
||
process restarts, and the writer keeps appending to an unlinked inode.
|
||
`syslog` was left intact as the only remaining record of the period.
|
||
|
||
Result: `/var/log` 100% → 35%. Verified over 60 s (four poll cycles) that
|
||
`phplog` stayed at 0 bytes with no new plexstreams warnings.
|
||
|
||
## Open items found in the same log
|
||
|
||
- **`appdata.backup` exhausted PHP's 256 MB limit 18 times**
|
||
(`Allowed memory size of 268435456 bytes exhausted`). Appdata backups may be
|
||
failing. Not investigated.
|
||
- **`zfs.master`**: `Trying to access array offset on null` ×130 — cosmetic
|
||
plugin bug.
|
||
|
||
## Lessons
|
||
|
||
- The six-day gap is the real damage: a full `/var/log` is silent. Nothing
|
||
alerts on it, and `syslog` for 2026-10-28 to 11-03 is incomplete.
|
||
- Worth a periodic check on `/var/log` usage; `go` already installs an rsyslog
|
||
noise filter, so log volume has been a problem here before.
|
||
- The dead `192.168.31.0/24` shim keeps surfacing. It was also the pointless
|
||
route Unraid's Tailscale node used to advertise, removed the same day. The
|
||
shim itself is still created in `go` for a Pi-hole that no longer exists.
|