docs: Herdr on Tailscale SSH, with linger as the real socket fix

Rewrites the Herdr doc to match what was actually built. The machine now
targets root@100.100.208.69 over Tailscale SSH instead of the LAN address,
so it no longer depends on the mikrotik-tailscale subnet router.

The "core did not start within 5 s" failure was the hc core socket living
under XDG_RUNTIME_DIR = /run/user/0, which pam_elogind creates and destroys
per login session. loginctl enable-linger root keeps that directory alive, so
every client agrees on the default socket path. The earlier Mac-side env pin
fixed the CLI but broke the GUI's plain-ssh "Add a machine"; linger
supersedes it and the wrapper is kept but unused.

Also records the stale ~/.ssh/config entry that pointed unraid at the deleted
100.100.208.70 on port 422, which is what the GUI had been listing, and the
go-script block that restores both the binary and linger at boot.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

docs/14-HERDR-REMOTE-MACHINES.md

@@ -5,140 +5,140 @@ programs and coding agents in terminals that stay alive, including on remote
 machines reached over SSH. The Unraid server is registered as the machine
 `unraid`.
 
-Added 2026-11-03.
+Set up 2026-11-03.
 
-## Registration
+## Current configuration
-
-The transport is a wrapper on the Mac, not a plain SSH target:
 
 ```bash
-herdr-core machine add --via ~/.local/bin/herdr-via-unraid --name unraid --install
+herdr-core machine add root@100.100.208.69 --name unraid --install
 ```
 
 ```
-unraid | state: connected | host: xtrm-unraid | hc: 0.1.17
+unraid | state: connected | target: root@100.100.208.69 | hc: 0.1.17
 ```
 
 Verify with `herdr-core exec --on unraid 'hostname'` → `XTRM-Unraid`.
 
-## Why a wrapper and not `machine add root@192.168.10.20:422`
+The target is Unraid's own Tailscale node over Tailscale SSH (port 22), not the
+LAN address. Consequences:
 
-`hc` starts a core process (`herdr-core serve`) on the remote and talks to it
+- It does not depend on the `mikrotik-tailscale` subnet router, which has failed
-over a unix socket under `XDG_RUNTIME_DIR`. On Unraid that resolves to
+  before (see the 2026-08-10 recovery note in `docs/09-TAILSCALE-VPN.md`). Only
-`/run/user/0`, which **elogind creates and destroys per login session**. So:
+  Unraid itself and the tailnet need to be healthy.
+- It works identically at home and away, with no configuration to change.
+  Tailscale still picks the efficient path — on the LAN it reports
+  `direct 192.168.10.20:41725` rather than relaying.
+- Authentication is tailnet identity, not an SSH key. Access therefore rests on
+  the tailnet SSH policy, which was set to `accept` on 2026-11-03. Under the
+  previous `check` policy this would have demanded a browser re-auth every
+  ~12 h and stalled.
+
+## Two problems had to be solved first
+
+### 1. The socket path — `core did not start within 5 s`
+
+`hc` runs a core process (`herdr-core serve`) on the remote and talks to it over
+a unix socket under `XDG_RUNTIME_DIR`. On Unraid that is `/run/user/0`, which
+**`pam_elogind.so` creates and destroys per login session**. So:
 
 1. `machine add` opens an SSH session and starts the core. The socket lands in
    `/run/user/0/herdr-core/default.sock`.
-2. The session ends. elogind removes `/run/user/0`, taking the socket with it.
+2. The session ends. elogind removes `/run/user/0` with the socket inside it.
 3. The next client connects in a new session, finds a fresh empty
-   `/run/user/0`, and fails with `core did not start within 5 s`.
+   `/run/user/0`, and times out after 5 s.
 
-The core is not at fault — it keeps running, holding a socket that has been
+The core is not at fault — it keeps running on a socket that has been unlinked.
-unlinked from the filesystem. `core.log` stays 0 bytes because the failure is
+`core.log` stays 0 bytes because the failure is client side. Unraid also prints
-client side. Unraid also warns `hc: no systemd user manager here: the core ends
+`hc: no systemd user manager here: the core ends with whatever started it`,
-with whatever started it`, which is why `hc` cannot supervise its own core here.
+which is why `hc` cannot supervise its own core here.
 
-The fix is to pin `XDG_RUNTIME_DIR` to a path that outlives a login session.
+**Fix: `loginctl enable-linger root`.** Unraid ships `loginctl` even without
-Doing it in the `--via` transport keeps the change on the Mac and leaves the
+systemd, and linger keeps `/run/user/0` alive across logouts, so the default
-server's shell configuration untouched.
+socket path is stable and every client agrees on it.
 
-`~/.local/bin/herdr-via-unraid`:
+An earlier attempt pinned `XDG_RUNTIME_DIR=/var/run/herdr-rt` from a Mac-side
+`--via` wrapper. That worked for the CLI but broke the GUI's "Add a machine",
+which uses plain ssh and so looked in `/run/user/0`:
 
-```sh
+```
-#!/bin/sh
+the core listens at /var/run/herdr-rt/herdr-core/default.sock,
-# Herdr appends the shell line to run as the last argument.
+but this shell looks at /run/user/0/herdr-core/default.sock
-exec ssh -T -o BatchMode=yes -o ConnectTimeout=10 \
+hint: set HC_SOCKET to the first path
-    -p 422 -i "$HOME/.ssh/id_ed25519_unraid" root@192.168.10.20 \
-    "export XDG_RUNTIME_DIR=/var/run/herdr-rt; mkdir -p \"\$XDG_RUNTIME_DIR\" && chmod 700 \"\$XDG_RUNTIME_DIR\"; $*"
 ```
 
-## Working from outside the LAN
+Linger supersedes that. The wrapper is retained at
+`~/.local/bin/herdr-via-unraid` (LAN primary, Tailscale fallback, chosen by a
+TCP probe before any ssh runs so a failed attempt cannot consume stdin) but is
+**not in use**. To go back to it: `herdr-core machine rm unraid`, then
+`machine add --via ~/.local/bin/herdr-via-unraid --name unraid`.
 
-No extra configuration is needed. The transport targets `192.168.10.20:422`, and
+### 2. A stale `~/.ssh/config` entry
-a Mac away from home reaches that over Tailscale via the `mikrotik-tailscale`
-subnet router, which is primary for `192.168.10.0/24`. This is not a special
-remote mode — it is the same path used on VLAN 20 at home, since that is also a
-different subnet from Unraid.
 
-Prerequisites on the client:
+The GUI lists candidates from `~/.ssh/config`, which still held the Tailscale IP
+of the node deleted on 2026-11-03 — and pointed it at a port that was never open
+there:
 
-| Requirement | Check |
+```
-|-------------|-------|
+Host 100.100.208.70 unraid     # node no longer exists
-| Tailscale running and logged in | `tailscale status` |
+  HostName 100.100.208.70
-| Accepts the hAP's subnet routes | `tailscale debug prefs` shows `"RouteAll": true` |
+  Port 422                      # sshd binds only 192.168.10.20 and 192.168.31.250
-| Route resolves over Tailscale | `route -n get 192.168.10.20` reports `utun*`, not `en0` |
+```
 
-### Fallback leg
+Replaced with two correct entries (backup at `~/.ssh/config.bak-20261103`):
 
-The LAN path depends on the `mikrotik-tailscale` container, which has failed
+```
-before (see the 2026-08-10 recovery note in `docs/09-TAILSCALE-VPN.md`). If it
+Host unraid                     # LAN address; routes over Tailscale when away
-dies while Unraid stays up, `192.168.10.20` becomes unreachable over Tailscale
+  HostName 192.168.10.20
-but Unraid's own node does not.
+  User root
+  Port 422
+  IdentityFile ~/.ssh/id_ed25519_unraid
 
-So the wrapper probes `192.168.10.20:422` first and falls back to Tailscale SSH
+Host unraid-ts 100.100.208.69   # Tailscale SSH: port 22, no key
-on `100.100.208.69:22`. The probe runs **before** any ssh starts, because a
+  HostName 100.100.208.69
-failed first attempt would consume stdin and corrupt the retry. It costs about
+  User root
-245 ms on the success path. `HERDR_VIA_TARGET=lan|ts` forces a leg for testing;
+```
-both were verified 2026-11-03.
-
-The fallback leg is fully unattended since the tailnet SSH ACL was set to
-`accept` on 2026-11-03. Verified the same day with `BatchMode=yes`: a plain
-command, a `-L` port forward (HTTP 302 from the Unraid web UI) and the Herdr leg
-itself all ran without a prompt.
-
-## Why the LAN address is the primary and not the Tailscale IP
-
-`--via` requires a transport that passes bytes untouched and **asks nothing**.
-When this was set up, Tailscale SSH sat behind the default tailnet ACL
-`action: "check"`, which demanded browser re-authentication every ~12 h — fine
-interactively, fatal for a long-lived Herdr connection. That ACL was changed to
-`accept` on 2026-11-03, so either address would now serve.
-
-The LAN address stays primary anyway: key auth on `192.168.10.20:422` depends on
-nothing but the SSH key, while Tailscale SSH depends on the tailnet policy
-staying permissive. From outside the LAN it still travels over Tailscale via the
-MikroTik subnet router (see `docs/09-TAILSCALE-VPN.md`), so there is no loss.
 
 ## Reboot persistence
 
-Unraid runs `/` from `rootfs` (RAM) and `/run` as tmpfs, so both of these are
+Unraid runs `/` from `rootfs` (RAM) and `/run` as tmpfs, so all of this is lost
-lost on every reboot:
+on reboot and is restored by `/boot/config/go`:
 
-| Path | Lost on reboot | Recovery |
+| Path | Lost on reboot | Restored by |
-|------|----------------|----------|
+|------|----------------|-------------|
-| `/var/run/herdr-rt` | yes | self-healing — the `--via` wrapper recreates it on every connection |
+| `/root/.local/bin/herdr-core` | yes | `install -m 755` from `/boot/config/herdr/herdr-core` |
-| `/root/.local/bin/herdr-core` | yes | **manual** — `herdr-core machine update`, or `machine add --install` again |
+| `/var/lib/elogind/linger/root` | yes | `loginctl enable-linger root` |
+| `/run/user/0` | yes | recreated by pam_elogind at first login, then held open by linger |
 
-To make the binary survive, stash it on the boot flash and restore it from the
+The `go` block:
-`go` script, in the same style as the existing `install`/`cp` lines there:
-
-```bash
-mkdir -p /boot/config/herdr
-cp /root/.local/bin/herdr-core /boot/config/herdr/herdr-core
-```
-
-then append to `/boot/config/go`:
 
 ```bash
 # Herdr (hc) remote machines. Unraid runs / from RAM, so the hc binary installed
 # by `herdr-core machine add --install` is lost on reboot. Restore it from /boot.
-# The runtime dir is not restored here: the Mac-side --via wrapper exports
-# XDG_RUNTIME_DIR=/var/run/herdr-rt and creates it on every connection, because
-# the default /run/user/0 is destroyed per login session by elogind.
 mkdir -p /root/.local/bin
 install -m 755 /boot/config/herdr/herdr-core /root/.local/bin/herdr-core 2>/dev/null
+
+# Keep /run/user/0 alive across logouts. pam_elogind creates and destroys it per
+# login session, and the hc core's socket lives there, so without linger the
+# socket disappears with the session that started the core and every later
+# client fails with "core did not start within 5 s". The linger marker is
+# /var/lib/elogind/linger/root, and /var/lib is RAM here, so it must be re-set
+# at every boot.
+loginctl enable-linger root 2>/dev/null
 ```
 
-`install -m 755` matters: `/boot` is FAT32 and does not carry the execute bit.
+`install -m 755` matters: `/boot` is FAT32 and carries no execute bit.
 
-**Status: applied 2026-11-03.** The `go` script was backed up to
+Applied 2026-11-03, `sh -n /boot/config/go` parses cleanly. Backups of the go
-`/boot/config/go.bak-20261103-203916` first.
+script are at `/boot/config/go.bak-*`. The binary restore was verified without
+rebooting, by deleting `/root/.local/bin/herdr-core` and running the line by
+hand: it came back `-rwxr-xr-x` reporting `hc 0.1.17`.
 
-Verified without rebooting, by deleting `/root/.local/bin/herdr-core` and running
+## Notes
-the `go` line by hand: the binary came back `-rwxr-xr-x` and reported `hc 0.1.17`.
-Herdr stayed `connected` throughout.
 
-One note if you ever call it over a bare SSH command: `~/.local/bin` is not on
+- `~/.local/bin` is not on `PATH` in a non-interactive shell on Unraid, so
-`PATH` in a non-interactive shell on Unraid, so `herdr-core` resolves only by
+  `herdr-core` resolves only by absolute path there. Herdr itself always uses
-absolute path. Herdr itself always uses the absolute path, so this does not
+  the absolute path, so this does not affect normal use.
-affect normal use.
+- Adding a machine that `hc` already knows is refused as a duplicate — it keys
+  on the core id (`3d4a2397fcb6…`), not the address. Re-pointing an existing
+  machine means `machine rm` then `machine add`, since there is no command to
+  change a target in place.

docs/CHANGELOG.md

@@ -19,6 +19,10 @@
 - Trade-off recorded in the docs: `check` acted as a second factor, so a compromised tailnet device now reaches root on Unraid without a further step. Accepted because every node in the tailnet is owner-controlled hardware.
 
 ### Tooling: Unraid registered as a Herdr (hc) machine
+- **[SERVICE]** Final state: the machine `unraid` targets `root@100.100.208.69` over Tailscale SSH, so it no longer depends on the `mikrotik-tailscale` subnet router. Works identically at home and away; Tailscale still reports `direct 192.168.10.20:41725` on the LAN rather than relaying.
+- **[SERVICE]** Root cause of `core did not start within 5 s` was the hc core socket living under `XDG_RUNTIME_DIR` = `/run/user/0`, which `pam_elogind.so` creates and destroys per login session. Fixed properly with `loginctl enable-linger root`, which keeps that directory alive so every client agrees on the default socket path. An earlier Mac-side `XDG_RUNTIME_DIR` pin fixed the CLI but broke the GUI's plain-ssh "Add a machine"; linger supersedes it and the wrapper is retained but unused.
+- **[SERVICE]** `/boot/config/go` now restores the hc binary **and** re-applies linger at boot, since `/`, `/run` and `/var/lib` are all RAM on Unraid. `sh -n` clean, backups at `/boot/config/go.bak-*`.
+- **[SERVICE]** Corrected a stale `~/.ssh/config` entry that still pointed `unraid` at the deleted node `100.100.208.70` on port 422 — an address and port combination that never worked, since sshd binds only 192.168.10.20 and 192.168.31.250. Replaced with `unraid` (LAN, key, 422) and `unraid-ts` (100.100.208.69, Tailscale SSH, 22). This stale entry was also what the Herdr GUI was listing.
 - **[SERVICE]** Unraid added to Herdr as the machine `unraid` (`hc` 0.1.17 on the remote). `herdr-core machine add` had been failing with `core did not start within 5 s`: `hc` puts its core socket under `XDG_RUNTIME_DIR`, which on Unraid is `/run/user/0` — a directory elogind creates and destroys per login session, so the socket disappeared the moment the session that started the core ended. The core kept running with an unlinked socket, and `core.log` stayed empty because the failure is client side.
 - **[SERVICE]** Fixed from the Mac only, via `~/.local/bin/herdr-via-unraid`, a `--via` transport that exports `XDG_RUNTIME_DIR=/var/run/herdr-rt` on every connection. Nothing on the server was changed. Uses key auth on 192.168.10.20:422 rather than Tailscale SSH, because `--via` needs a transport that asks nothing and the default tailnet ACL `check` would demand periodic browser re-auth.
 - **[SERVICE]** Reboot caveat: `/` is rootfs (RAM) and `/run` is tmpfs on Unraid, so `/root/.local/bin/herdr-core` is lost at every boot and needs `herdr-core machine update` to restore. A `/boot/config/go` entry now restores it at boot from `/boot/config/herdr/herdr-core` (`install -m 755`, because /boot is FAT32 and carries no execute bit). Applied 2026-11-03, with the go script backed up to `go.bak-20261103-203916` and verified by deleting the binary and running the line by hand. Detail in `docs/14-HERDR-REMOTE-MACHINES.md`.