From 1613f7f4584f059f2fa029038ba79d6bea06ddd4 Mon Sep 17 00:00:00 2001 From: Kaloyan Danchev Date: Tue, 3 Nov 2026 23:37:46 +0200 Subject: [PATCH] docs: Herdr on Tailscale SSH, with linger as the real socket fix Rewrites the Herdr doc to match what was actually built. The machine now targets root@100.100.208.69 over Tailscale SSH instead of the LAN address, so it no longer depends on the mikrotik-tailscale subnet router. The "core did not start within 5 s" failure was the hc core socket living under XDG_RUNTIME_DIR = /run/user/0, which pam_elogind creates and destroys per login session. loginctl enable-linger root keeps that directory alive, so every client agrees on the default socket path. The earlier Mac-side env pin fixed the CLI but broke the GUI's plain-ssh "Add a machine"; linger supersedes it and the wrapper is kept but unused. Also records the stale ~/.ssh/config entry that pointed unraid at the deleted 100.100.208.70 on port 422, which is what the GUI had been listing, and the go-script block that restores both the binary and linger at boot. Co-Authored-By: Claude Opus 5 --- docs/14-HERDR-REMOTE-MACHINES.md | 188 +++++++++++++++---------------- docs/CHANGELOG.md | 4 + 2 files changed, 98 insertions(+), 94 deletions(-) diff --git a/docs/14-HERDR-REMOTE-MACHINES.md b/docs/14-HERDR-REMOTE-MACHINES.md index 0ffc1d1..6ad83f9 100644 --- a/docs/14-HERDR-REMOTE-MACHINES.md +++ b/docs/14-HERDR-REMOTE-MACHINES.md @@ -5,140 +5,140 @@ programs and coding agents in terminals that stay alive, including on remote machines reached over SSH. The Unraid server is registered as the machine `unraid`. -Added 2026-11-03. +Set up 2026-11-03. -## Registration - -The transport is a wrapper on the Mac, not a plain SSH target: +## Current configuration ```bash -herdr-core machine add --via ~/.local/bin/herdr-via-unraid --name unraid --install +herdr-core machine add root@100.100.208.69 --name unraid --install ``` ``` -unraid | state: connected | host: xtrm-unraid | hc: 0.1.17 +unraid | state: connected | target: root@100.100.208.69 | hc: 0.1.17 ``` Verify with `herdr-core exec --on unraid 'hostname'` → `XTRM-Unraid`. -## Why a wrapper and not `machine add root@192.168.10.20:422` +The target is Unraid's own Tailscale node over Tailscale SSH (port 22), not the +LAN address. Consequences: -`hc` starts a core process (`herdr-core serve`) on the remote and talks to it -over a unix socket under `XDG_RUNTIME_DIR`. On Unraid that resolves to -`/run/user/0`, which **elogind creates and destroys per login session**. So: +- It does not depend on the `mikrotik-tailscale` subnet router, which has failed + before (see the 2026-08-10 recovery note in `docs/09-TAILSCALE-VPN.md`). Only + Unraid itself and the tailnet need to be healthy. +- It works identically at home and away, with no configuration to change. + Tailscale still picks the efficient path — on the LAN it reports + `direct 192.168.10.20:41725` rather than relaying. +- Authentication is tailnet identity, not an SSH key. Access therefore rests on + the tailnet SSH policy, which was set to `accept` on 2026-11-03. Under the + previous `check` policy this would have demanded a browser re-auth every + ~12 h and stalled. + +## Two problems had to be solved first + +### 1. The socket path — `core did not start within 5 s` + +`hc` runs a core process (`herdr-core serve`) on the remote and talks to it over +a unix socket under `XDG_RUNTIME_DIR`. On Unraid that is `/run/user/0`, which +**`pam_elogind.so` creates and destroys per login session**. So: 1. `machine add` opens an SSH session and starts the core. The socket lands in `/run/user/0/herdr-core/default.sock`. -2. The session ends. elogind removes `/run/user/0`, taking the socket with it. +2. The session ends. elogind removes `/run/user/0` with the socket inside it. 3. The next client connects in a new session, finds a fresh empty - `/run/user/0`, and fails with `core did not start within 5 s`. + `/run/user/0`, and times out after 5 s. -The core is not at fault — it keeps running, holding a socket that has been -unlinked from the filesystem. `core.log` stays 0 bytes because the failure is -client side. Unraid also warns `hc: no systemd user manager here: the core ends -with whatever started it`, which is why `hc` cannot supervise its own core here. +The core is not at fault — it keeps running on a socket that has been unlinked. +`core.log` stays 0 bytes because the failure is client side. Unraid also prints +`hc: no systemd user manager here: the core ends with whatever started it`, +which is why `hc` cannot supervise its own core here. -The fix is to pin `XDG_RUNTIME_DIR` to a path that outlives a login session. -Doing it in the `--via` transport keeps the change on the Mac and leaves the -server's shell configuration untouched. +**Fix: `loginctl enable-linger root`.** Unraid ships `loginctl` even without +systemd, and linger keeps `/run/user/0` alive across logouts, so the default +socket path is stable and every client agrees on it. -`~/.local/bin/herdr-via-unraid`: +An earlier attempt pinned `XDG_RUNTIME_DIR=/var/run/herdr-rt` from a Mac-side +`--via` wrapper. That worked for the CLI but broke the GUI's "Add a machine", +which uses plain ssh and so looked in `/run/user/0`: -```sh -#!/bin/sh -# Herdr appends the shell line to run as the last argument. -exec ssh -T -o BatchMode=yes -o ConnectTimeout=10 \ - -p 422 -i "$HOME/.ssh/id_ed25519_unraid" root@192.168.10.20 \ - "export XDG_RUNTIME_DIR=/var/run/herdr-rt; mkdir -p \"\$XDG_RUNTIME_DIR\" && chmod 700 \"\$XDG_RUNTIME_DIR\"; $*" +``` +the core listens at /var/run/herdr-rt/herdr-core/default.sock, +but this shell looks at /run/user/0/herdr-core/default.sock +hint: set HC_SOCKET to the first path ``` -## Working from outside the LAN +Linger supersedes that. The wrapper is retained at +`~/.local/bin/herdr-via-unraid` (LAN primary, Tailscale fallback, chosen by a +TCP probe before any ssh runs so a failed attempt cannot consume stdin) but is +**not in use**. To go back to it: `herdr-core machine rm unraid`, then +`machine add --via ~/.local/bin/herdr-via-unraid --name unraid`. -No extra configuration is needed. The transport targets `192.168.10.20:422`, and -a Mac away from home reaches that over Tailscale via the `mikrotik-tailscale` -subnet router, which is primary for `192.168.10.0/24`. This is not a special -remote mode — it is the same path used on VLAN 20 at home, since that is also a -different subnet from Unraid. +### 2. A stale `~/.ssh/config` entry -Prerequisites on the client: +The GUI lists candidates from `~/.ssh/config`, which still held the Tailscale IP +of the node deleted on 2026-11-03 — and pointed it at a port that was never open +there: -| Requirement | Check | -|-------------|-------| -| Tailscale running and logged in | `tailscale status` | -| Accepts the hAP's subnet routes | `tailscale debug prefs` shows `"RouteAll": true` | -| Route resolves over Tailscale | `route -n get 192.168.10.20` reports `utun*`, not `en0` | +``` +Host 100.100.208.70 unraid # node no longer exists + HostName 100.100.208.70 + Port 422 # sshd binds only 192.168.10.20 and 192.168.31.250 +``` -### Fallback leg +Replaced with two correct entries (backup at `~/.ssh/config.bak-20261103`): -The LAN path depends on the `mikrotik-tailscale` container, which has failed -before (see the 2026-08-10 recovery note in `docs/09-TAILSCALE-VPN.md`). If it -dies while Unraid stays up, `192.168.10.20` becomes unreachable over Tailscale -but Unraid's own node does not. +``` +Host unraid # LAN address; routes over Tailscale when away + HostName 192.168.10.20 + User root + Port 422 + IdentityFile ~/.ssh/id_ed25519_unraid -So the wrapper probes `192.168.10.20:422` first and falls back to Tailscale SSH -on `100.100.208.69:22`. The probe runs **before** any ssh starts, because a -failed first attempt would consume stdin and corrupt the retry. It costs about -245 ms on the success path. `HERDR_VIA_TARGET=lan|ts` forces a leg for testing; -both were verified 2026-11-03. - -The fallback leg is fully unattended since the tailnet SSH ACL was set to -`accept` on 2026-11-03. Verified the same day with `BatchMode=yes`: a plain -command, a `-L` port forward (HTTP 302 from the Unraid web UI) and the Herdr leg -itself all ran without a prompt. - -## Why the LAN address is the primary and not the Tailscale IP - -`--via` requires a transport that passes bytes untouched and **asks nothing**. -When this was set up, Tailscale SSH sat behind the default tailnet ACL -`action: "check"`, which demanded browser re-authentication every ~12 h — fine -interactively, fatal for a long-lived Herdr connection. That ACL was changed to -`accept` on 2026-11-03, so either address would now serve. - -The LAN address stays primary anyway: key auth on `192.168.10.20:422` depends on -nothing but the SSH key, while Tailscale SSH depends on the tailnet policy -staying permissive. From outside the LAN it still travels over Tailscale via the -MikroTik subnet router (see `docs/09-TAILSCALE-VPN.md`), so there is no loss. +Host unraid-ts 100.100.208.69 # Tailscale SSH: port 22, no key + HostName 100.100.208.69 + User root +``` ## Reboot persistence -Unraid runs `/` from `rootfs` (RAM) and `/run` as tmpfs, so both of these are -lost on every reboot: +Unraid runs `/` from `rootfs` (RAM) and `/run` as tmpfs, so all of this is lost +on reboot and is restored by `/boot/config/go`: -| Path | Lost on reboot | Recovery | -|------|----------------|----------| -| `/var/run/herdr-rt` | yes | self-healing — the `--via` wrapper recreates it on every connection | -| `/root/.local/bin/herdr-core` | yes | **manual** — `herdr-core machine update`, or `machine add --install` again | +| Path | Lost on reboot | Restored by | +|------|----------------|-------------| +| `/root/.local/bin/herdr-core` | yes | `install -m 755` from `/boot/config/herdr/herdr-core` | +| `/var/lib/elogind/linger/root` | yes | `loginctl enable-linger root` | +| `/run/user/0` | yes | recreated by pam_elogind at first login, then held open by linger | -To make the binary survive, stash it on the boot flash and restore it from the -`go` script, in the same style as the existing `install`/`cp` lines there: - -```bash -mkdir -p /boot/config/herdr -cp /root/.local/bin/herdr-core /boot/config/herdr/herdr-core -``` - -then append to `/boot/config/go`: +The `go` block: ```bash # Herdr (hc) remote machines. Unraid runs / from RAM, so the hc binary installed # by `herdr-core machine add --install` is lost on reboot. Restore it from /boot. -# The runtime dir is not restored here: the Mac-side --via wrapper exports -# XDG_RUNTIME_DIR=/var/run/herdr-rt and creates it on every connection, because -# the default /run/user/0 is destroyed per login session by elogind. mkdir -p /root/.local/bin install -m 755 /boot/config/herdr/herdr-core /root/.local/bin/herdr-core 2>/dev/null + +# Keep /run/user/0 alive across logouts. pam_elogind creates and destroys it per +# login session, and the hc core's socket lives there, so without linger the +# socket disappears with the session that started the core and every later +# client fails with "core did not start within 5 s". The linger marker is +# /var/lib/elogind/linger/root, and /var/lib is RAM here, so it must be re-set +# at every boot. +loginctl enable-linger root 2>/dev/null ``` -`install -m 755` matters: `/boot` is FAT32 and does not carry the execute bit. +`install -m 755` matters: `/boot` is FAT32 and carries no execute bit. -**Status: applied 2026-11-03.** The `go` script was backed up to -`/boot/config/go.bak-20261103-203916` first. +Applied 2026-11-03, `sh -n /boot/config/go` parses cleanly. Backups of the go +script are at `/boot/config/go.bak-*`. The binary restore was verified without +rebooting, by deleting `/root/.local/bin/herdr-core` and running the line by +hand: it came back `-rwxr-xr-x` reporting `hc 0.1.17`. -Verified without rebooting, by deleting `/root/.local/bin/herdr-core` and running -the `go` line by hand: the binary came back `-rwxr-xr-x` and reported `hc 0.1.17`. -Herdr stayed `connected` throughout. +## Notes -One note if you ever call it over a bare SSH command: `~/.local/bin` is not on -`PATH` in a non-interactive shell on Unraid, so `herdr-core` resolves only by -absolute path. Herdr itself always uses the absolute path, so this does not -affect normal use. +- `~/.local/bin` is not on `PATH` in a non-interactive shell on Unraid, so + `herdr-core` resolves only by absolute path there. Herdr itself always uses + the absolute path, so this does not affect normal use. +- Adding a machine that `hc` already knows is refused as a duplicate — it keys + on the core id (`3d4a2397fcb6…`), not the address. Re-pointing an existing + machine means `machine rm` then `machine add`, since there is no command to + change a target in place. diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index 7660626..c7dc1b1 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -19,6 +19,10 @@ - Trade-off recorded in the docs: `check` acted as a second factor, so a compromised tailnet device now reaches root on Unraid without a further step. Accepted because every node in the tailnet is owner-controlled hardware. ### Tooling: Unraid registered as a Herdr (hc) machine +- **[SERVICE]** Final state: the machine `unraid` targets `root@100.100.208.69` over Tailscale SSH, so it no longer depends on the `mikrotik-tailscale` subnet router. Works identically at home and away; Tailscale still reports `direct 192.168.10.20:41725` on the LAN rather than relaying. +- **[SERVICE]** Root cause of `core did not start within 5 s` was the hc core socket living under `XDG_RUNTIME_DIR` = `/run/user/0`, which `pam_elogind.so` creates and destroys per login session. Fixed properly with `loginctl enable-linger root`, which keeps that directory alive so every client agrees on the default socket path. An earlier Mac-side `XDG_RUNTIME_DIR` pin fixed the CLI but broke the GUI's plain-ssh "Add a machine"; linger supersedes it and the wrapper is retained but unused. +- **[SERVICE]** `/boot/config/go` now restores the hc binary **and** re-applies linger at boot, since `/`, `/run` and `/var/lib` are all RAM on Unraid. `sh -n` clean, backups at `/boot/config/go.bak-*`. +- **[SERVICE]** Corrected a stale `~/.ssh/config` entry that still pointed `unraid` at the deleted node `100.100.208.70` on port 422 — an address and port combination that never worked, since sshd binds only 192.168.10.20 and 192.168.31.250. Replaced with `unraid` (LAN, key, 422) and `unraid-ts` (100.100.208.69, Tailscale SSH, 22). This stale entry was also what the Herdr GUI was listing. - **[SERVICE]** Unraid added to Herdr as the machine `unraid` (`hc` 0.1.17 on the remote). `herdr-core machine add` had been failing with `core did not start within 5 s`: `hc` puts its core socket under `XDG_RUNTIME_DIR`, which on Unraid is `/run/user/0` — a directory elogind creates and destroys per login session, so the socket disappeared the moment the session that started the core ended. The core kept running with an unlinked socket, and `core.log` stayed empty because the failure is client side. - **[SERVICE]** Fixed from the Mac only, via `~/.local/bin/herdr-via-unraid`, a `--via` transport that exports `XDG_RUNTIME_DIR=/var/run/herdr-rt` on every connection. Nothing on the server was changed. Uses key auth on 192.168.10.20:422 rather than Tailscale SSH, because `--via` needs a transport that asks nothing and the default tailnet ACL `check` would demand periodic browser re-auth. - **[SERVICE]** Reboot caveat: `/` is rootfs (RAM) and `/run` is tmpfs on Unraid, so `/root/.local/bin/herdr-core` is lost at every boot and needs `herdr-core machine update` to restore. A `/boot/config/go` entry now restores it at boot from `/boot/config/herdr/herdr-core` (`install -m 755`, because /boot is FAT32 and carries no execute bit). Applied 2026-11-03, with the go script backed up to `go.bak-20261103-203916` and verified by deleting the binary and running the line by hand. Detail in `docs/14-HERDR-REMOTE-MACHINES.md`.