jazzymc/infrastructure
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: Herdr on Tailscale SSH, with linger as the real socket fix

Browse Source 
Rewrites the Herdr doc to match what was actually built. The machine now
targets root@100.100.208.69 over Tailscale SSH instead of the LAN address,
so it no longer depends on the mikrotik-tailscale subnet router.

The "core did not start within 5 s" failure was the hc core socket living
under XDG_RUNTIME_DIR = /run/user/0, which pam_elogind creates and destroys
per login session. loginctl enable-linger root keeps that directory alive, so
every client agrees on the default socket path. The earlier Mac-side env pin
fixed the CLI but broke the GUI's plain-ssh "Add a machine"; linger
supersedes it and the wrapper is kept but unused.

Also records the stale ~/.ssh/config entry that pointed unraid at the deleted
100.100.208.70 on port 422, which is what the GUI had been listing, and the
go-script block that restores both the binary and linger at boot.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kaloyan DanchevandClaude Opus 5 committed 2026-11-03 23:37:46 +02:00
1 parent 27f5a7d5af
commit 1613f7f458
2 files changed
+98 -94
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/14-HERDR-REMOTE-MACHINES.md
+94 -94
View File
@@ -5,140 +5,140 @@ programs and coding agents in terminals that stay alive, including on remote
machines reached over SSH. The Unraid server is registered as the machine
`unraid`.
Added 2026-11-03.
Set up 2026-11-03.
## Registration
The transport is a wrapper on the Mac, not a plain SSH target:
## Current configuration
```bash
herdr-core machine add --via ~/.local/bin/herdr-via-unraid --name unraid --install
herdr-core machine add root@100.100.208.69 --name unraid --install
```
```
unraid | state: connected | host: xtrm-unraid | hc: 0.1.17
unraid | state: connected | target: root@100.100.208.69 | hc: 0.1.17
```
Verify with `herdr-core exec --on unraid 'hostname'` → `XTRM-Unraid`.
## Why a wrapper and not `machine add root@192.168.10.20:422`
The target is Unraid's own Tailscale node over Tailscale SSH (port 22), not the
LAN address. Consequences:
`hc` starts a core process (`herdr-core serve`) on the remote and talks to it
over a unix socket under `XDG_RUNTIME_DIR`. On Unraid that resolves to
`/run/user/0`, which **elogind creates and destroys per login session**. So:
- It does not depend on the `mikrotik-tailscale` subnet router, which has failed
before (see the 2026-08-10 recovery note in `docs/09-TAILSCALE-VPN.md`). Only
Unraid itself and the tailnet need to be healthy.
- It works identically at home and away, with no configuration to change.
Tailscale still picks the efficient path — on the LAN it reports
`direct 192.168.10.20:41725` rather than relaying.
- Authentication is tailnet identity, not an SSH key. Access therefore rests on
the tailnet SSH policy, which was set to `accept` on 2026-11-03. Under the
previous `check` policy this would have demanded a browser re-auth every
~12 h and stalled.
## Two problems had to be solved first
### 1. The socket path — `core did not start within 5 s`
`hc` runs a core process (`herdr-core serve`) on the remote and talks to it over
a unix socket under `XDG_RUNTIME_DIR`. On Unraid that is `/run/user/0`, which
**`pam_elogind.so` creates and destroys per login session**. So:
1. `machine add` opens an SSH session and starts the core. The socket lands in
`/run/user/0/herdr-core/default.sock`.
2. The session ends. elogind removes `/run/user/0`, taking the socket with it.
2. The session ends. elogind removes `/run/user/0` with the socket inside it.
3. The next client connects in a new session, finds a fresh empty
`/run/user/0`, and fails with `core did not start within 5 s`.
`/run/user/0`, and times out after 5 s.
The core is not at fault — it keeps running, holding a socket that has been
unlinked from the filesystem. `core.log` stays 0 bytes because the failure is
client side. Unraid also warns `hc: no systemd user manager here: the core ends
with whatever started it`, which is why `hc` cannot supervise its own core here.
The core is not at fault — it keeps running on a socket that has been unlinked.
`core.log` stays 0 bytes because the failure is client side. Unraid also prints
`hc: no systemd user manager here: the core ends with whatever started it`,
which is why `hc` cannot supervise its own core here.
The fix is to pin `XDG_RUNTIME_DIR` to a path that outlives a login session.
Doing it in the `--via` transport keeps the change on the Mac and leaves the
server's shell configuration untouched.
**Fix: `loginctl enable-linger root`.** Unraid ships `loginctl` even without
systemd, and linger keeps `/run/user/0` alive across logouts, so the default
socket path is stable and every client agrees on it.
`~/.local/bin/herdr-via-unraid`:
An earlier attempt pinned `XDG_RUNTIME_DIR=/var/run/herdr-rt` from a Mac-side
`--via` wrapper. That worked for the CLI but broke the GUI's "Add a machine",
which uses plain ssh and so looked in `/run/user/0`:
```sh
#!/bin/sh
# Herdr appends the shell line to run as the last argument.
exec ssh -T -o BatchMode=yes -o ConnectTimeout=10 \
-p 422 -i "$HOME/.ssh/id_ed25519_unraid" root@192.168.10.20 \
"export XDG_RUNTIME_DIR=/var/run/herdr-rt; mkdir -p \"\$XDG_RUNTIME_DIR\" && chmod 700 \"\$XDG_RUNTIME_DIR\"; $*"
```
the core listens at /var/run/herdr-rt/herdr-core/default.sock,
but this shell looks at /run/user/0/herdr-core/default.sock
hint: set HC_SOCKET to the first path
```
## Working from outside the LAN
Linger supersedes that. The wrapper is retained at
`~/.local/bin/herdr-via-unraid` (LAN primary, Tailscale fallback, chosen by a
TCP probe before any ssh runs so a failed attempt cannot consume stdin) but is
**not in use**. To go back to it: `herdr-core machine rm unraid`, then
`machine add --via ~/.local/bin/herdr-via-unraid --name unraid`.
No extra configuration is needed. The transport targets `192.168.10.20:422`, and
a Mac away from home reaches that over Tailscale via the `mikrotik-tailscale`
subnet router, which is primary for `192.168.10.0/24`. This is not a special
remote mode — it is the same path used on VLAN 20 at home, since that is also a
different subnet from Unraid.
### 2. A stale `~/.ssh/config` entry
Prerequisites on the client:
The GUI lists candidates from `~/.ssh/config`, which still held the Tailscale IP
of the node deleted on 2026-11-03 — and pointed it at a port that was never open
there:
| Requirement | Check |
|-------------|-------|
| Tailscale running and logged in | `tailscale status` |
| Accepts the hAP's subnet routes | `tailscale debug prefs` shows `"RouteAll": true` |
| Route resolves over Tailscale | `route -n get 192.168.10.20` reports `utun*`, not `en0` |
```
Host 100.100.208.70 unraid # node no longer exists
HostName 100.100.208.70
Port 422 # sshd binds only 192.168.10.20 and 192.168.31.250
```
### Fallback leg
Replaced with two correct entries (backup at `~/.ssh/config.bak-20261103`):
The LAN path depends on the `mikrotik-tailscale` container, which has failed
before (see the 2026-08-10 recovery note in `docs/09-TAILSCALE-VPN.md`). If it
dies while Unraid stays up, `192.168.10.20` becomes unreachable over Tailscale
but Unraid's own node does not.
```
Host unraid # LAN address; routes over Tailscale when away
HostName 192.168.10.20
User root
Port 422
IdentityFile ~/.ssh/id_ed25519_unraid
So the wrapper probes `192.168.10.20:422` first and falls back to Tailscale SSH
on `100.100.208.69:22`. The probe runs **before** any ssh starts, because a
failed first attempt would consume stdin and corrupt the retry. It costs about
245 ms on the success path. `HERDR_VIA_TARGET=lan|ts` forces a leg for testing;
both were verified 2026-11-03.
The fallback leg is fully unattended since the tailnet SSH ACL was set to
`accept` on 2026-11-03. Verified the same day with `BatchMode=yes`: a plain
command, a `-L` port forward (HTTP 302 from the Unraid web UI) and the Herdr leg
itself all ran without a prompt.
## Why the LAN address is the primary and not the Tailscale IP
`--via` requires a transport that passes bytes untouched and **asks nothing**.
When this was set up, Tailscale SSH sat behind the default tailnet ACL
`action: "check"`, which demanded browser re-authentication every ~12 h — fine
interactively, fatal for a long-lived Herdr connection. That ACL was changed to
`accept` on 2026-11-03, so either address would now serve.
The LAN address stays primary anyway: key auth on `192.168.10.20:422` depends on
nothing but the SSH key, while Tailscale SSH depends on the tailnet policy
staying permissive. From outside the LAN it still travels over Tailscale via the
MikroTik subnet router (see `docs/09-TAILSCALE-VPN.md`), so there is no loss.
Host unraid-ts 100.100.208.69 # Tailscale SSH: port 22, no key
HostName 100.100.208.69
User root
```
## Reboot persistence
Unraid runs `/` from `rootfs` (RAM) and `/run` as tmpfs, so both of these are
lost on every reboot:
Unraid runs `/` from `rootfs` (RAM) and `/run` as tmpfs, so all of this is lost
on reboot and is restored by `/boot/config/go`:
| Path | Lost on reboot | Recovery |
|------|----------------|----------|
| `/var/run/herdr-rt` | yes | self-healing — the `--via` wrapper recreates it on every connection |
| `/root/.local/bin/herdr-core` | yes | **manual** — `herdr-core machine update`, or `machine add --install` again |
| Path | Lost on reboot | Restored by |
|------|----------------|-------------|
| `/root/.local/bin/herdr-core` | yes | `install -m 755` from `/boot/config/herdr/herdr-core` |
| `/var/lib/elogind/linger/root` | yes | `loginctl enable-linger root` |
| `/run/user/0` | yes | recreated by pam_elogind at first login, then held open by linger |
To make the binary survive, stash it on the boot flash and restore it from the
`go` script, in the same style as the existing `install`/`cp` lines there:
```bash
mkdir -p /boot/config/herdr
cp /root/.local/bin/herdr-core /boot/config/herdr/herdr-core
```
then append to `/boot/config/go`:
The `go` block:
```bash
# Herdr (hc) remote machines. Unraid runs / from RAM, so the hc binary installed
# by `herdr-core machine add --install` is lost on reboot. Restore it from /boot.
# The runtime dir is not restored here: the Mac-side --via wrapper exports
# XDG_RUNTIME_DIR=/var/run/herdr-rt and creates it on every connection, because
# the default /run/user/0 is destroyed per login session by elogind.
mkdir -p /root/.local/bin
install -m 755 /boot/config/herdr/herdr-core /root/.local/bin/herdr-core 2>/dev/null
# Keep /run/user/0 alive across logouts. pam_elogind creates and destroys it per
# login session, and the hc core's socket lives there, so without linger the
# socket disappears with the session that started the core and every later
# client fails with "core did not start within 5 s". The linger marker is
# /var/lib/elogind/linger/root, and /var/lib is RAM here, so it must be re-set
# at every boot.
loginctl enable-linger root 2>/dev/null
```
`install -m 755` matters: `/boot` is FAT32 and does not carry the execute bit.
`install -m 755` matters: `/boot` is FAT32 and carries no execute bit.
**Status: applied 2026-11-03.** The `go` script was backed up to
`/boot/config/go.bak-20261103-203916` first.
Applied 2026-11-03, `sh -n /boot/config/go` parses cleanly. Backups of the go
script are at `/boot/config/go.bak-*`. The binary restore was verified without
rebooting, by deleting `/root/.local/bin/herdr-core` and running the line by
hand: it came back `-rwxr-xr-x` reporting `hc 0.1.17`.
Verified without rebooting, by deleting `/root/.local/bin/herdr-core` and running
the `go` line by hand: the binary came back `-rwxr-xr-x` and reported `hc 0.1.17`.
Herdr stayed `connected` throughout.
## Notes
One note if you ever call it over a bare SSH command: `~/.local/bin` is not on
`PATH` in a non-interactive shell on Unraid, so `herdr-core` resolves only by
absolute path. Herdr itself always uses the absolute path, so this does not
affect normal use.
- `~/.local/bin` is not on `PATH` in a non-interactive shell on Unraid, so
`herdr-core` resolves only by absolute path there. Herdr itself always uses
the absolute path, so this does not affect normal use.
- Adding a machine that `hc` already knows is refused as a duplicate — it keys
on the core id (`3d4a2397fcb6…`), not the address. Re-pointing an existing
machine means `machine rm` then `machine add`, since there is no command to
change a target in place.
docs/CHANGELOG.md
+4
View File
@@ -19,6 +19,10 @@
- Trade-off recorded in the docs: `check` acted as a second factor, so a compromised tailnet device now reaches root on Unraid without a further step. Accepted because every node in the tailnet is owner-controlled hardware.
### Tooling: Unraid registered as a Herdr (hc) machine
- **[SERVICE]** Final state: the machine `unraid` targets `root@100.100.208.69` over Tailscale SSH, so it no longer depends on the `mikrotik-tailscale` subnet router. Works identically at home and away; Tailscale still reports `direct 192.168.10.20:41725` on the LAN rather than relaying.
- **[SERVICE]** Root cause of `core did not start within 5 s` was the hc core socket living under `XDG_RUNTIME_DIR` = `/run/user/0`, which `pam_elogind.so` creates and destroys per login session. Fixed properly with `loginctl enable-linger root`, which keeps that directory alive so every client agrees on the default socket path. An earlier Mac-side `XDG_RUNTIME_DIR` pin fixed the CLI but broke the GUI's plain-ssh "Add a machine"; linger supersedes it and the wrapper is retained but unused.
- **[SERVICE]** `/boot/config/go` now restores the hc binary **and** re-applies linger at boot, since `/`, `/run` and `/var/lib` are all RAM on Unraid. `sh -n` clean, backups at `/boot/config/go.bak-*`.
- **[SERVICE]** Corrected a stale `~/.ssh/config` entry that still pointed `unraid` at the deleted node `100.100.208.70` on port 422 — an address and port combination that never worked, since sshd binds only 192.168.10.20 and 192.168.31.250. Replaced with `unraid` (LAN, key, 422) and `unraid-ts` (100.100.208.69, Tailscale SSH, 22). This stale entry was also what the Herdr GUI was listing.
- **[SERVICE]** Unraid added to Herdr as the machine `unraid` (`hc` 0.1.17 on the remote). `herdr-core machine add` had been failing with `core did not start within 5 s`: `hc` puts its core socket under `XDG_RUNTIME_DIR`, which on Unraid is `/run/user/0` — a directory elogind creates and destroys per login session, so the socket disappeared the moment the session that started the core ended. The core kept running with an unlinked socket, and `core.log` stayed empty because the failure is client side.
- **[SERVICE]** Fixed from the Mac only, via `~/.local/bin/herdr-via-unraid`, a `--via` transport that exports `XDG_RUNTIME_DIR=/var/run/herdr-rt` on every connection. Nothing on the server was changed. Uses key auth on 192.168.10.20:422 rather than Tailscale SSH, because `--via` needs a transport that asks nothing and the default tailnet ACL `check` would demand periodic browser re-auth.
- **[SERVICE]** Reboot caveat: `/` is rootfs (RAM) and `/run` is tmpfs on Unraid, so `/root/.local/bin/herdr-core` is lost at every boot and needs `herdr-core machine update` to restore. A `/boot/config/go` entry now restores it at boot from `/boot/config/herdr/herdr-core` (`install -m 755`, because /boot is FAT32 and carries no execute bit). Applied 2026-11-03, with the go script backed up to `go.bak-20261103-203916` and verified by deleting the binary and running the line by hand. Detail in `docs/14-HERDR-REMOTE-MACHINES.md`.