infrastructure/docs/14-HERDR-REMOTE-MACHINES.md

Herdr (hc) Remote Machines — Unraid

Herdr (Herdr.app, bundle dev.herdr.app, CLI herdr-core aka hc) runs programs and coding agents in terminals that stay alive, including on remote machines reached over SSH. The Unraid server is registered as the machine unraid.

Set up 2026-11-03.

Current configuration

herdr-core machine add root@100.100.208.69 --name unraid --install

unraid | state: connected | target: root@100.100.208.69 | hc: 0.1.17

Verify with herdr-core exec --on unraid 'hostname' → XTRM-Unraid.

The target is Unraid's own Tailscale node over Tailscale SSH (port 22), not the LAN address. Consequences:

• It does not depend on the mikrotik-tailscale subnet router, which has failed before (see the 2026-08-10 recovery note in docs/09-TAILSCALE-VPN.md). Only Unraid itself and the tailnet need to be healthy.
• It works identically at home and away, with no configuration to change. Tailscale still picks the efficient path — on the LAN it reports direct 192.168.10.20:41725 rather than relaying.
• Authentication is tailnet identity, not an SSH key. Access therefore rests on the tailnet SSH policy, which was set to accept on 2026-11-03. Under the previous check policy this would have demanded a browser re-auth every ~12 h and stalled.

Two problems had to be solved first

1. The socket path — core did not start within 5 s

hc runs a core process (herdr-core serve) on the remote and talks to it over a unix socket under XDG_RUNTIME_DIR. On Unraid that is /run/user/0, which pam_elogind.so creates and destroys per login session. So:

1. machine add opens an SSH session and starts the core. The socket lands in /run/user/0/herdr-core/default.sock.
2. The session ends. elogind removes /run/user/0 with the socket inside it.
3. The next client connects in a new session, finds a fresh empty /run/user/0, and times out after 5 s.

The core is not at fault — it keeps running on a socket that has been unlinked. core.log stays 0 bytes because the failure is client side. Unraid also prints hc: no systemd user manager here: the core ends with whatever started it, which is why hc cannot supervise its own core here.

Fix: loginctl enable-linger root. Unraid ships loginctl even without systemd, and linger keeps /run/user/0 alive across logouts, so the default socket path is stable and every client agrees on it.

An earlier attempt pinned XDG_RUNTIME_DIR=/var/run/herdr-rt from a Mac-side --via wrapper. That worked for the CLI but broke the GUI's "Add a machine", which uses plain ssh and so looked in /run/user/0:

the core listens at /var/run/herdr-rt/herdr-core/default.sock,
but this shell looks at /run/user/0/herdr-core/default.sock
hint: set HC_SOCKET to the first path

Linger supersedes that. The wrapper is retained at ~/.local/bin/herdr-via-unraid (LAN primary, Tailscale fallback, chosen by a TCP probe before any ssh runs so a failed attempt cannot consume stdin) but is not in use. To go back to it: herdr-core machine rm unraid, then machine add --via ~/.local/bin/herdr-via-unraid --name unraid.

2. A stale ~/.ssh/config entry

The GUI lists candidates from ~/.ssh/config, which still held the Tailscale IP of the node deleted on 2026-11-03 — and pointed it at a port that was never open there:

Host 100.100.208.70 unraid     # node no longer exists
  HostName 100.100.208.70
  Port 422                      # sshd binds only 192.168.10.20 and 192.168.31.250

Replaced with two correct entries (backup at ~/.ssh/config.bak-20261103):

Host unraid                     # LAN address; routes over Tailscale when away
  HostName 192.168.10.20
  User root
  Port 422
  IdentityFile ~/.ssh/id_ed25519_unraid

Host unraid-ts 100.100.208.69   # Tailscale SSH: port 22, no key
  HostName 100.100.208.69
  User root

Reboot persistence

Unraid runs / from rootfs (RAM) and /run as tmpfs, so all of this is lost on reboot and is restored by /boot/config/go:

Path	Lost on reboot	Restored by
/root/.local/bin/herdr-core	yes	install -m 755 from /boot/config/herdr/herdr-core
/usr/local/bin/hc, /usr/local/bin/herdr-core	yes	ln -sf to the restored binary
/var/lib/elogind/linger/root	yes	loginctl enable-linger root
/run/user/0	yes	recreated by pam_elogind at first login, then held open by linger

The go block:

# Herdr (hc) remote machines. Unraid runs / from RAM, so the hc binary installed
# by `herdr-core machine add --install` is lost on reboot. Restore it from /boot.
mkdir -p /root/.local/bin
install -m 755 /boot/config/herdr/herdr-core /root/.local/bin/herdr-core 2>/dev/null

# Put it on PATH under both names. ~/.local/bin is not on PATH in a
# non-interactive shell, and clients speak the protocol with `ssh box hc connect`,
# so without this they get "hc: command not found".
ln -sf /root/.local/bin/herdr-core /usr/local/bin/hc
ln -sf /root/.local/bin/herdr-core /usr/local/bin/herdr-core

# Keep /run/user/0 alive across logouts. pam_elogind creates and destroys it per
# login session, and the hc core's socket lives there, so without linger the
# socket disappears with the session that started the core and every later
# client fails with "core did not start within 5 s". The linger marker is
# /var/lib/elogind/linger/root, and /var/lib is RAM here, so it must be re-set
# at every boot.
loginctl enable-linger root 2>/dev/null

install -m 755 matters: /boot is FAT32 and carries no execute bit.

Applied 2026-11-03, sh -n /boot/config/go parses cleanly. Backups of the go script are at /boot/config/go.bak-*. The binary restore was verified without rebooting, by deleting /root/.local/bin/herdr-core and running the line by hand: it came back -rwxr-xr-x reporting hc 0.1.17.

Notes

• The binary installs to /root/.local/bin/herdr-core, which is not on PATH in a non-interactive shell and is not named hc. The desktop app works regardless because it uses the absolute path it installed, but any other client fails. Symlinks in /usr/local/bin fix that (restored by the go script above).
• Adding a machine that hc already knows is refused as a duplicate — it keys on the core id (3d4a2397fcb6…), not the address. Re-pointing an existing machine means machine rm then machine add, since there is no command to change a target in place.

Other clients (mobile, app builders)

hc connect speaks the protocol over stdin/stdout, so any client attaches with ssh <box> hc connect. hc schema prints the API as JSON Schema. What a client needs for this server:

Field	Value
Host	100.100.208.69, or xtrm-unraid.tailf789e1.ts.net
Port	22 — not 422; sshd binds only 192.168.10.20 and 192.168.31.250, so 422 is closed on the Tailscale address
User	root
Password / key	none — Tailscale SSH authenticates by tailnet identity
Command	hc connect

The client device must itself be an active node in the tailnet (tailf789e1.ts.net), because Tailscale SSH authorises by source node. The ACL is accept for autogroup:member -> autogroup:self, so the owner's own devices are covered once they are online and logged in.

Before the /usr/local/bin symlinks existed, this failed at the first hurdle:

ssh root@100.100.208.69 'hc connect'
-> /bin/bash: line 1: hc: command not found