infrastructure/docs/14-HERDR-REMOTE-MACHINES.md

Herdr (hc) Remote Machines — Unraid

Herdr (Herdr.app, bundle dev.herdr.app, CLI herdr-core aka hc) runs programs and coding agents in terminals that stay alive, including on remote machines reached over SSH. The Unraid server is registered as the machine unraid.

Added 2026-11-03.

Registration

The transport is a wrapper on the Mac, not a plain SSH target:

herdr-core machine add --via ~/.local/bin/herdr-via-unraid --name unraid --install

unraid | state: connected | host: xtrm-unraid | hc: 0.1.17

Verify with herdr-core exec --on unraid 'hostname' → XTRM-Unraid.

Why a wrapper and not machine add root@192.168.10.20:422

hc starts a core process (herdr-core serve) on the remote and talks to it over a unix socket under XDG_RUNTIME_DIR. On Unraid that resolves to /run/user/0, which elogind creates and destroys per login session. So:

1. machine add opens an SSH session and starts the core. The socket lands in /run/user/0/herdr-core/default.sock.
2. The session ends. elogind removes /run/user/0, taking the socket with it.
3. The next client connects in a new session, finds a fresh empty /run/user/0, and fails with core did not start within 5 s.

The core is not at fault — it keeps running, holding a socket that has been unlinked from the filesystem. core.log stays 0 bytes because the failure is client side. Unraid also warns hc: no systemd user manager here: the core ends with whatever started it, which is why hc cannot supervise its own core here.

The fix is to pin XDG_RUNTIME_DIR to a path that outlives a login session. Doing it in the --via transport keeps the change on the Mac and leaves the server's shell configuration untouched.

~/.local/bin/herdr-via-unraid:

#!/bin/sh
# Herdr appends the shell line to run as the last argument.
exec ssh -T -o BatchMode=yes -o ConnectTimeout=10 \
    -p 422 -i "$HOME/.ssh/id_ed25519_unraid" root@192.168.10.20 \
    "export XDG_RUNTIME_DIR=/var/run/herdr-rt; mkdir -p \"\$XDG_RUNTIME_DIR\" && chmod 700 \"\$XDG_RUNTIME_DIR\"; $*"

Working from outside the LAN

No extra configuration is needed. The transport targets 192.168.10.20:422, and a Mac away from home reaches that over Tailscale via the mikrotik-tailscale subnet router, which is primary for 192.168.10.0/24. This is not a special remote mode — it is the same path used on VLAN 20 at home, since that is also a different subnet from Unraid.

Prerequisites on the client:

Requirement	Check
Tailscale running and logged in	tailscale status
Accepts the hAP's subnet routes	tailscale debug prefs shows "RouteAll": true
Route resolves over Tailscale	route -n get 192.168.10.20 reports utun*, not en0

Fallback leg

The LAN path depends on the mikrotik-tailscale container, which has failed before (see the 2026-08-10 recovery note in docs/09-TAILSCALE-VPN.md). If it dies while Unraid stays up, 192.168.10.20 becomes unreachable over Tailscale but Unraid's own node does not.

So the wrapper probes 192.168.10.20:422 first and falls back to Tailscale SSH on 100.100.208.69:22. The probe runs before any ssh starts, because a failed first attempt would consume stdin and corrupt the retry. It costs about 245 ms on the success path. HERDR_VIA_TARGET=lan|ts forces a leg for testing; both were verified 2026-11-03.

The fallback leg is fully unattended since the tailnet SSH ACL was set to accept on 2026-11-03. Verified the same day with BatchMode=yes: a plain command, a -L port forward (HTTP 302 from the Unraid web UI) and the Herdr leg itself all ran without a prompt.

Why the LAN address is the primary and not the Tailscale IP

--via requires a transport that passes bytes untouched and asks nothing. When this was set up, Tailscale SSH sat behind the default tailnet ACL action: "check", which demanded browser re-authentication every ~12 h — fine interactively, fatal for a long-lived Herdr connection. That ACL was changed to accept on 2026-11-03, so either address would now serve.

The LAN address stays primary anyway: key auth on 192.168.10.20:422 depends on nothing but the SSH key, while Tailscale SSH depends on the tailnet policy staying permissive. From outside the LAN it still travels over Tailscale via the MikroTik subnet router (see docs/09-TAILSCALE-VPN.md), so there is no loss.

Reboot persistence

Unraid runs / from rootfs (RAM) and /run as tmpfs, so both of these are lost on every reboot:

Path	Lost on reboot	Recovery
/var/run/herdr-rt	yes	self-healing — the --via wrapper recreates it on every connection
/root/.local/bin/herdr-core	yes	manual — herdr-core machine update, or machine add --install again

To make the binary survive, stash it on the boot flash and restore it from the go script, in the same style as the existing install/cp lines there:

mkdir -p /boot/config/herdr
cp /root/.local/bin/herdr-core /boot/config/herdr/herdr-core

then append to /boot/config/go:

# Herdr (hc) remote machines. Unraid runs / from RAM, so the hc binary installed
# by `herdr-core machine add --install` is lost on reboot. Restore it from /boot.
# The runtime dir is not restored here: the Mac-side --via wrapper exports
# XDG_RUNTIME_DIR=/var/run/herdr-rt and creates it on every connection, because
# the default /run/user/0 is destroyed per login session by elogind.
mkdir -p /root/.local/bin
install -m 755 /boot/config/herdr/herdr-core /root/.local/bin/herdr-core 2>/dev/null

install -m 755 matters: /boot is FAT32 and does not carry the execute bit.

Status: applied 2026-11-03. The go script was backed up to /boot/config/go.bak-20261103-203916 first.

Verified without rebooting, by deleting /root/.local/bin/herdr-core and running the go line by hand: the binary came back -rwxr-xr-x and reported hc 0.1.17. Herdr stayed connected throughout.

One note if you ever call it over a bare SSH command: ~/.local/bin is not on PATH in a non-interactive shell on Unraid, so herdr-core resolves only by absolute path. Herdr itself always uses the absolute path, so this does not affect normal use.