infrastructure/docs/14-HERDR-REMOTE-MACHINES.md

Herdr (hc) Remote Machines — Unraid

Herdr (Herdr.app, bundle dev.herdr.app, CLI herdr-core aka hc) runs programs and coding agents in terminals that stay alive, including on remote machines reached over SSH. The Unraid server is registered as the machine unraid.

Added 2026-11-03.

Registration

The transport is a wrapper on the Mac, not a plain SSH target:

herdr-core machine add --via ~/.local/bin/herdr-via-unraid --name unraid --install

unraid | state: connected | host: xtrm-unraid | hc: 0.1.17

Verify with herdr-core exec --on unraid 'hostname' → XTRM-Unraid.

Why a wrapper and not machine add root@192.168.10.20:422

hc starts a core process (herdr-core serve) on the remote and talks to it over a unix socket under XDG_RUNTIME_DIR. On Unraid that resolves to /run/user/0, which elogind creates and destroys per login session. So:

1. machine add opens an SSH session and starts the core. The socket lands in /run/user/0/herdr-core/default.sock.
2. The session ends. elogind removes /run/user/0, taking the socket with it.
3. The next client connects in a new session, finds a fresh empty /run/user/0, and fails with core did not start within 5 s.

The core is not at fault — it keeps running, holding a socket that has been unlinked from the filesystem. core.log stays 0 bytes because the failure is client side. Unraid also warns hc: no systemd user manager here: the core ends with whatever started it, which is why hc cannot supervise its own core here.

The fix is to pin XDG_RUNTIME_DIR to a path that outlives a login session. Doing it in the --via transport keeps the change on the Mac and leaves the server's shell configuration untouched.

~/.local/bin/herdr-via-unraid:

#!/bin/sh
# Herdr appends the shell line to run as the last argument.
exec ssh -T -o BatchMode=yes -o ConnectTimeout=10 \
    -p 422 -i "$HOME/.ssh/id_ed25519_unraid" root@192.168.10.20 \
    "export XDG_RUNTIME_DIR=/var/run/herdr-rt; mkdir -p \"\$XDG_RUNTIME_DIR\" && chmod 700 \"\$XDG_RUNTIME_DIR\"; $*"

Why the LAN address and not the Tailscale IP

--via requires a transport that passes bytes untouched and asks nothing. Tailscale SSH on 100.100.208.69 sits behind the default tailnet ACL action: "check", which demands browser re-authentication each period — fine interactively, fatal for a long-lived Herdr connection.

Key auth on 192.168.10.20:422 asks nothing, and from outside the LAN it still travels over Tailscale via the MikroTik subnet router (see docs/09-TAILSCALE-VPN.md). Changing that ACL rule to accept for autogroup:self would make the 100.x address usable here too.

Reboot persistence

Unraid runs / from rootfs (RAM) and /run as tmpfs, so both of these are lost on every reboot:

Path	Lost on reboot	Recovery
/var/run/herdr-rt	yes	self-healing — the --via wrapper recreates it on every connection
/root/.local/bin/herdr-core	yes	manual — herdr-core machine update, or machine add --install again

To make the binary survive, stash it on the boot flash and restore it from the go script, in the same style as the existing install/cp lines there:

mkdir -p /boot/config/herdr
cp /root/.local/bin/herdr-core /boot/config/herdr/herdr-core

then append to /boot/config/go:

# Herdr (hc) remote machines. Unraid runs / from RAM, so the hc binary installed
# by `herdr-core machine add --install` is lost on reboot. Restore it from /boot.
# The runtime dir is not restored here: the Mac-side --via wrapper exports
# XDG_RUNTIME_DIR=/var/run/herdr-rt and creates it on every connection, because
# the default /run/user/0 is destroyed per login session by elogind.
mkdir -p /root/.local/bin
install -m 755 /boot/config/herdr/herdr-core /root/.local/bin/herdr-core 2>/dev/null

install -m 755 matters: /boot is FAT32 and does not carry the execute bit.

Status: applied 2026-11-03. The go script was backed up to /boot/config/go.bak-20261103-203916 first.

Verified without rebooting, by deleting /root/.local/bin/herdr-core and running the go line by hand: the binary came back -rwxr-xr-x and reported hc 0.1.17. Herdr stayed connected throughout.

One note if you ever call it over a bare SSH command: ~/.local/bin is not on PATH in a non-interactive shell on Unraid, so herdr-core resolves only by absolute path. Herdr itself always uses the absolute path, so this does not affect normal use.