infrastructure/docs/14-HERDR-REMOTE-MACHINES.md

Herdr (hc) Remote Machines — Unraid

Herdr (Herdr.app, bundle dev.herdr.app, CLI herdr-core aka hc) runs programs and coding agents in terminals that stay alive, including on remote machines reached over SSH. The Unraid server is registered as the machine unraid.

Set up 2026-11-03.

Current configuration

herdr-core machine add root@100.100.208.69 --name unraid --install

unraid | state: connected | target: root@100.100.208.69 | hc: 0.1.17

Verify with herdr-core exec --on unraid 'hostname' → XTRM-Unraid.

The target is Unraid's own Tailscale node over Tailscale SSH (port 22), not the LAN address. Consequences:

• It does not depend on the mikrotik-tailscale subnet router, which has failed before (see the 2026-08-10 recovery note in docs/09-TAILSCALE-VPN.md). Only Unraid itself and the tailnet need to be healthy.
• It works identically at home and away, with no configuration to change. Tailscale still picks the efficient path — on the LAN it reports direct 192.168.10.20:41725 rather than relaying.
• Authentication is tailnet identity, not an SSH key. Access therefore rests on the tailnet SSH policy, which was set to accept on 2026-11-03. Under the previous check policy this would have demanded a browser re-auth every ~12 h and stalled.

Two problems had to be solved first

1. The socket path — core did not start within 5 s

hc runs a core process (herdr-core serve) on the remote and talks to it over a unix socket under XDG_RUNTIME_DIR. On Unraid that is /run/user/0, which pam_elogind.so creates and destroys per login session. So:

1. machine add opens an SSH session and starts the core. The socket lands in /run/user/0/herdr-core/default.sock.
2. The session ends. elogind removes /run/user/0 with the socket inside it.
3. The next client connects in a new session, finds a fresh empty /run/user/0, and times out after 5 s.

The core is not at fault — it keeps running on a socket that has been unlinked. core.log stays 0 bytes because the failure is client side. Unraid also prints hc: no systemd user manager here: the core ends with whatever started it, which is why hc cannot supervise its own core here.

Fix: loginctl enable-linger root. Unraid ships loginctl even without systemd, and linger keeps /run/user/0 alive across logouts, so the default socket path is stable and every client agrees on it.

An earlier attempt pinned XDG_RUNTIME_DIR=/var/run/herdr-rt from a Mac-side --via wrapper. That worked for the CLI but broke the GUI's "Add a machine", which uses plain ssh and so looked in /run/user/0:

the core listens at /var/run/herdr-rt/herdr-core/default.sock,
but this shell looks at /run/user/0/herdr-core/default.sock
hint: set HC_SOCKET to the first path

Linger supersedes that. The wrapper is retained at ~/.local/bin/herdr-via-unraid (LAN primary, Tailscale fallback, chosen by a TCP probe before any ssh runs so a failed attempt cannot consume stdin) but is not in use. To go back to it: herdr-core machine rm unraid, then machine add --via ~/.local/bin/herdr-via-unraid --name unraid.

2. A stale ~/.ssh/config entry

The GUI lists candidates from ~/.ssh/config, which still held the Tailscale IP of the node deleted on 2026-11-03 — and pointed it at a port that was never open there:

Host 100.100.208.70 unraid     # node no longer exists
  HostName 100.100.208.70
  Port 422                      # sshd binds only 192.168.10.20 and 192.168.31.250

Replaced with two correct entries (backup at ~/.ssh/config.bak-20261103):

Host unraid                     # LAN address; routes over Tailscale when away
  HostName 192.168.10.20
  User root
  Port 422
  IdentityFile ~/.ssh/id_ed25519_unraid

Host unraid-ts 100.100.208.69   # Tailscale SSH: port 22, no key
  HostName 100.100.208.69
  User root

Reboot persistence

Unraid runs / from rootfs (RAM) and /run as tmpfs, so all of this is lost on reboot and is restored by /boot/config/go:

Path	Lost on reboot	Restored by
/root/.local/bin/herdr-core	yes	install -m 755 from /boot/config/herdr/herdr-core
/usr/local/bin/hc, /usr/local/bin/herdr-core	yes	ln -sf to the restored binary
/var/lib/elogind/linger/root	yes	loginctl enable-linger root
/run/user/0	yes	recreated by pam_elogind at first login, then held open by linger

The go block:

# Herdr (hc) remote machines. Unraid runs / from RAM, so the hc binary installed
# by `herdr-core machine add --install` is lost on reboot. Restore it from /boot.
mkdir -p /root/.local/bin
install -m 755 /boot/config/herdr/herdr-core /root/.local/bin/herdr-core 2>/dev/null

# Put it on PATH under both names. ~/.local/bin is not on PATH in a
# non-interactive shell, and clients speak the protocol with `ssh box hc connect`,
# so without this they get "hc: command not found".
ln -sf /root/.local/bin/herdr-core /usr/local/bin/hc
ln -sf /root/.local/bin/herdr-core /usr/local/bin/herdr-core

# Keep /run/user/0 alive across logouts. pam_elogind creates and destroys it per
# login session, and the hc core's socket lives there, so without linger the
# socket disappears with the session that started the core and every later
# client fails with "core did not start within 5 s". The linger marker is
# /var/lib/elogind/linger/root, and /var/lib is RAM here, so it must be re-set
# at every boot.
loginctl enable-linger root 2>/dev/null

install -m 755 matters: /boot is FAT32 and carries no execute bit.

Applied 2026-11-03, sh -n /boot/config/go parses cleanly. Backups of the go script are at /boot/config/go.bak-*. The binary restore was verified without rebooting, by deleting /root/.local/bin/herdr-core and running the line by hand: it came back -rwxr-xr-x reporting hc 0.1.17.

A boot-time restore is not enough on its own

On 2026-11-05 the hc binary disappeared with no reboot, at 25 days uptime, so the go block above cannot be the only guard. /boot/config/herdr/herdr-selfheal.sh restores the binary from the flash stash, re-links both /usr/local/bin names and re-enables linger if it has lapsed. It is idempotent and silent when nothing is wrong.

The script existed from 2026-11-05 but was not scheduled anywhere — it had run exactly once, by hand. Scheduled 2026-11-07, every 10 minutes:

*/10 * * * * /bin/sh /boot/config/herdr/herdr-selfheal.sh

It belongs in the printf ... | crontab - line in /boot/config/go, because that line replaces the whole crontab at boot: an entry added with crontab -e would be wiped at the next restart. /bin/sh is required because /boot is vfat and carries no execute bit. The live crontab and what the go file will set at boot were diffed and are identical, so there is no drift.

Verified without a reboot: deleting /usr/local/bin/hc took hc off PATH, the script relinked it, logged relinked /usr/local/bin/hc, and hc --version reported 0.1.17 again. The core and its 8 live terminals were unaffected — the running core is already a process and does not re-resolve the symlink.

File drops (hc put) and the shot helper

hc put <terminal> <file> sends a local file to the terminal's machine and prints where it landed. It takes machine/t7, so the same command works whether the terminal is here or on the Mac — which makes it the transport for getting a Mac screenshot into a session running on this box:

hc put unraid/t13 shot.png
-> {"bytes":67,"path":"/root/.local/state/herdr-core/default/files/t13/shot.png"}

Property	Value
Destination	/root/.local/state/herdr-core/<session>/files/<terminal>/<name>
Survives a reboot	no — /root/.local/state is the RAM rootfs
Relocatable	no — 0.1.17 honours no XDG_* override for it
Extension	preserved, so .png stays .png

Right for screenshots, wrong for anything large or worth keeping.

Inside a terminal, $HC_TERMINAL_ID is that terminal's own address and $HC_TERMINAL_SOCKET points at /run/user/0/herdr-core/default.sock. So a session can tell a client exactly where to push a file instead of the client having to guess which terminal is in front of the user. hc also records the client identity per terminal ("identity":"ghostty").

shot reads those drops. Real copy at /boot/config/shell/bin/shot, installed to /root/.local/bin/shot by /boot/config/shell/install.sh — a real executable rather than a shell function, because a non-interactive shell (an agent's tool call, an ssh command, cron) never sources .bashrc but does inherit PATH.

Command	Result
shot	the newest drop for this terminal, absolute path
shot -l, shot -n 3	drops newest-first
shot -d	the drop folder
shot -a	drops across every terminal

With no drops it prints the hc put command that would create one.

Reaching the Mac from here (not yet set up)

hc machine ls on this core returns {"machines":[]} — the relationship is one-directional from the Mac today. Registering the Mac here would add hc file read|write|edit --on mac ("an agent's hands there") and hc exec --on mac, which is how a session on this box gets at Mac files without a mount. Two caveats: hc file read is text only, so Mac images must travel by hc put; and the Mac is a laptop, so register it by its tailnet name (100.68.118.59, kaloyans-macbook-air) and expect it to be unreachable when asleep.

Verified 2026-11-07: the Mac answers over Tailscale in 16 ms (direct, via 192.168.20.101) and port 22 is open, so macOS Remote Login is already on. It does not advertise Tailscale SSH (sshHostKeys: false; macOS cannot act as a Tailscale SSH server), so unlike this box there is no keyless path — it needs this host's unraid@xtrm key in the Mac's authorized_keys. A commented Host mac entry is staged in /boot/config/ssh/root/config.

Notes

• The binary installs to /root/.local/bin/herdr-core, which is not on PATH in a non-interactive shell and is not named hc. The desktop app works regardless because it uses the absolute path it installed, but any other client fails. Symlinks in /usr/local/bin fix that (restored by the go script above).
• Adding a machine that hc already knows is refused as a duplicate — it keys on the core id (3d4a2397fcb6…), not the address. Re-pointing an existing machine means machine rm then machine add, since there is no command to change a target in place.

Other clients (mobile, app builders)

hc connect speaks the protocol over stdin/stdout, so any client attaches with ssh <box> hc connect. hc schema prints the API as JSON Schema. What a client needs for this server:

Field	Value
Host	100.100.208.69, or xtrm-unraid.tailf789e1.ts.net
Port	22 — not 422; sshd binds only 192.168.10.20 and 192.168.31.250, so 422 is closed on the Tailscale address
User	root
Password / key	none — Tailscale SSH authenticates by tailnet identity
Command	hc connect

The client device must itself be an active node in the tailnet (tailf789e1.ts.net), because Tailscale SSH authorises by source node. The ACL is accept for autogroup:member -> autogroup:self, so the owner's own devices are covered once they are online and logged in.

Before the /usr/local/bin symlinks existed, this failed at the first hurdle:

ssh root@100.100.208.69 'hc connect'
-> /bin/bash: line 1: hc: command not found