infrastructure/docs/14-HERDR-REMOTE-MACHINES.md

# Herdr (hc) Remote Machines — Unraid

Herdr (`Herdr.app`, bundle `dev.herdr.app`, CLI `herdr-core` aka `hc`) runs
programs and coding agents in terminals that stay alive, including on remote
machines reached over SSH. The Unraid server is registered as the machine
`unraid`.

Set up 2026-11-03.

## Current configuration

```bash
herdr-core machine add root@100.100.208.69 --name unraid --install
```

```
unraid | state: connected | target: root@100.100.208.69 | hc: 0.1.17
```

Verify with `herdr-core exec --on unraid 'hostname'` → `XTRM-Unraid`.

The target is Unraid's own Tailscale node over Tailscale SSH (port 22), not the
LAN address. Consequences:

- It does not depend on the `mikrotik-tailscale` subnet router, which has failed
  before (see the 2026-08-10 recovery note in `docs/09-TAILSCALE-VPN.md`). Only
  Unraid itself and the tailnet need to be healthy.
- It works identically at home and away, with no configuration to change.
  Tailscale still picks the efficient path — on the LAN it reports
  `direct 192.168.10.20:41725` rather than relaying.
- Authentication is tailnet identity, not an SSH key. Access therefore rests on
  the tailnet SSH policy, which was set to `accept` on 2026-11-03. Under the
  previous `check` policy this would have demanded a browser re-auth every
  ~12 h and stalled.

## Two problems had to be solved first

### 1. The socket path — `core did not start within 5 s`

`hc` runs a core process (`herdr-core serve`) on the remote and talks to it over
a unix socket under `XDG_RUNTIME_DIR`. On Unraid that is `/run/user/0`, which
**`pam_elogind.so` creates and destroys per login session**. So:

1. `machine add` opens an SSH session and starts the core. The socket lands in
   `/run/user/0/herdr-core/default.sock`.
2. The session ends. elogind removes `/run/user/0` with the socket inside it.
3. The next client connects in a new session, finds a fresh empty
   `/run/user/0`, and times out after 5 s.

The core is not at fault — it keeps running on a socket that has been unlinked.
`core.log` stays 0 bytes because the failure is client side. Unraid also prints
`hc: no systemd user manager here: the core ends with whatever started it`,
which is why `hc` cannot supervise its own core here.

**Fix: `loginctl enable-linger root`.** Unraid ships `loginctl` even without
systemd, and linger keeps `/run/user/0` alive across logouts, so the default
socket path is stable and every client agrees on it.

An earlier attempt pinned `XDG_RUNTIME_DIR=/var/run/herdr-rt` from a Mac-side
`--via` wrapper. That worked for the CLI but broke the GUI's "Add a machine",
which uses plain ssh and so looked in `/run/user/0`:

```
the core listens at /var/run/herdr-rt/herdr-core/default.sock,
but this shell looks at /run/user/0/herdr-core/default.sock
hint: set HC_SOCKET to the first path
```

Linger supersedes that. The wrapper is retained at
`~/.local/bin/herdr-via-unraid` (LAN primary, Tailscale fallback, chosen by a
TCP probe before any ssh runs so a failed attempt cannot consume stdin) but is
**not in use**. To go back to it: `herdr-core machine rm unraid`, then
`machine add --via ~/.local/bin/herdr-via-unraid --name unraid`.

### 2. A stale `~/.ssh/config` entry

The GUI lists candidates from `~/.ssh/config`, which still held the Tailscale IP
of the node deleted on 2026-11-03 — and pointed it at a port that was never open
there:

```
Host 100.100.208.70 unraid     # node no longer exists
  HostName 100.100.208.70
  Port 422                      # sshd binds only 192.168.10.20 and 192.168.31.250
```

Replaced with two correct entries (backup at `~/.ssh/config.bak-20261103`):

```
Host unraid                     # LAN address; routes over Tailscale when away
  HostName 192.168.10.20
  User root
  Port 422
  IdentityFile ~/.ssh/id_ed25519_unraid

Host unraid-ts 100.100.208.69   # Tailscale SSH: port 22, no key
  HostName 100.100.208.69
  User root
```

## Reboot persistence

Unraid runs `/` from `rootfs` (RAM) and `/run` as tmpfs, so all of this is lost
on reboot and is restored by `/boot/config/go`:

| Path | Lost on reboot | Restored by |
|------|----------------|-------------|
| `/root/.local/bin/herdr-core` | yes | `install -m 755` from `/boot/config/herdr/herdr-core` |
| `/usr/local/bin/hc`, `/usr/local/bin/herdr-core` | yes | `ln -sf` to the restored binary |
| `/var/lib/elogind/linger/root` | yes | `loginctl enable-linger root` |
| `/run/user/0` | yes | recreated by pam_elogind at first login, then held open by linger |

The `go` block:

```bash
# Herdr (hc) remote machines. Unraid runs / from RAM, so the hc binary installed
# by `herdr-core machine add --install` is lost on reboot. Restore it from /boot.
mkdir -p /root/.local/bin
install -m 755 /boot/config/herdr/herdr-core /root/.local/bin/herdr-core 2>/dev/null

# Put it on PATH under both names. ~/.local/bin is not on PATH in a
# non-interactive shell, and clients speak the protocol with `ssh box hc connect`,
# so without this they get "hc: command not found".
ln -sf /root/.local/bin/herdr-core /usr/local/bin/hc
ln -sf /root/.local/bin/herdr-core /usr/local/bin/herdr-core

# Keep /run/user/0 alive across logouts. pam_elogind creates and destroys it per
# login session, and the hc core's socket lives there, so without linger the
# socket disappears with the session that started the core and every later
# client fails with "core did not start within 5 s". The linger marker is
# /var/lib/elogind/linger/root, and /var/lib is RAM here, so it must be re-set
# at every boot.
loginctl enable-linger root 2>/dev/null
```

`install -m 755` matters: `/boot` is FAT32 and carries no execute bit.

Applied 2026-11-03, `sh -n /boot/config/go` parses cleanly. Backups of the go
script are at `/boot/config/go.bak-*`. The binary restore was verified without
rebooting, by deleting `/root/.local/bin/herdr-core` and running the line by
hand: it came back `-rwxr-xr-x` reporting `hc 0.1.17`.

## Notes

- The binary installs to `/root/.local/bin/herdr-core`, which is not on `PATH`
  in a non-interactive shell and is not named `hc`. The desktop app works
  regardless because it uses the absolute path it installed, but any other
  client fails. Symlinks in `/usr/local/bin` fix that (restored by the go
  script above).
- Adding a machine that `hc` already knows is refused as a duplicate — it keys
  on the core id (`3d4a2397fcb6…`), not the address. Re-pointing an existing
  machine means `machine rm` then `machine add`, since there is no command to
  change a target in place.

## Other clients (mobile, app builders)

`hc connect` speaks the protocol over stdin/stdout, so any client attaches with
`ssh <box> hc connect`. `hc schema` prints the API as JSON Schema. What a client
needs for this server:

| Field | Value |
|-------|-------|
| Host | `100.100.208.69`, or `xtrm-unraid.tailf789e1.ts.net` |
| Port | **22** — not 422; sshd binds only 192.168.10.20 and 192.168.31.250, so 422 is closed on the Tailscale address |
| User | `root` |
| Password / key | none — Tailscale SSH authenticates by tailnet identity |
| Command | `hc connect` |

The client device must itself be an active node in the tailnet
(`tailf789e1.ts.net`), because Tailscale SSH authorises by source node. The ACL
is `accept` for `autogroup:member` -> `autogroup:self`, so the owner's own
devices are covered once they are online and logged in.

Before the `/usr/local/bin` symlinks existed, this failed at the first hurdle:

```
ssh root@100.100.208.69 'hc connect'
-> /bin/bash: line 1: hc: command not found
```