infrastructure/docs/14-HERDR-REMOTE-MACHINES.md

# Herdr (hc) Remote Machines — Unraid

Herdr (`Herdr.app`, bundle `dev.herdr.app`, CLI `herdr-core` aka `hc`) runs
programs and coding agents in terminals that stay alive, including on remote
machines reached over SSH. The Unraid server is registered as the machine
`unraid`.

Set up 2026-11-03.

## Current configuration

```bash
herdr-core machine add root@100.100.208.69 --name unraid --install
```

```
unraid | state: connected | target: root@100.100.208.69 | hc: 0.1.17
```

Verify with `herdr-core exec --on unraid 'hostname'` → `XTRM-Unraid`.

The target is Unraid's own Tailscale node over Tailscale SSH (port 22), not the
LAN address. Consequences:

- It does not depend on the `mikrotik-tailscale` subnet router, which has failed
  before (see the 2026-08-10 recovery note in `docs/09-TAILSCALE-VPN.md`). Only
  Unraid itself and the tailnet need to be healthy.
- It works identically at home and away, with no configuration to change.
  Tailscale still picks the efficient path — on the LAN it reports
  `direct 192.168.10.20:41725` rather than relaying.
- Authentication is tailnet identity, not an SSH key. Access therefore rests on
  the tailnet SSH policy, which was set to `accept` on 2026-11-03. Under the
  previous `check` policy this would have demanded a browser re-auth every
  ~12 h and stalled.

## Two problems had to be solved first

### 1. The socket path — `core did not start within 5 s`

`hc` runs a core process (`herdr-core serve`) on the remote and talks to it over
a unix socket under `XDG_RUNTIME_DIR`. On Unraid that is `/run/user/0`, which
**`pam_elogind.so` creates and destroys per login session**. So:

1. `machine add` opens an SSH session and starts the core. The socket lands in
   `/run/user/0/herdr-core/default.sock`.
2. The session ends. elogind removes `/run/user/0` with the socket inside it.
3. The next client connects in a new session, finds a fresh empty
   `/run/user/0`, and times out after 5 s.

The core is not at fault — it keeps running on a socket that has been unlinked.
`core.log` stays 0 bytes because the failure is client side. Unraid also prints
`hc: no systemd user manager here: the core ends with whatever started it`,
which is why `hc` cannot supervise its own core here.

**Fix: `loginctl enable-linger root`.** Unraid ships `loginctl` even without
systemd, and linger keeps `/run/user/0` alive across logouts, so the default
socket path is stable and every client agrees on it.

An earlier attempt pinned `XDG_RUNTIME_DIR=/var/run/herdr-rt` from a Mac-side
`--via` wrapper. That worked for the CLI but broke the GUI's "Add a machine",
which uses plain ssh and so looked in `/run/user/0`:

```
the core listens at /var/run/herdr-rt/herdr-core/default.sock,
but this shell looks at /run/user/0/herdr-core/default.sock
hint: set HC_SOCKET to the first path
```

Linger supersedes that. The wrapper is retained at
`~/.local/bin/herdr-via-unraid` (LAN primary, Tailscale fallback, chosen by a
TCP probe before any ssh runs so a failed attempt cannot consume stdin) but is
**not in use**. To go back to it: `herdr-core machine rm unraid`, then
`machine add --via ~/.local/bin/herdr-via-unraid --name unraid`.

### 2. A stale `~/.ssh/config` entry

The GUI lists candidates from `~/.ssh/config`, which still held the Tailscale IP
of the node deleted on 2026-11-03 — and pointed it at a port that was never open
there:

```
Host 100.100.208.70 unraid     # node no longer exists
  HostName 100.100.208.70
  Port 422                      # sshd binds only 192.168.10.20 and 192.168.31.250
```

Replaced with two correct entries (backup at `~/.ssh/config.bak-20261103`):

```
Host unraid                     # LAN address; routes over Tailscale when away
  HostName 192.168.10.20
  User root
  Port 422
  IdentityFile ~/.ssh/id_ed25519_unraid

Host unraid-ts 100.100.208.69   # Tailscale SSH: port 22, no key
  HostName 100.100.208.69
  User root
```

## Reboot persistence

Unraid runs `/` from `rootfs` (RAM) and `/run` as tmpfs, so all of this is lost
on reboot and is restored by `/boot/config/go`:

| Path | Lost on reboot | Restored by |
|------|----------------|-------------|
| `/root/.local/bin/herdr-core` | yes | `install -m 755` from `/boot/config/herdr/herdr-core` |
| `/usr/local/bin/hc`, `/usr/local/bin/herdr-core` | yes | `ln -sf` to the restored binary |
| `/var/lib/elogind/linger/root` | yes | `loginctl enable-linger root` |
| `/run/user/0` | yes | recreated by pam_elogind at first login, then held open by linger |

The `go` block:

```bash
# Herdr (hc) remote machines. Unraid runs / from RAM, so the hc binary installed
# by `herdr-core machine add --install` is lost on reboot. Restore it from /boot.
mkdir -p /root/.local/bin
install -m 755 /boot/config/herdr/herdr-core /root/.local/bin/herdr-core 2>/dev/null

# Put it on PATH under both names. ~/.local/bin is not on PATH in a
# non-interactive shell, and clients speak the protocol with `ssh box hc connect`,
# so without this they get "hc: command not found".
ln -sf /root/.local/bin/herdr-core /usr/local/bin/hc
ln -sf /root/.local/bin/herdr-core /usr/local/bin/herdr-core

# Keep /run/user/0 alive across logouts. pam_elogind creates and destroys it per
# login session, and the hc core's socket lives there, so without linger the
# socket disappears with the session that started the core and every later
# client fails with "core did not start within 5 s". The linger marker is
# /var/lib/elogind/linger/root, and /var/lib is RAM here, so it must be re-set
# at every boot.
loginctl enable-linger root 2>/dev/null
```

`install -m 755` matters: `/boot` is FAT32 and carries no execute bit.

Applied 2026-11-03, `sh -n /boot/config/go` parses cleanly. Backups of the go
script are at `/boot/config/go.bak-*`. The binary restore was verified without
rebooting, by deleting `/root/.local/bin/herdr-core` and running the line by
hand: it came back `-rwxr-xr-x` reporting `hc 0.1.17`.

### A boot-time restore is not enough on its own

On 2026-11-05 the `hc` binary disappeared with **no reboot**, at 25 days uptime,
so the `go` block above cannot be the only guard.
`/boot/config/herdr/herdr-selfheal.sh` restores the binary from the flash stash,
re-links both `/usr/local/bin` names and re-enables linger if it has lapsed. It
is idempotent and silent when nothing is wrong.

The script existed from 2026-11-05 but **was not scheduled anywhere** — it had
run exactly once, by hand. Scheduled 2026-11-07, every 10 minutes:

```
*/10 * * * * /bin/sh /boot/config/herdr/herdr-selfheal.sh
```

It belongs in the `printf ... | crontab -` line in `/boot/config/go`, because
that line **replaces the whole crontab** at boot: an entry added with
`crontab -e` would be wiped at the next restart. `/bin/sh` is required because
`/boot` is vfat and carries no execute bit. The live crontab and what the `go`
file will set at boot were diffed and are identical, so there is no drift.

Verified without a reboot: deleting `/usr/local/bin/hc` took `hc` off `PATH`,
the script relinked it, logged `relinked /usr/local/bin/hc`, and `hc --version`
reported `0.1.17` again. The core and its 8 live terminals were unaffected —
the running core is already a process and does not re-resolve the symlink.

## File drops (`hc put`) and the `shot` helper

`hc put <terminal> <file>` sends a **local** file to the terminal's machine and
prints where it landed. It takes `machine/t7`, so the same command works whether
the terminal is here or on the Mac — which makes it the transport for getting a
Mac screenshot into a session running on this box:

```
hc put unraid/t13 shot.png
-> {"bytes":67,"path":"/root/.local/state/herdr-core/default/files/t13/shot.png"}
```

| Property | Value |
|----------|-------|
| Destination | `/root/.local/state/herdr-core/<session>/files/<terminal>/<name>` |
| Survives a reboot | **no** — `/root/.local/state` is the RAM rootfs |
| Relocatable | no — 0.1.17 honours no `XDG_*` override for it |
| Extension | preserved, so `.png` stays `.png` |

Right for screenshots, wrong for anything large or worth keeping.

Inside a terminal, `$HC_TERMINAL_ID` is that terminal's own address and
`$HC_TERMINAL_SOCKET` points at `/run/user/0/herdr-core/default.sock`. So a
session can tell a client exactly where to push a file instead of the client
having to guess which terminal is in front of the user. `hc` also records the
client identity per terminal (`"identity":"ghostty"`).

`shot` reads those drops. Real copy at `/boot/config/shell/bin/shot`, installed
to `/root/.local/bin/shot` by `/boot/config/shell/install.sh` — a real
executable rather than a shell function, because a non-interactive shell (an
agent's tool call, an `ssh` command, cron) never sources `.bashrc` but does
inherit `PATH`.

| Command | Result |
|---------|--------|
| `shot` | the newest drop for this terminal, absolute path |
| `shot -l`, `shot -n 3` | drops newest-first |
| `shot -d` | the drop folder |
| `shot -a` | drops across every terminal |

With no drops it prints the `hc put` command that would create one.

### Reaching the Mac from here (not yet set up)

`hc machine ls` on this core returns `{"machines":[]}` — the relationship is
one-directional from the Mac today. Registering the Mac here would add
`hc file read|write|edit --on mac` ("an agent's hands there") and
`hc exec --on mac`, which is how a session on this box gets at Mac files
without a mount. Two caveats: `hc file read` is **text only**, so Mac images
must travel by `hc put`; and the Mac is a laptop, so register it by its tailnet
name (`100.68.118.59`, `kaloyans-macbook-air`) and expect it to be unreachable
when asleep.

Verified 2026-11-07: the Mac answers over Tailscale in 16 ms (direct, via
`192.168.20.101`) and port 22 is open, so macOS Remote Login is already on. It
does **not** advertise Tailscale SSH (`sshHostKeys: false`; macOS cannot act as
a Tailscale SSH server), so unlike this box there is no keyless path — it needs
this host's `unraid@xtrm` key in the Mac's `authorized_keys`. A commented `Host
mac` entry is staged in `/boot/config/ssh/root/config`.

## Notes

- The binary installs to `/root/.local/bin/herdr-core`, which is not on `PATH`
  in a non-interactive shell and is not named `hc`. The desktop app works
  regardless because it uses the absolute path it installed, but any other
  client fails. Symlinks in `/usr/local/bin` fix that (restored by the go
  script above).
- Adding a machine that `hc` already knows is refused as a duplicate — it keys
  on the core id (`3d4a2397fcb6…`), not the address. Re-pointing an existing
  machine means `machine rm` then `machine add`, since there is no command to
  change a target in place.

## Other clients (mobile, app builders)

`hc connect` speaks the protocol over stdin/stdout, so any client attaches with
`ssh <box> hc connect`. `hc schema` prints the API as JSON Schema. What a client
needs for this server:

| Field | Value |
|-------|-------|
| Host | `100.100.208.69`, or `xtrm-unraid.tailf789e1.ts.net` |
| Port | **22** — not 422; sshd binds only 192.168.10.20 and 192.168.31.250, so 422 is closed on the Tailscale address |
| User | `root` |
| Password / key | none — Tailscale SSH authenticates by tailnet identity |
| Command | `hc connect` |

The client device must itself be an active node in the tailnet
(`tailf789e1.ts.net`), because Tailscale SSH authorises by source node. The ACL
is `accept` for `autogroup:member` -> `autogroup:self`, so the owner's own
devices are covered once they are online and logged in.

Before the `/usr/local/bin` symlinks existed, this failed at the first hurdle:

```
ssh root@100.100.208.69 'hc connect'
-> /bin/bash: line 1: hc: command not found
```