# Herdr (hc) Remote Machines — Unraid

Herdr (`Herdr.app`, bundle `dev.herdr.app`, CLI `herdr-core` aka `hc`) runs
programs and coding agents in terminals that stay alive, including on remote
machines reached over SSH. The Unraid server is registered as the machine
`unraid`.

Set up 2026-11-03.

## Current configuration

```bash
herdr-core machine add root@100.100.208.69 --name unraid --install
```

```
unraid | state: connected | target: root@100.100.208.69 | hc: 0.1.17
```

Verify with `herdr-core exec --on unraid 'hostname'` → `XTRM-Unraid`.

The target is Unraid's own Tailscale node over Tailscale SSH (port 22), not the
LAN address. Consequences:

- It does not depend on the `mikrotik-tailscale` subnet router, which has failed
  before (see the 2026-08-10 recovery note in `docs/09-TAILSCALE-VPN.md`). Only
  Unraid itself and the tailnet need to be healthy.
- It works identically at home and away, with no configuration to change.
  Tailscale still picks the efficient path — on the LAN it reports
  `direct 192.168.10.20:41725` rather than relaying.
- Authentication is tailnet identity, not an SSH key. Access therefore rests on
  the tailnet SSH policy, which was set to `accept` on 2026-11-03. Under the
  previous `check` policy this would have demanded a browser re-auth every
  ~12 h and stalled.

## Two problems had to be solved first

### 1. The socket path — `core did not start within 5 s`

`hc` runs a core process (`herdr-core serve`) on the remote and talks to it over
a unix socket under `XDG_RUNTIME_DIR`. On Unraid that is `/run/user/0`, which
**`pam_elogind.so` creates and destroys per login session**. So:

1. `machine add` opens an SSH session and starts the core. The socket lands in
   `/run/user/0/herdr-core/default.sock`.
2. The session ends. elogind removes `/run/user/0` with the socket inside it.
3. The next client connects in a new session, finds a fresh empty
   `/run/user/0`, and times out after 5 s.

The core is not at fault — it keeps running on a socket that has been unlinked.
`core.log` stays 0 bytes because the failure is client side. Unraid also prints
`hc: no systemd user manager here: the core ends with whatever started it`,
which is why `hc` cannot supervise its own core here.

**Fix: `loginctl enable-linger root`.** Unraid ships `loginctl` even without
systemd, and linger keeps `/run/user/0` alive across logouts, so the default
socket path is stable and every client agrees on it.

An earlier attempt pinned `XDG_RUNTIME_DIR=/var/run/herdr-rt` from a Mac-side
`--via` wrapper. That worked for the CLI but broke the GUI's "Add a machine",
which uses plain ssh and so looked in `/run/user/0`:

```
the core listens at /var/run/herdr-rt/herdr-core/default.sock,
but this shell looks at /run/user/0/herdr-core/default.sock
hint: set HC_SOCKET to the first path
```

Linger supersedes that. The wrapper is retained at
`~/.local/bin/herdr-via-unraid` (LAN primary, Tailscale fallback, chosen by a
TCP probe before any ssh runs so a failed attempt cannot consume stdin) but is
**not in use**. To go back to it: `herdr-core machine rm unraid`, then
`machine add --via ~/.local/bin/herdr-via-unraid --name unraid`.

### 2. A stale `~/.ssh/config` entry

The GUI lists candidates from `~/.ssh/config`, which still held the Tailscale IP
of the node deleted on 2026-11-03 — and pointed it at a port that was never open
there:

```
Host 100.100.208.70 unraid     # node no longer exists
  HostName 100.100.208.70
  Port 422                      # sshd binds only 192.168.10.20 and 192.168.31.250
```

Replaced with two correct entries (backup at `~/.ssh/config.bak-20261103`):

```
Host unraid                     # LAN address; routes over Tailscale when away
  HostName 192.168.10.20
  User root
  Port 422
  IdentityFile ~/.ssh/id_ed25519_unraid

Host unraid-ts 100.100.208.69   # Tailscale SSH: port 22, no key
  HostName 100.100.208.69
  User root
```

## Reboot persistence

Unraid runs `/` from `rootfs` (RAM) and `/run` as tmpfs, so all of this is lost
on reboot and is restored by `/boot/config/go`:

| Path | Lost on reboot | Restored by |
|------|----------------|-------------|
| `/root/.local/bin/herdr-core` | yes | `install -m 755` from `/boot/config/herdr/herdr-core` |
| `/usr/local/bin/hc`, `/usr/local/bin/herdr-core` | yes | `ln -sf` to the restored binary |
| `/var/lib/elogind/linger/root` | yes | `loginctl enable-linger root` |
| `/run/user/0` | yes | recreated by pam_elogind at first login, then held open by linger |

The `go` block:

```bash
# Herdr (hc) remote machines. Unraid runs / from RAM, so the hc binary installed
# by `herdr-core machine add --install` is lost on reboot. Restore it from /boot.
mkdir -p /root/.local/bin
install -m 755 /boot/config/herdr/herdr-core /root/.local/bin/herdr-core 2>/dev/null

# Put it on PATH under both names. ~/.local/bin is not on PATH in a
# non-interactive shell, and clients speak the protocol with `ssh box hc connect`,
# so without this they get "hc: command not found".
ln -sf /root/.local/bin/herdr-core /usr/local/bin/hc
ln -sf /root/.local/bin/herdr-core /usr/local/bin/herdr-core

# Keep /run/user/0 alive across logouts. pam_elogind creates and destroys it per
# login session, and the hc core's socket lives there, so without linger the
# socket disappears with the session that started the core and every later
# client fails with "core did not start within 5 s". The linger marker is
# /var/lib/elogind/linger/root, and /var/lib is RAM here, so it must be re-set
# at every boot.
loginctl enable-linger root 2>/dev/null
```

`install -m 755` matters: `/boot` is FAT32 and carries no execute bit.

Applied 2026-11-03, `sh -n /boot/config/go` parses cleanly. Backups of the go
script are at `/boot/config/go.bak-*`. The binary restore was verified without
rebooting, by deleting `/root/.local/bin/herdr-core` and running the line by
hand: it came back `-rwxr-xr-x` reporting `hc 0.1.17`.

### A boot-time restore is not enough on its own

On 2026-11-05 the `hc` binary disappeared with **no reboot**, at 25 days uptime,
so the `go` block above cannot be the only guard.
`/boot/config/herdr/herdr-selfheal.sh` restores the binary from the flash stash,
re-links both `/usr/local/bin` names and re-enables linger if it has lapsed. It
is idempotent and silent when nothing is wrong.

The script existed from 2026-11-05 but **was not scheduled anywhere** — it had
run exactly once, by hand. Scheduled 2026-11-07, every 10 minutes:

```
*/10 * * * * /bin/sh /boot/config/herdr/herdr-selfheal.sh
```

It belongs in the `printf ... | crontab -` line in `/boot/config/go`, because
that line **replaces the whole crontab** at boot: an entry added with
`crontab -e` would be wiped at the next restart. `/bin/sh` is required because
`/boot` is vfat and carries no execute bit. The live crontab and what the `go`
file will set at boot were diffed and are identical, so there is no drift.

Verified without a reboot: deleting `/usr/local/bin/hc` took `hc` off `PATH`,
the script relinked it, logged `relinked /usr/local/bin/hc`, and `hc --version`
reported `0.1.17` again. The core and its 8 live terminals were unaffected —
the running core is already a process and does not re-resolve the symlink.

## File drops (`hc put`) and the `shot` helper

`hc put <terminal> <file>` sends a **local** file to the terminal's machine and
prints where it landed. It takes `machine/t7`, so the same command works whether
the terminal is here or on the Mac — which makes it the transport for getting a
Mac screenshot into a session running on this box:

```
hc put unraid/t13 shot.png
-> {"bytes":67,"path":"/root/.local/state/herdr-core/default/files/t13/shot.png"}
```

| Property | Value |
|----------|-------|
| Destination | `/root/.local/state/herdr-core/<session>/files/<terminal>/<name>` |
| Survives a reboot | **no** — `/root/.local/state` is the RAM rootfs |
| Relocatable | no — 0.1.17 honours no `XDG_*` override for it |
| Extension | preserved, so `.png` stays `.png` |

Right for screenshots, wrong for anything large or worth keeping.

Inside a terminal, `$HC_TERMINAL_ID` is that terminal's own address and
`$HC_TERMINAL_SOCKET` points at `/run/user/0/herdr-core/default.sock`. So a
session can tell a client exactly where to push a file instead of the client
having to guess which terminal is in front of the user. `hc` also records the
client identity per terminal (`"identity":"ghostty"`).

`shot` reads those drops. Real copy at `/boot/config/shell/bin/shot`, installed
to `/root/.local/bin/shot` by `/boot/config/shell/install.sh` — a real
executable rather than a shell function, because a non-interactive shell (an
agent's tool call, an `ssh` command, cron) never sources `.bashrc` but does
inherit `PATH`.

| Command | Result |
|---------|--------|
| `shot` | the newest drop for this terminal, absolute path |
| `shot -l`, `shot -n 3` | drops newest-first |
| `shot -d` | the drop folder |
| `shot -a` | drops across every terminal |

With no drops it prints the `hc put` command that would create one.

### Reaching the Mac from here

Set up 2026-11-08. `ssh mac` works from this box, by key over Tailscale.

| Property | Value |
|----------|-------|
| Alias | `mac`, `macbook` in `/boot/config/ssh/root/config` |
| Address | `100.68.118.59` — the tailnet IP, not a LAN one, because it is a laptop |
| User | `kaloyandanchev` (uid 501, in `com.apple.access_ssh`) |
| Key | `/root/.ssh/id_ed25519`, `SHA256:DT7YBFfdI9+Ccb4Dfet+Q/xXkhqfmqeQi6ecRpSFQSY` |
| Auth | key only — macOS cannot be a Tailscale SSH server, so there is no keyless path |

The key had been offered and refused for a day. The cause was mundane: it had
never reached the Mac, because an earlier paste ran only part of its command.
The fix avoided the clipboard entirely by pulling the key over the link that
already worked in the other direction:

```bash
# on the Mac
K=$(ssh unraid 'cat /root/.ssh/id_ed25519.pub')
grep -qF "$K" ~/.ssh/authorized_keys || echo "$K" >> ~/.ssh/authorized_keys
```

`hc machine ls` on **this** core still returns `{"machines":[]}` and was left
that way: ssh covers what registering would have bought, and `hc file read` is
text-only so Mac images would still have to travel by `hc put`.

**What ssh to the Mac cannot do:** `screencapture` fails with `could not create
image from rect`. macOS withholds Screen Recording permission from a non-GUI
session. Screen capture therefore belongs to Hammerspoon, in the GUI session.

### Addressing terminals across both machines

`hc ls --json` **from the Mac** lists every live terminal on both machines in
one array — local ones as `t7`, this box's as `unraid/t13` — with `status`,
`agent`, `cwd` and a `where` label. The envelope is
`{"hidden_finished":N,"machines":[...],"terminals":[...]}`.

That is what makes a single Mac-side hotkey able to target any session
anywhere, without knowing in advance which machine is in front of the user.

### The screenshot hotkey (Mac side)

Appended to `~/.hammerspoon/init.lua` on 2026-11-08 (160 → 266 lines, backup
`init.lua.bak-20261108-093027`).

| Key | Action |
|-----|--------|
| `Cmd+Shift+U` | Clipboard image, or capture a region if there is none; pick a terminal; push and prompt |
| `Cmd+Ctrl+U` | The same, straight to the terminal chosen last time |

It runs `hc put --json <target> <file>`, parses the returned path, then
`hc send <target> "Look at the screenshot at <path>"` so the agent is told
without anything being typed.

Implementation notes that matter if it is ever edited:

- `hs.pasteboard.readImage()` reads the pasteboard natively — no `pngpaste`, no
  Homebrew, which keeps the "no second environment" constraint.
- Every `hc` call goes through `hs.task`, never `hs.execute`: a put to this box
  travels over ssh, and blocking Hammerspoon's main thread would freeze every
  other hotkey.
- Bindings avoid the `Cmd+Shift` letters the read-aloud hotkeys already use
  (Z, X, D, A, R).

Verified end to end on 2026-11-08: a 24×24 PNG made on this box, copied to the
Mac, pushed back with `hc put unraid/t13`, named by `shot` and read in-session
byte-intact.

## Notes

- The binary installs to `/root/.local/bin/herdr-core`, which is not on `PATH`
  in a non-interactive shell and is not named `hc`. The desktop app works
  regardless because it uses the absolute path it installed, but any other
  client fails. Symlinks in `/usr/local/bin` fix that (restored by the go
  script above).
- Adding a machine that `hc` already knows is refused as a duplicate — it keys
  on the core id (`3d4a2397fcb6…`), not the address. Re-pointing an existing
  machine means `machine rm` then `machine add`, since there is no command to
  change a target in place.

## Other clients (mobile, app builders)

`hc connect` speaks the protocol over stdin/stdout, so any client attaches with
`ssh <box> hc connect`. `hc schema` prints the API as JSON Schema. What a client
needs for this server:

| Field | Value |
|-------|-------|
| Host | `100.100.208.69`, or `xtrm-unraid.tailf789e1.ts.net` |
| Port | **22** — not 422; sshd binds only 192.168.10.20 and 192.168.31.250, so 422 is closed on the Tailscale address |
| User | `root` |
| Password / key | none — Tailscale SSH authenticates by tailnet identity |
| Command | `hc connect` |

The client device must itself be an active node in the tailnet
(`tailf789e1.ts.net`), because Tailscale SSH authorises by source node. The ACL
is `accept` for `autogroup:member` -> `autogroup:self`, so the owner's own
devices are covered once they are online and logged in.

Before the `/usr/local/bin` symlinks existed, this failed at the first hurdle:

```
ssh root@100.100.208.69 'hc connect'
-> /bin/bash: line 1: hc: command not found
```