infrastructure/docs/12-DEVELOPMENT-ENVIRONMENT.md

# Development Environment

**Last Updated:** 2026-03-08

Web-based development environment running directly on Unraid, providing VS Code IDE with full host access to Claude Code, Cooperator CLI, Docker, and all project repositories.

---

## OpenVSCode Server

| Property | Value |
|----------|-------|
| **URL** | https://code.xtrm-lab.org |
| **Auth** | Authentik forward auth (SSO) |
| **Port** | 3100 (host-native, not a container) |
| **Binary** | `/mnt/user/appdata/openvscode/current/` (symlink) |
| **Config** | `/mnt/user/appdata/openvscode/config/` |
| **Boot Script** | `/mnt/user/appdata/openvscode/start.sh` |
| **Log** | `/mnt/user/appdata/openvscode/server.log` |

**Why host-native?** Running directly on Unraid (not in a container) means the VS Code terminal has full access to `claude`, `cooperator`, `node`, `npm`, `docker`, `git`, and all host tools. No volume mount hacks or container-breaking updates.

### Persistence

All data lives on the array (`/mnt/user/`) — survives Unraid OS updates:

| Component | Path | Purpose |
|-----------|------|---------|
| Server binary | `/mnt/user/appdata/openvscode/openvscode-server-v1.109.5-linux-x64/` | VS Code server |
| Symlink | `/mnt/user/appdata/openvscode/current` → version dir | Easy version switching |
| VS Code config | `/mnt/user/appdata/openvscode/config/` | Extensions, settings, themes |
| Start script | `/mnt/user/appdata/openvscode/start.sh` | Startup with PATH setup |

### Updating OpenVSCode Server

```bash
# Download new version
cd /mnt/user/appdata/openvscode
curl -fsSL "https://github.com/gitpod-io/openvscode-server/releases/download/openvscode-server-vX.Y.Z/openvscode-server-vX.Y.Z-linux-x64.tar.gz" -o new.tar.gz
tar xzf new.tar.gz && rm new.tar.gz

# Switch symlink and restart
ln -sfn openvscode-server-vX.Y.Z-linux-x64 current
pkill -f "openvscode-server.*--port 3100"
/mnt/user/appdata/openvscode/start.sh
```

Extensions and settings are preserved (stored separately in `config/`).

### Traefik Routing

Defined in `/mnt/user/appdata/traefik/dynamic.yml`:

```yaml
openvscode-secure:
  rule: "Host(`code.xtrm-lab.org`)"
  entryPoints: [https]
  middlewares: [default-headers, authentik-forward-auth]
  tls:
    certResolver: cloudflare
  service: openvscode

# ...
openvscode:
  loadBalancer:
    servers:
      - url: "http://192.168.10.20:3100"
```

---

## Claude Code

| Property | Value |
|----------|-------|
| **Version** | 2.1.71 |
| **Binary** | `/mnt/user/appdata/claude-code/.npm-global/bin/claude` |
| **Symlink** | `/root/.local/bin/claude` |
| **Config** | `/mnt/user/appdata/claude-code/.claude.json` → `/root/.claude.json` |
| **Settings** | `/mnt/user/appdata/claude-code/.claude/` → `/root/.claude/` |
| **Boot Script** | `/mnt/user/appdata/claude-code/install-claude.sh` |

### Persistence

npm global prefix set to `/mnt/user/appdata/claude-code/.npm-global/` (array-backed). Boot script creates symlinks from `/root/` to persistent paths.

### Updating Claude Code

```bash
source /root/.bashrc
npm install -g @anthropic-ai/claude-code
claude --version
```

---

## Cooperator CLI

| Property | Value |
|----------|-------|
| **Version** | 3.36.1 |
| **Binary** | `/mnt/user/appdata/claude-code/.npm-global/bin/cooperator` |
| **Config** | `~/.cooperator/.env` (Shortcut token, Confluence, git config) |
| **Registry** | `@ampeco:registry=https://gitlab.com/api/v4/projects/71775017/packages/npm/` |
| **npm auth** | `/root/.npmrc` (GitLab PAT) |

### What Cooperator Install Sets Up

- **Commands** — `~/.claude/commands/cooperator` → cooperator's claude-commands
- **Agents** — `~/.claude/agents/implementation-task-executor.md`
- **Skills** — 12 cooperator skills (shortcut-operations, create-feature-story, gitlab-operations, etc.)
- **Shortcut API** — validated via `~/.cooperator/.env` token

### Updating Cooperator

```bash
source /root/.bashrc
npm install -g @ampeco/cooperator
cooperator --version
```

**Note:** `/root/.npmrc` is in RAM — recreated on boot if needed. The GitLab PAT is stored in `/boot/config/go` would need a persistent `.npmrc` setup if token changes frequently.

---

## GitLab CLI (glab)

| Property | Value |
|----------|-------|
| **Version** | 1.89.0 |
| **Binary** | `/usr/local/bin/glab` (RAM — lost on reboot) |
| **Config** | `~/.config/glab-cli/config.yml` |
| **Auth** | GitLab PAT (same as npm registry token) |

**Note:** glab binary at `/usr/local/bin/` is lost on Unraid reboot. Add to boot script or persist to appdata.

---

## Python (via uv)

| Property | Value |
|----------|-------|
| **uv** | `/root/.local/bin/uv` |
| **Python** | 3.12.13 (managed by uv) |
| **mikrotik-mcp venv** | `/mnt/user/projects/mikrotik-mcp/venv/` |
| **unraid-mcp venv** | `/mnt/user/projects/unraid-mcp/.venv/` |

---

## Custom Skills

6 custom skills synced from Mac to `/mnt/user/appdata/claude-code/custom-skills/`:

| Skill | Description |
|-------|-------------|
| ev-compliance-story | EV regulatory compliance story creation |
| ev-protocol-expert | OCPP/OCPI/AFIR protocol expertise |
| frontend-designer | Nova/Vue component design |
| mikrotik-admin | MikroTik router management via MCP |
| prd-generator | Product requirements documents |
| unraid-admin | Unraid server management via MCP |

Symlinked to `~/.claude/skills/` alongside 12 cooperator skills (18 total).

---

## MCP Servers

### Registered (TODO)

The following MCP servers need to be registered via `claude mcp add` on Unraid:

| Server | Command | Status |
|--------|---------|--------|
| **shortcut** | `node /mnt/user/appdata/claude-code/mcp-server-shortcut/dist/index.js` | Built, needs `claude mcp add` |
| **mikrotik** | `/mnt/user/projects/mikrotik-mcp/venv/bin/python -m mikrotik_mcp.server` | Venv ready, needs `claude mcp add` |
| **unraid** | `/mnt/user/projects/unraid-mcp/.venv/bin/python -m unraid_mcp.main` | Venv ready, needs `claude mcp add` |
| **playwright** | `npx -y @playwright/mcp@latest --isolated` | npx available, needs `claude mcp add` |
| **smartbear** | `npx -y @smartbear/mcp@latest` | npx available, needs `claude mcp add` |

### Environment Variables for MCPs

- **mikrotik**: `DEVICES_PATH=/mnt/user/projects/mikrotik-mcp/devices.json`
- **unraid**: `UNRAID_API_URL`, `UNRAID_API_KEY`, `UNRAID_MCP_TRANSPORT=stdio`, `UNRAID_VERIFY_SSL=false`
- **shortcut**: `SHORTCUT_API_TOKEN` (from `~/.cooperator/.env`)

---

## Projects Workspace

All projects at `/mnt/user/projects/`, opened as default folder in VS Code.

### Personal Projects (Gitea)

| Project | Gitea Repo | Description |
|---------|-----------|-------------|
| infrastructure | jazzymc/infrastructure | This repo — home lab documentation |
| claude-skills | jazzymc/claude-skills | Claude Code custom skills |
| mikrotik-mcp | jazzymc/mikrotik-mcp | MikroTik MCP server |
| unraid-mcp | jazzymc/unraid-mcp | Unraid MCP server |
| unraid-glass | jazzymc/unraid-glass | Unraid dashboard plugin |
| openclaw | jazzymc/openclaw | OpenClaw game project |
| nanobot-mcp | jazzymc/nanobot-mcp | Nanobot MCP server |
| nanobot-hkuds | jazzymc/nanobot-hkuds | Nanobot HKU DS |
| xtrm-agent | jazzymc/xtrm-agent | AI agent framework |
| geekmagic-smalltv | jazzymc/geekmagic-smalltv | SmallTV firmware |
| homarr | jazzymc/homarr | Homarr dashboard fork |
| shortcut-daily-sync | jazzymc/shortcut-daily-sync | Shortcut sync tool |

**Remote URL format:** `https://jazzymc:<token>@git.xtrm-lab.org/jazzymc/<repo>.git`

### AMPECO Work Projects

| Project | Source | Type |
|---------|--------|------|
| backend | GitLab (ampeco/apps/charge/backend) | Git clone |
| crm | GitLab (ampeco/apps/charge/crm) | Git clone |
| marketplace | GitLab (ampeco/apps/charge/marketplace) | Git clone |
| mobile-2 | GitLab (ampeco/apps/charge/mobile-2) | Git clone |
| ad-hoc-payment-web-app | GitLab (ampeco/apps/charge/external-apps/) | Git clone |
| dev-proxy | GitLab (ampeco/apps/shared/dev-proxy) | Git clone |
| ampeco-custom-dashboard-widgets-boilerplate | GitHub (ampeco/) | Git clone |
| docs | Local rsync | Reference docs |
| stories | Local rsync | Product stories |
| booking-ewa | Local rsync | Booking app |
| ewa-ui | Local rsync | EWA frontend |
| design-tokens | Local rsync | Design system tokens |
| ampeco-backup | Local rsync | Configuration backups |
| central_registry | Local rsync | Service registry |
| CCode-UI-Distribution-1.0.0 | Local rsync | UI distribution |
| automations | Local rsync | Automation scripts |

**GitLab auth:** OAuth2 PAT in remote URLs.

---

## Shell Environment

Root's login shell is **zsh 5.9**. bash stays as the fallback and as what scripts
use; both shells source the same environment and alias files, so they cannot drift.

| Property | Value |
|----------|-------|
| **Login shell** | `/bin/zsh`, set by `install.sh` — *not* by `/boot/config/passwd`, see Gotchas |
| **zsh package** | `/boot/extra/zsh-5.9-x86_64-1_slack15.0.txz` (Slackware 15.0 patches, md5 verified) |
| **Config source** | `/boot/config/shell/` on the flash drive |
| **Installer** | `bash /boot/config/shell/install.sh`, also the `reloadrc` alias |
| **Shell state** | `/mnt/user/appdata/shell/` — history, zsh completion dump, git credentials |
| **tmux** | `/mnt/user/appdata/tmux/tmux.conf` → `/root/.tmux.conf` (tmux 3.5a AppImage) |
| **SSH client** | `/boot/config/ssh/root/config` (`/root/.ssh` → `/boot/config/ssh/root`) |

### Files

| Installed | Source | Purpose |
|-----------|--------|---------|
| `/root/.zshrc` | `shell/zshrc` | Interactive zsh: prompt, completion, keys, history |
| `/root/.zshenv` | `shell/zshenv` | Environment for every zsh |
| `/root/.zprofile` | `shell/zprofile` | PATH for login zsh — see Gotchas |
| `/root/.bashrc` | `shell/bashrc` | Interactive bash |
| `/root/.bash_profile` | written by `install.sh` | Sources `.bashrc` |
| `/root/.gitconfig` | `shell/gitconfig` | Credential helper, `safe.directory` |
| — | `shell/env.sh` | PATH, `EDITOR`/`PAGER`, `HOST_LABEL`; sourced by both shells |
| — | `shell/aliases.sh` | Aliases and functions; sourced by both shells |

Prompt (both shells, same shape):

```
09:41 root@XTRM-U ~ampeco/backend (develop*)
❯
```

No framework, no plugin manager, no Nerd Font — `vcs_info` and zsh builtins only.
`*` is unstaged, `+` staged; the arrow turns red and the exit code is shown on
failure. Named directories shorten the share paths (`~projects`, `~ampeco`,
`~appdata`, `~vault`, `~flash`), and `cdpath` makes `cd backend` work from
anywhere, matching the `tmux-work` bookmarks.

### Commands in `shell/bin/`

`install.sh` installs everything in `/boot/config/shell/bin/` to
`/root/.local/bin` at mode 755. These are real executables, not shell
functions, because a non-interactive shell — an agent's tool call, `ssh box
cmd`, cron — never sources `.bashrc` but does inherit `PATH`.

| Command | Purpose |
|---------|---------|
| `shot` | Name the files `hc put` dropped into this Herdr terminal |
| `omac` | Act on an Unraid file as its Mac counterpart: open, path, `file://` URL, OSC 8 link, or verify it has synced |
| `clip` | Put text into the Mac's pasteboard from a session here, over OSC 52 |
| `macget` | Turn a Mac path (as a drag onto the terminal produces) into a file readable here |

`omac` and `macget` translate through the shared Syncthing folders, using one
table at `/boot/config/shell/syncmap` so the two directions cannot drift. The
Unraid paths in
Syncthing's `config.xml` are *container* paths and must be resolved through the
`syncthing` container's mounts (`/data1` → `/mnt/user/projects`):

| Folder id | Unraid | Mac |
|-----------|--------|-----|
| `jmmgo-qzpqp` (obsidian) | `/mnt/user/projects/obsidian` | `/Users/kaloyandanchev/Obsidian` |
| `homelab` | `/mnt/user/projects/homelab` | `/Users/kaloyandanchev/Homelab` |
| `ampeco` | `/mnt/user/projects/ampeco` | `/Users/kaloyandanchev/Ampeco` |
| `personal` | `/mnt/user/projects/personal` | `/Users/kaloyandanchev/Personal` |
| `claude-skills` | `/mnt/user/appdata/claude-code/.claude/skills` | `/Users/kaloyandanchev/.claude/skills` |
| `cooperator` | `/mnt/user/appdata/cooperator` | `/Users/kaloyandanchev/.cooperator` |

`/mnt/user/projects/infrastructure` is **not** synced, so this repo has no Mac
counterpart and cannot be click-opened. `omac` reports that rather than
returning a path that does not exist.

Full detail, including the Hammerspoon screenshot hotkey on the Mac, is in
`14-HERDR-REMOTE-MACHINES.md`.

### Persistence

`/` is a RAM rootfs, so every dotfile in `/root` is rebuilt at boot and anything
edited there is lost. The flash drive holds the real copies; `install.sh` lays
them down and is idempotent. It runs **twice**: from `/boot/config/go` before the
array mounts, so a shell is usable even if the array never comes up, and again
from `install-claude.sh` once `/mnt/user/appdata/shell/` is reachable. The zsh
binary itself comes back because `rc.local` installs `/boot/extra/*.txz` at boot.

Shell state goes on the NVMe cache pool, not the flash drive, which must not take
a write per command.

### Gotchas

1. **`/boot/config/passwd` cannot set root's shell.** `/etc/rc.d/rc.S.cont` reads
   that file but hardcodes the shell:
   `sed -i "s|^root:.*|root:x:0:0:$COMMENT:/root:/bin/bash|" /etc/passwd`.
   The shell field is discarded, so editing it does nothing. `install.sh` changes
   `/etc/passwd` afterwards instead, and only if `/bin/zsh` exists — if it is
   missing it installs it from `/boot/extra`, and failing that leaves root on bash
   and logs why, rather than locking root out of ssh and the web terminal. The
   Unraid web terminal reads root's shell from the passwd file too
   (`OpenTerminal.php`: `posix_getpwuid(0)['shell']`).
2. **`/etc/zprofile` clobbers PATH for login shells.** Slackware hardcodes
   `PATH="/usr/local/bin:/usr/bin:/bin"` there, and zsh reads it *after*
   `~/.zshenv`. A login shell — what ssh and the web terminal start — therefore
   lost the custom PATH, and `sc-story`, `vault`, `tok` and `claude` fell off it.
   Set in `~/.zprofile` (read after `/etc/zprofile`) and re-asserted at the top of
   `~/.zshrc`, the last startup file zsh reads.
3. **git needs `safe.directory` for the project shares.** `/mnt/user/projects` is
   owned by `jazzymc`, not root, so git returned *detected dubious ownership* and
   every branch lookup came back empty. Entries for `/mnt/user/projects/*` and
   `/mnt/user/appdata/*` are in the flash `gitconfig`; a trailing `/*` covers every
   repository underneath (git 2.56).

### Reverting to bash

```bash
sed -i 's|^\(root:.*\):/bin/zsh$|\1:/bin/bash|' /etc/passwd   # immediate
# then comment out the install.sh call in /boot/config/go to make it stick
```

---

## Boot Sequence

`/boot/config/go` triggers on Unraid boot:

1. **Shell config** — `bash /boot/config/shell/install.sh`
   - Installs root's dotfiles from `/boot/config/shell/` and sets `/bin/zsh` as the login shell
   - Deliberately before the array wait, so a usable shell exists even if the array never mounts
2. **Wait for array** — polls for `/mnt/user/appdata/claude-code` (up to 5 min)
3. **Claude Code setup** — `/mnt/user/appdata/claude-code/install-claude.sh`
   - Creates symlinks (`/root/.local/bin/claude`, `/root/.claude`, `/root/.claude.json`)
   - Re-runs `install.sh`, now that `/mnt/user/appdata/shell/` is reachable
4. **OpenVSCode Server** — `/mnt/user/appdata/openvscode/start.sh`
   - Kills any existing instance
   - Starts on port 3100 with persistent config dir
   - Sources Claude/Cooperator PATH for terminal sessions

---

## Architecture Diagram

```
Browser → https://code.xtrm-lab.org
         ↓
    Traefik (443) → Authentik SSO check
         ↓
    OpenVSCode Server (:3100, host-native)
         ↓
    Unraid Host Shell
    ├── claude (2.1.71)
    ├── cooperator (3.36.1)
    ├── glab (1.89.0)
    ├── node (22.18.0) / npm (10.9.3) / bun (1.3.10)
    ├── uv + python 3.12
    ├── docker / docker compose
    ├── git
    └── /mnt/user/projects/
        ├── ampeco/     (18 AMPECO work projects)
        ├── infrastructure/
        ├── claude-skills/
        ├── mikrotik-mcp/
        └── ... (12 personal repos)
```