Files
infrastructure/docs/02-PHASE2-FOSSORIAL-STACK.md
jazzymc 62a6267026
All checks were successful
ci/woodpecker/push/woodpecker Pipeline was successful
Add infrastructure documentation
2026-01-18 16:57:25 +02:00

15 KiB

Phase 2: The "Fossorial" Tunnel Stack (Pangolin, Gerbil, Newt)

Goal

Deploy the Fossorial tunnel stack using your MikroTik's static IP (62.73.120.142) to host tunnels without requiring a VPS, with MikroTik container fallback for resilience.


What is Fossorial?

Fossorial is a self-hosted tunnel solution consisting of:

Component Purpose Role
Pangolin Central controller/dashboard Manages tunnels, provides web UI
Gerbil WireGuard manager Handles WireGuard peer configuration
Newt Tunnel connector Lightweight agent that "dials out" to establish tunnels

Why Fossorial over plain WireGuard?

  • Automatic peer management
  • Web-based tunnel configuration
  • Self-healing connections via Newt
  • Easier certificate/identity management

Current WireGuard State (MikroTik)

Interface: back-to-home-vpn
├── Listen Port: 59188 (non-standard - good!)
├── Address: 192.168.216.1/24
├── Public Key: 3e+p++SJ6f5EURt6WCKApOLMQHWpURm/vn/0s9+EKzs=
└── Peers: 3 configured

Port 51820 Status: NOT in use - available for Fossorial


Architecture Overview

                              Internet
                                 │
                    ┌────────────▼────────────┐
                    │  MikroTik (62.73.120.142)│
                    │  Port Forward:          │
                    │  UDP 51820 → Unraid     │
                    │  TCP 443 → Traefik      │
                    └────────────┬────────────┘
                                 │
              ┌──────────────────┼──────────────────┐
              │                  │                  │
              ▼                  ▼                  ▼
    ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
    │ Pangolin        │ │ Gerbil          │ │ Newt            │
    │ (Controller)    │ │ (WG Manager)    │ │ (Connector)     │
    │ :3000 web UI    │ │ :51820 WG       │ │ Outbound only   │
    │                 │ │ :8080 API       │ │                 │
    └────────┬────────┘ └────────┬────────┘ └────────┬────────┘
             │                   │                   │
             └───────────────────┴───────────────────┘
                        Internal Network
                        192.168.31.0/24

Implementation Steps

Step 2.1: Create Docker Network for Fossorial

docker network create --driver bridge fossorial

Step 2.2: Deploy Pangolin (Controller)

Unraid Docker Template:

<?xml version="1.0"?>
<Container version="2">
  <Name>pangolin</Name>
  <Repository>fossoriumtech/pangolin:latest</Repository>
  <Registry>https://hub.docker.com/r/fossoriumtech/pangolin</Registry>
  <Network>fossorial</Network>
  <Shell>sh</Shell>
  <Privileged>false</Privileged>
  <Overview>Pangolin - Fossorial tunnel controller and dashboard</Overview>
  <Category>Network:VPN</Category>
  <WebUI>https://pangolin.xtrm-lab.org</WebUI>
  <ExtraParams>--restart unless-stopped</ExtraParams>

  <!-- Ports -->
  <Config Name="Web UI" Target="3000" Default="3000" Mode="tcp" Type="Port" Display="always" Required="true">3000</Config>

  <!-- Volumes -->
  <Config Name="Data" Target="/app/data" Default="/mnt/user/appdata/pangolin/data" Mode="rw" Type="Path" Display="always" Required="true">/mnt/user/appdata/pangolin/data</Config>
  <Config Name="Config" Target="/app/config" Default="/mnt/user/appdata/pangolin/config" Mode="rw" Type="Path" Display="always" Required="true">/mnt/user/appdata/pangolin/config</Config>

  <!-- Environment -->
  <Config Name="BASE_URL" Target="PANGOLIN_BASE_URL" Default="https://pangolin.xtrm-lab.org" Mode="" Type="Variable" Display="always" Required="true">https://pangolin.xtrm-lab.org</Config>
  <Config Name="SECRET_KEY" Target="PANGOLIN_SECRET_KEY" Default="" Mode="" Type="Variable" Display="always" Required="true" Mask="true">GENERATE_A_SECURE_32_CHAR_KEY</Config>

  <!-- Traefik Labels -->
  <Config Name="traefik.enable" Target="traefik.enable" Type="Label" Display="always">true</Config>
  <Config Name="traefik.http.routers.pangolin.rule" Target="traefik.http.routers.pangolin.rule" Type="Label" Display="always">Host(`pangolin.xtrm-lab.org`)</Config>
  <Config Name="traefik.http.routers.pangolin.entrypoints" Target="traefik.http.routers.pangolin.entrypoints" Type="Label" Display="always">https</Config>
  <Config Name="traefik.http.routers.pangolin.tls.certresolver" Target="traefik.http.routers.pangolin.tls.certresolver" Type="Label" Display="always">cloudflare</Config>
  <Config Name="traefik.http.routers.pangolin.middlewares" Target="traefik.http.routers.pangolin.middlewares" Type="Label" Display="always">default-headers@file</Config>
  <Config Name="traefik.http.services.pangolin.loadbalancer.server.port" Target="traefik.http.services.pangolin.loadbalancer.server.port" Type="Label" Display="always">3000</Config>
  <Config Name="traefik.docker.network" Target="traefik.docker.network" Type="Label" Display="always">dockerproxy</Config>

  <!-- AutoKuma -->
  <Config Name="kuma" Target="kuma" Type="Label" Display="advanced">https://pangolin.xtrm-lab.org</Config>
  <Config Name="kuma.name" Target="kuma.name" Type="Label" Display="advanced">Pangolin Dashboard</Config>
  <Config Name="kuma.type" Target="kuma.type" Type="Label" Display="advanced">http</Config>

  <!-- Tailscale (optional) -->
  <Config Name="TailScale Fallback State Directory" Target="CA_TS_FALLBACK_DIR" Type="Variable" Display="advanced">/app/data</Config>
</Container>

Generate Secret Key:

openssl rand -hex 32

Step 2.3: Deploy Gerbil (WireGuard Manager)

Unraid Docker Template:

<?xml version="1.0"?>
<Container version="2">
  <Name>gerbil</Name>
  <Repository>fossoriumtech/gerbil:latest</Repository>
  <Registry>https://hub.docker.com/r/fossoriumtech/gerbil</Registry>
  <Network>fossorial</Network>
  <Shell>sh</Shell>
  <Privileged>true</Privileged>
  <Overview>Gerbil - Fossorial WireGuard manager</Overview>
  <Category>Network:VPN</Category>
  <ExtraParams>--cap-add=NET_ADMIN --cap-add=SYS_MODULE --sysctl net.ipv4.ip_forward=1 --sysctl net.ipv4.conf.all.src_valid_mark=1</ExtraParams>

  <!-- Ports -->
  <Config Name="WireGuard UDP" Target="51820" Default="51820" Mode="udp" Type="Port" Display="always" Required="true">51820</Config>
  <Config Name="API" Target="8080" Default="8080" Mode="tcp" Type="Port" Display="always" Required="true">8080</Config>

  <!-- Volumes -->
  <Config Name="WireGuard Config" Target="/etc/wireguard" Default="/mnt/user/appdata/gerbil/wireguard" Mode="rw" Type="Path" Display="always" Required="true">/mnt/user/appdata/gerbil/wireguard</Config>
  <Config Name="Data" Target="/app/data" Default="/mnt/user/appdata/gerbil/data" Mode="rw" Type="Path" Display="always" Required="true">/mnt/user/appdata/gerbil/data</Config>

  <!-- Environment -->
  <Config Name="PANGOLIN_URL" Target="GERBIL_PANGOLIN_URL" Default="http://pangolin:3000" Mode="" Type="Variable" Display="always" Required="true">http://pangolin:3000</Config>
  <Config Name="PUBLIC_IP" Target="GERBIL_PUBLIC_IP" Default="" Mode="" Type="Variable" Display="always" Required="true">62.73.120.142</Config>
  <Config Name="PUBLIC_PORT" Target="GERBIL_PUBLIC_PORT" Default="51820" Mode="" Type="Variable" Display="always" Required="true">51820</Config>
  <Config Name="WG_INTERFACE" Target="GERBIL_WG_INTERFACE" Default="wg0" Mode="" Type="Variable" Display="always" Required="true">wg0</Config>
  <Config Name="API_KEY" Target="GERBIL_API_KEY" Default="" Mode="" Type="Variable" Display="always" Required="true" Mask="true">SAME_AS_PANGOLIN_SECRET</Config>

  <!-- AutoKuma -->
  <Config Name="kuma" Target="kuma" Type="Label" Display="advanced">http://192.168.31.2:8080/health</Config>
  <Config Name="kuma.name" Target="kuma.name" Type="Label" Display="advanced">Gerbil WireGuard</Config>
  <Config Name="kuma.type" Target="kuma.type" Type="Label" Display="advanced">http</Config>
</Container>

Step 2.4: Deploy Newt (Connector)

Unraid Docker Template:

<?xml version="1.0"?>
<Container version="2">
  <Name>newt</Name>
  <Repository>fossoriumtech/newt:latest</Repository>
  <Registry>https://hub.docker.com/r/fossoriumtech/newt</Registry>
  <Network>fossorial</Network>
  <Shell>sh</Shell>
  <Privileged>false</Privileged>
  <Overview>Newt - Fossorial tunnel connector (dials out to establish tunnels)</Overview>
  <Category>Network:VPN</Category>
  <ExtraParams>--restart unless-stopped</ExtraParams>

  <!-- Volumes -->
  <Config Name="Data" Target="/app/data" Default="/mnt/user/appdata/newt/data" Mode="rw" Type="Path" Display="always" Required="true">/mnt/user/appdata/newt/data</Config>

  <!-- Environment -->
  <Config Name="PANGOLIN_URL" Target="NEWT_PANGOLIN_URL" Default="" Mode="" Type="Variable" Display="always" Required="true">https://pangolin.xtrm-lab.org</Config>
  <Config Name="ENDPOINT" Target="NEWT_ENDPOINT" Default="" Mode="" Type="Variable" Display="always" Required="true">62.73.120.142:51820</Config>
  <Config Name="API_KEY" Target="NEWT_API_KEY" Default="" Mode="" Type="Variable" Display="always" Required="true" Mask="true">GENERATE_VIA_PANGOLIN_UI</Config>
  <Config Name="TUNNEL_NAME" Target="NEWT_TUNNEL_NAME" Default="unraid-local" Mode="" Type="Variable" Display="always" Required="true">unraid-local</Config>

  <!-- AutoKuma -->
  <Config Name="kuma" Target="kuma" Type="Label" Display="advanced">docker</Config>
  <Config Name="kuma.name" Target="kuma.name" Type="Label" Display="advanced">Newt Connector</Config>
  <Config Name="kuma.type" Target="kuma.type" Type="Label" Display="advanced">docker</Config>
</Container>

Step 2.5: MikroTik Port Forward for WireGuard

Add NAT rule for Fossorial WireGuard:

# Connect via SSH
ssh -i /root/.ssh/mikrotik_key -p 2222 unraid@192.168.31.1

# Add port forward
/ip/firewall/nat add chain=dstnat \
    action=dst-nat \
    to-addresses=192.168.31.2 \
    to-ports=51820 \
    protocol=udp \
    dst-address=62.73.120.142 \
    dst-port=51820 \
    comment="Fossorial WireGuard"

# Add firewall rule to allow
/ip/firewall/filter add chain=forward \
    action=accept \
    protocol=udp \
    dst-address=192.168.31.2 \
    dst-port=51820 \
    comment="Allow Fossorial WireGuard" \
    place-before=14

Verification:

/ip/firewall/nat print where comment~"Fossorial"

Step 2.6: Connect Networks (fossorial ↔ dockerproxy)

Pangolin needs to be accessible via Traefik. Either:

Option A: Connect Pangolin to both networks

docker network connect dockerproxy pangolin

Option B: Use Traefik external routing in dynamic.yml

# Add to /mnt/user/appdata/traefik/dynamic.yml
http:
  routers:
    pangolin-secure:
      rule: "Host(`pangolin.xtrm-lab.org`)"
      entryPoints:
        - https
      middlewares:
        - default-headers
      tls:
        certResolver: cloudflare
      service: pangolin

  services:
    pangolin:
      loadBalancer:
        servers:
          - url: "http://192.168.31.2:3000"

Step 2.7: MikroTik Container Fallback (Optional)

Deploy a lightweight Gerbil instance on MikroTik for resilience:

Prerequisites:

  • USB storage connected to MikroTik (already present: usb1)
  • Container mode enabled

MikroTik Commands:

# Create container for Gerbil fallback
/container/config set registry-url=https://registry-1.docker.io tmpdir=usb1/tmp

# Pull gerbil image
/container add \
    remote-image=fossoriumtech/gerbil:latest \
    interface=docker-bridge \
    root-dir=usb1/gerbil \
    start-on-boot=yes \
    comment="Fossorial Gerbil Fallback"

# Configure environment
/container/envs add name=gerbil-env key=GERBIL_PUBLIC_IP value="62.73.120.142"
/container/envs add name=gerbil-env key=GERBIL_PUBLIC_PORT value="51821"
/container/envs add name=gerbil-env key=GERBIL_MODE value="standalone"

Note: MikroTik containers have limited resources. This is a fallback for critical services only (Pi-hole access, Authentik).


Service Interruption Assessment

Action Risk Impact Mitigation
Deploy Pangolin/Gerbil/Newt NONE New containers -
Port forward 51820 LOW New port, existing WG on 59188 unaffected -
Connect fossorial network LOW Container networking Test connectivity
MikroTik container MEDIUM Router resources Monitor CPU/memory

Existing WireGuard (back-to-home-vpn) Impact: NONE

  • Uses port 59188, not 51820
  • Completely separate interface

Verification Checklist

  • All three containers running: docker ps | grep -E "pangolin|gerbil|newt"
  • Pangolin web UI accessible: https://pangolin.xtrm-lab.org
  • Gerbil API responding: curl http://192.168.31.2:8080/health
  • MikroTik NAT rule in place: ssh ... "/ip/firewall/nat print"
  • External WireGuard test: Connect from external network to 62.73.120.142:51820
  • Newt connected in Pangolin dashboard

Initial Pangolin Setup

  1. Navigate to https://pangolin.xtrm-lab.org
  2. Create admin account
  3. Add Gerbil node:
    • Name: unraid-gerbil
    • API URL: http://gerbil:8080
    • API Key: (same as GERBIL_API_KEY)
  4. Create a tunnel:
    • Name: home-services
    • Assign to Gerbil node
  5. Generate Newt API key in Pangolin UI
  6. Update Newt container with the API key

Rollback Procedure

  1. Stop containers:

    docker stop newt gerbil pangolin
    docker rm newt gerbil pangolin
    
  2. Remove MikroTik NAT:

    /ip/firewall/nat remove [find comment="Fossorial WireGuard"]
    
  3. Remove network:

    docker network rm fossorial
    
  4. Clean up data (if desired):

    rm -rf /mnt/user/appdata/pangolin /mnt/user/appdata/gerbil /mnt/user/appdata/newt
    

Files Modified

File/System Change Backup Required
MikroTik NAT Add UDP 51820 forward N/A (can remove)
/mnt/user/appdata/traefik/dynamic.yml Add pangolin route YES
New directories created /mnt/user/appdata/pangolin,gerbil,newt N/A

Dependencies for Next Phase

Phase 3 (Authentik) can now use Fossorial tunnels to:

  • Expose Authentik externally without Cloudflare dependency
  • Create secure tunnels for mobile OIDC authentication