diff --git a/docs/00-CURRENT-STATE.md b/docs/00-CURRENT-STATE.md index 18ca13f..5a3e9bb 100644 --- a/docs/00-CURRENT-STATE.md +++ b/docs/00-CURRENT-STATE.md @@ -11,13 +11,12 @@ graph TB subgraph Internet WAN["WAN: 62.73.120.142"] - DNS_EXT["dns.xtrm-lab.org
DoH/DoT/DoQ"] end subgraph MikroTik["MikroTik hAP ax³ (192.168.31.1)"] ROUTER["RouterOS 7.20.6"] subgraph MK_Containers["Docker Containers"] - AGH_MK["AdGuard Home
172.17.0.5:5355
PRIMARY DNS"] + AGH_MK["AdGuard Home
172.17.0.5:5355
dns.xtrm-lab.org"] TS["Tailscale
172.17.0.4"] end end @@ -31,6 +30,10 @@ graph TB end subgraph Unraid["Unraid Server (192.168.31.2)"] + subgraph SharedServices["Shared Services"] + POSTGRES["PostgreSQL 17
172.18.0.13"] + REDIS["Redis
172.18.0.14"] + end subgraph Core["Core Services"] TRAEFIK["Traefik
172.18.0.3"] HOMARR["Homarr
172.18.0.4"] @@ -40,171 +43,80 @@ graph TB VAULT["Vaultwarden
172.18.0.15"] end subgraph DNS_Unraid["DNS Services"] - AGH_UR["AdGuard Home
192.168.31.4:53
SECONDARY DNS"] - UNBOUND["Unbound
192.168.31.5"] + AGH_UR["AdGuard Home
192.168.31.4
dns2.xtrm-lab.org"] end subgraph DevOps["DevOps"] GITEA["Gitea
172.18.0.31"] WOODPECKER["Woodpecker CI
172.18.0.32"] end - subgraph Monitoring["Monitoring"] - UPTIME["Uptime Kuma
172.18.0.20"] + subgraph NetBoxStack["Network Inventory"] NETBOX["NetBox
172.18.0.61"] - DIODE["NetBox Discovery
172.24.0.10"] + DIODE["Diode Stack
172.18.0.70-74"] + NETDISCO["NetDisco
172.18.0.41-42"] end - subgraph Media["Media"] - PLEX["Plex"] - NEXTCLOUD["Nextcloud
172.18.0.24"] - end - end - - subgraph LAN["LAN Devices (192.168.31.x)"] - CLIENTS["Clients"] end WAN --> ROUTER - DNS_EXT --> ROUTER ROUTER --> AGH_MK - ROUTER --> TS ROUTER --> SW SW --> Unraid SW --> AP - AP --> CLIENTS - SW --> CLIENTS - AGH_MK -.->|"Upstream DoH"| QUAD9["Quad9 DNS"] - AGH_UR -.->|"Upstream DoH"| QUAD9 - CLIENTS -->|"DNS Queries"| AGH_MK - CLIENTS -.->|"Failover"| AGH_UR + AGH_MK -.->|sync| AGH_UR ``` --- -## MikroTik hAP ax³ Router (192.168.31.1) +## Service Architecture Diagram -| Parameter | Value | -|-----------|-------| -| RouterOS Version | 7.20.6 (stable) | -| WAN IP (Static) | 62.73.120.142 | -| LAN Subnet | 192.168.31.0/24 | -| Docker Bridge | 172.17.0.0/24 | -| SSH Access | Port 2222, user: jazzymc | +```mermaid +flowchart TB + subgraph SharedServices["Shared Infrastructure"] + PG[("PostgreSQL 17
172.18.0.13")] + RD[("Redis
172.18.0.14")] + end -**Interfaces:** -- `ether1` - WAN (62.73.120.142/23) -- `bridge` - LAN (192.168.31.1/24) -- `docker-bridge` - Container network (172.17.0.1/24) -- `back-to-home-vpn` - WireGuard VPN (192.168.216.1/24) + subgraph NetBoxStack["Network Inventory Stack"] + NB["NetBox
172.18.0.61"] + NBW["NetBox Worker
172.18.0.62"] + NBC[("Redis Cache
172.18.0.64")] + + subgraph Diode["Diode Discovery"] + DI["Ingress
172.18.0.70"] + DIN["Ingester
172.18.0.71"] + DRE["Reconciler
172.18.0.72"] + DHY["Hydra
172.18.0.73"] + DAU["Auth
172.18.0.74"] + DAG["Agent
host network"] + end + + subgraph NetDisco["NetDisco"] + NDW["Web
172.18.0.41"] + NDB["Backend
172.18.0.42"] + end + end -### Running Containers on MikroTik + subgraph DevOps["DevOps Stack"] + GIT["Gitea
172.18.0.31"] + WPS["Woodpecker Server
172.18.0.32"] + WPA["Woodpecker Agent
172.18.0.33"] + end -| Container | IP | Storage | Purpose | -|-----------|-----|---------|---------| -| tailscale | 172.17.0.4 | usb1/tailscale/root | Tailscale VPN client | -| adguardhome | 172.17.0.5 | disk1/agh-root + usb1 mount | DNS with DoH/DoT/DoQ | - -### AdGuard Home (MikroTik) - PRIMARY DNS - -| Service | Port | Protocol | Status | -|---------|------|----------|--------| -| DNS | 5355 (NAT from 53) | UDP/TCP | Active | -| Web UI | 80 | HTTP | Active | -| DoH | 443 | HTTPS | Active | -| DoT | 853 | TCP | Active | -| DoQ | 8853 | UDP | Active | - -**Configuration:** -- Upstream: Quad9 DoH (https://dns10.quad9.net/dns-query) -- TLS Certificate: Let's Encrypt wildcard (*.xtrm-lab.org) -- Server Name: dns.xtrm-lab.org -- Certificate Expiry: 2026-04-02 -- Credentials: jazzymc / 7RqWElENNbZnPW - -**Persistence:** root-dir on disk1 + data mount on usb1 (survives container restart) - ---- - -## MikroTik CSS326-24G-2S+ Switch (192.168.31.9) - -| Parameter | Value | -|-----------|-------| -| Role | Managed Layer 2 Switch | -| Ports | 24x Gigabit + 2x SFP | -| OS | SwOS | -| Web UI | https://sw.xtrm-lab.org | - ---- - -## MikroTik cAP ac (192.168.31.6) - -| Parameter | Value | -|-----------|-------| -| Role | CAPsMAN Managed Access Point | -| RouterOS Version | 7.20.1 (stable) | -| Identity | CAP XL ac | - ---- - -## Unraid Server (192.168.31.2) - -**Tailscale IP:** 100.100.208.70 -**SSH Access:** `ssh -i ~/.ssh/id_ed25519_unraid root@192.168.31.2 -p 422` - -### Docker Networks - -| Network | Subnet | Purpose | -|---------|--------|---------| -| br0 | 192.168.31.0/24 | LAN macvlan (AdGuard Home) | -| dockerproxy | 172.18.0.0/16 | Traefik-accessible services | -| diode_default | 172.24.0.0/16 | NetBox Discovery (Diode) | -| bridge | 172.17.0.0/16 | Default Docker bridge | - -### Key Services - -| Service | Container | IP | External URL | -|---------|-----------|---|--------------| -| **Core** |||| -| Reverse Proxy | traefik | 172.18.0.3 | traefik.xtrm-lab.org | -| Dashboard | homarr | 172.18.0.4 | xtrm-lab.org | -| **Security** |||| -| Identity Provider | authentik | 172.18.0.11 | auth.xtrm-lab.org | -| Password Manager | vaultwarden | 172.18.0.15 | vault.xtrm-lab.org | -| **DNS** |||| -| AdGuard Home | adguardhome | 192.168.31.4 | - | -| Unbound | unbound | 192.168.31.5 | - | -| **DevOps** |||| -| Git Server | gitea | 172.18.0.31 | git.xtrm-lab.org | -| CI/CD Server | woodpecker-server | 172.18.0.32 | ci.xtrm-lab.org | -| **Monitoring** |||| -| Uptime Kuma | UptimeKuma | 172.18.0.20 | uptime.xtrm-lab.org | -| NetBox | netbox | 172.18.0.61 | netbox.xtrm-lab.org | -| **Media** |||| -| Plex | plex | host | plex.xtrm-lab.org | -| Nextcloud | Nextcloud | 172.18.0.24 | nextcloud.xtrm-lab.org | -| **Remote Access** |||| -| RustDesk | rustdesk-hbbs/hbbr | bridge | rustdesk.xtrm-lab.org | - -### AdGuard Home (Unraid) - SECONDARY DNS - -| Setting | Value | -|---------|-------| -| IP Address | 192.168.31.4 | -| Network | br0 (macvlan) | -| Web UI | http://192.168.31.4:3000 | -| DNS | 192.168.31.4:53 | -| DoT | 192.168.31.4:853 | -| Credentials | jazzymc / 7RqWElENNbZnPW | - -**Configuration (synced with MikroTik):** -- Upstream: Quad9 DoH -- TLS Certificate: Let's Encrypt wildcard -- 6 Clients configured -- Custom filtering rules (SentinelOne, Jamf) - -**Data Location:** /mnt/user/appdata/adguardhome/ - -**Stopped Services:** -- binhex-official-pihole (replaced by AdGuard Home) -- nebula-sync (incompatible with AdGuard Home) + PG --> NB + PG --> GIT + PG --> NDW + PG --> DRE + PG --> DHY + RD --> DIN + RD --> DRE + RD --> NBW + NBC --> NB + + DAG -->|gRPC| DI + DI --> DIN + DIN --> RD + DRE --> NB + NDB --> NDW +``` --- @@ -213,184 +125,167 @@ graph TB ```mermaid flowchart TB subgraph External["External Access"] - DOH["DoH: https://dns.xtrm-lab.org/dns-query"] - DOT["DoT: tls://dns.xtrm-lab.org:853"] - DOQ["DoQ: quic://dns.xtrm-lab.org:8853"] + DOH1["DoH: dns.xtrm-lab.org"] + DOT1["DoT: dns.xtrm-lab.org:853"] + DOH2["DoH: dns2.xtrm-lab.org"] + DOT2["DoT: dns2.xtrm-lab.org:853"] end subgraph MikroTik["MikroTik Router"] NAT["NAT: 53 → 5355"] - AGH1["AdGuard Home
172.17.0.5:5355
PRIMARY"] + AGH1["AdGuard Home
PRIMARY"] end subgraph Unraid["Unraid Server"] - AGH2["AdGuard Home
192.168.31.4:53
SECONDARY"] + AGH2["AdGuard Home
SECONDARY"] + end + + subgraph Sync["Configuration Sync"] + AGHSYNC["adguardhome-sync
Every 30 min"] end subgraph Upstream["Upstream DNS"] - Q9["Quad9 DoH
dns10.quad9.net"] + Q9["Quad9 DoH"] end - subgraph Clients["LAN Clients"] - C1["IPhone Dancho"] - C2["IPhone Kimi"] - C3["Laptop Dari"] - C4["Laptop Kimi"] - C5["PC Dancho"] - C6["ROG Ally Teodor"] - end - - External --> MikroTik - Clients -->|"Primary"| NAT + DOH1 --> AGH1 + DOT1 --> AGH1 + DOH2 --> AGH2 + DOT2 --> AGH2 NAT --> AGH1 - Clients -.->|"Failover"| AGH2 AGH1 --> Q9 AGH2 --> Q9 + AGH1 <-.->|sync| AGHSYNC + AGHSYNC <-.->|sync| AGH2 ``` --- -## Configured Clients (Both AdGuard Instances) +## Container Summary -| Client | MAC Address | Tags | -|--------|-------------|------| -| IPhone (Dancho) | f2:b8:14:61:c8:27 | - | -| IPhone (Kimi) | 2a:2b:ba:86:d4:af | user_child | -| Laptop (Dari) | 34:f6:4b:b3:14:83 | user_child | -| Laptop (Kimi) | 90:91:64:70:0d:86 | user_child | -| PC (Dancho) | 70:85:c2:75:64:e5 | - | -| ROG Ally (Teodor) | cc:5e:f8:d3:37:d3 | user_child | +### Shared Services ---- +| Container | IP | Purpose | Consumers | +|-----------|-----|---------|-----------| +| postgresql17 | 172.18.0.13 | PostgreSQL 17 | NetBox, Gitea, NetDisco, Authentik, Diode | +| Redis | 172.18.0.14 | Redis Queue | Diode, NetBox Worker | -## Custom Filtering Rules - -``` -||dv-eu-prod.sentinelone.net^ -||euce1-soc360.sentinelone.net^ -||ampeco.jamfcloud.com^ -||*.jamfcloud.com^ -``` - ---- - -## NAT/Port Forwarding (MikroTik) - -| Rule | Protocol | Port | Destination | Purpose | -|------|----------|------|-------------|---------| -| HTTP | TCP | 80 | 192.168.31.2:8001 | Traefik | -| HTTPS | TCP | 443 | 192.168.31.2:44301 | Traefik | -| DNS UDP | UDP | 53→5355 | 172.17.0.5 | AdGuard Home | -| DNS TCP | TCP | 53→5355 | 172.17.0.5 | AdGuard Home | -| DoT | TCP | 853 | 172.17.0.5 | DNS over TLS | -| DoQ | UDP | 8853 | 172.17.0.5 | DNS over QUIC | -| Plex | TCP | 32400 | 192.168.31.2 | Plex Media | -| RustDesk | TCP/UDP | 21115-21119 | 192.168.31.2 | RustDesk | - ---- - -## Reference Documents - -- [Phase 1: DNS Portability](./01-PHASE1-DNS-PORTABILITY.md) -- [Phase 7: Gitea GitOps](./08-PHASE7-GITEA-GITOPS.md) -- [Changelog](./06-CHANGELOG.md) - ---- - -## Network Discovery & Management - -### NetBox (IPAM/DCIM) +### Network Inventory (NetBox & Discovery) | Container | IP | Purpose | |-----------|-----|---------| -| netbox | 172.18.0.61 | Web UI (netbox.xtrm-lab.org) | -| netbox-postgres | - | Database | -| netbox-redis | - | Cache | -| netbox-redis-cache | - | Redis cache | -| netbox-worker | - | Background tasks | - -**Plugins Installed:** -- netboxlabs-diode-netbox-plugin (NetBox Discovery integration) - -### NetBox Discovery (Diode) - -NetBox Labs Diode provides automated network discovery and data ingestion into NetBox. - -| Container | IP | Purpose | -|-----------|-----|---------| -| diode-ingress-nginx-1 | 172.24.0.10 | API Gateway | -| diode-diode-auth-1 | - | OAuth2 authentication | -| diode-diode-ingester-1 | - | Data ingestion service | -| diode-diode-reconciler-1 | - | Data reconciliation | -| diode-hydra-1 | - | OAuth2 provider (Ory Hydra) | -| diode-postgres-1 | - | Database | -| diode-redis-1 | - | Cache | -| diode-discovery-agent | host network | Network scanner (orb-agent) | - -**Data Location:** /mnt/user/appdata/diode/ - -**Discovery Agent Configuration:** -- Schedule: Every 30 minutes -- Target: 192.168.31.0/24 -- Ports scanned: 22, 80, 161, 443 -- Site: Home - -**OAuth2 Credentials:** -- diode-ingest: For data ingestion -- netbox-to-diode: For NetBox plugin -- diode-to-netbox: For reconciler - -### NetDisco - -NetDisco provides SNMP-based network discovery and ARP table collection. - -| Container | IP | Purpose | -|-----------|-----|---------| -| netdisco-web | 172.18.0.41 | Web UI (netdisco.xtrm-lab.org) | +| netbox | 172.18.0.61 | Web UI | +| netbox-worker | 172.18.0.62 | Background tasks | +| netbox-redis-cache | 172.18.0.64 | Query cache | +| diode-ingress | 172.18.0.70 | API Gateway (nginx) | +| diode-ingester | 172.18.0.71 | Data ingestion | +| diode-reconciler | 172.18.0.72 | NetBox sync | +| diode-hydra | 172.18.0.73 | OAuth2 (Ory Hydra) | +| diode-auth | 172.18.0.74 | Token service | +| diode-agent | host | Network scanner | +| netdisco-web | 172.18.0.41 | Web UI | | netdisco-backend | 172.18.0.42 | SNMP poller | -**Database:** postgresql17 (shared) -- Database: netdisco_db -- User: netdisco_user +### Infrastructure -**Discovered Data:** -- 4 SNMP-enabled devices -- 42 ARP entries (all network hosts) +| Container | IP | Purpose | +|-----------|-----|---------| +| traefik | 172.18.0.3 | Reverse proxy | +| dockersocket | - | Docker socket proxy | +| adguardhome | 192.168.31.4 | DNS (Secondary) | +| adguardhome-sync | 172.18.0.65 | Config sync | -### NetDisco to NetBox Sync +### DevOps -A scheduled sync script pushes NetDisco data to NetBox via Diode. +| Container | IP | Purpose | +|-----------|-----|---------| +| gitea | 172.18.0.31 | Git hosting | +| woodpecker-server | 172.18.0.32 | CI/CD server | +| woodpecker-agent | 172.18.0.33 | CI/CD agent | -**Location:** /mnt/user/appdata/netdisco-netbox-sync/ +### Security -| File | Purpose | -|------|---------| -| sync.py | Python sync script | -| Dockerfile | Container build file | -| docker-compose.yml | Deployment config | +| Container | IP | Purpose | +|-----------|-----|---------| +| authentik | 172.18.0.11 | Identity provider | +| authentik-worker | - | Background tasks | +| vaultwarden | 172.18.0.15 | Password manager | -**Sync Configuration:** -- Source: NetDisco PostgreSQL database -- Target: NetBox via Diode gRPC API -- Data synced: Devices (with vendor, model, OS) and IP addresses (with MAC) +### Monitoring -**Run manually:** -```bash -cd /mnt/user/appdata/netdisco-netbox-sync -docker compose run --rm netdisco-netbox-sync -``` +| Container | IP | Purpose | +|-----------|-----|---------| +| UptimeKuma | 172.18.0.20 | Uptime monitoring | +| Uptime-Kuma-API | 172.18.0.18 | REST API | +| AutoKuma | 172.18.0.19 | Auto-monitor creation | +| NetAlertX | - | Network alerting | +| speedtest-tracker | - | Speed tests | --- -## Agent Service Account +## RAM Usage (as of 2026-01-23) -A dedicated service account `agent` was created for automated tools: +**Total: 15GB | Used: 12GB (80%) | Available: 2.7GB** -| Device | Username | Auth Method | Port | -|--------|----------|-------------|------| -| Unraid | agent | SSH Key + Password | 422 | -| MikroTik Router | agent | SSH Key | 2222 | -| MikroTik AP | agent | Password | 2222 | -| MikroTik Switch | N/A | No SSH (SwOS) | - | +| Container | RAM | % | +|-----------|-----|---| +| unimus | 1.62 GB | 10.5% | +| karakeep | 664 MB | 4.2% | +| netdisco-web | 534 MB | 3.4% | +| n8n | 293 MB | 1.9% | +| netdisco-backend | 281 MB | 1.8% | +| netbox-worker | 230 MB | 1.5% | +| plex | 161 MB | 1.0% | +| postgresql17 | 136 MB | 0.9% | +| All others | <130 MB each | <1% | -**Credentials:** See docs/AGENT-CREDENTIALS.md (gitignored, local only) +--- + +## Removed Services (2026-01-23) + +The following services were removed as redundant (AdGuard Home provides DoH/DoT natively): + +| Service | Reason | +|---------|--------| +| Unbound | AdGuard uses upstream DoH directly | +| DoH-Server | AdGuard has built-in DoH | +| stunnel-dot | AdGuard has built-in DoT | +| Pangolin | Not in use | + +--- + +## External URLs + +| Service | URL | +|---------|-----| +| Dashboard | https://xtrm-lab.org | +| Traefik | https://traefik.xtrm-lab.org | +| Authentik | https://auth.xtrm-lab.org | +| Gitea | https://git.xtrm-lab.org | +| Woodpecker CI | https://ci.xtrm-lab.org | +| NetBox | https://netbox.xtrm-lab.org | +| NetDisco | https://netdisco.xtrm-lab.org | +| Uptime Kuma | https://uptime.xtrm-lab.org | +| Plex | https://plex.xtrm-lab.org | +| Nextcloud | https://cloud.xtrm-lab.org | +| Vaultwarden | https://vault.xtrm-lab.org | +| DNS (Primary) | dns.xtrm-lab.org (MikroTik) | +| DNS (Secondary) | dns2.xtrm-lab.org (Unraid) | + +--- + +## FolderView2 Categories + +| Category | Containers | +|----------|------------| +| Infrastructure | traefik, dockersocket, adguardhome, adguardhome-sync | +| Security | authentik, authentik-worker, vaultwarden | +| Monitoring | UptimeKuma, Uptime-Kuma-API, AutoKuma, NetAlertX, speedtest-tracker | +| DevOps | gitea, woodpecker-server, woodpecker-agent, postgresql17, Redis | +| Media | plex, Libation, transmission | +| Storage/Backup | rustfs, UrBackup, TimeMachine, Nextcloud | +| Productivity | actual-budget, n8n, karakeep, homarr | +| Smart Home | HomeAssistant_inabox | +| Remote Access | rustdesk-hbbs, rustdesk-hbbr | +| Management | portainer, unimus | +| Network Inventory | netbox, netbox-worker, netbox-redis-cache, diode-*, netdisco-* | diff --git a/docs/06-CHANGELOG.md b/docs/06-CHANGELOG.md index 76113fd..afe841b 100644 --- a/docs/06-CHANGELOG.md +++ b/docs/06-CHANGELOG.md @@ -251,3 +251,31 @@ See git history for earlier changes. - netbox-redis (was 172.18.0.63) --- + +## 2026-01-23 - Service Cleanup & Documentation Update + +### Services Removed +- [REMOVED] Unbound - redundant (AdGuard has upstream DoH) +- [REMOVED] DoH-Server - redundant (AdGuard has built-in DoH) +- [REMOVED] stunnel-dot - redundant (AdGuard has built-in DoT) +- [REMOVED] Pangolin - not in use + +### DNS Configuration +- [CONFIG] Unraid AdGuard: dns2.xtrm-lab.org (was dns.xtrm-lab.org) +- [CONFIG] MikroTik AdGuard: dns.xtrm-lab.org (primary) + +### Container Management +- [LABELS] Added net.unraid.docker.managed to all containers +- [LABELS] Added WebUI URLs to containers with web interfaces +- [LABELS] Updated icons to PNG format (from SVG) + +### FolderView2 +- [CATEGORY] Added "Network Inventory" for NetBox/Diode/NetDisco + +### Documentation +- [DOCS] Updated 00-CURRENT-STATE.md with current architecture +- [DOCS] Added Mermaid diagrams for network topology +- [DOCS] Added RAM usage statistics +- [DOCS] Documented removed services + +---