Major documentation restructure - consolidated docs

New Structure: - 01-NETWORK-MAP.md - Network topology, IPs, Docker networks, services - 02-SERVICES-CRITICAL.md - DNS, Auth, Routing (P0/P1 services) - 03-SERVICES-OTHER.md - All non-critical services - 04-HARDWARE-INVENTORY.md - Physical devices and specs - 05-CHANGELOG.md - Major events only New Folders: - docs/archive/ - Legacy docs (read-only reference) - docs/wip/ - Planned changes and ideas - UPGRADE-2026-HARDWARE.md - N5 Air + N100 migration plan - GITOPS-CONTAINERS.md - Phase 2 container GitOps Changes: - Moved all 22 legacy docs to archive/ - Consolidated container IPs, physical map, and services into single network map - Extracted critical vs non-critical service classification - Simplified changelog to major events only Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-25 11:17:33 +02:00
parent ed17dea2d6
commit b250493d5a
31 changed files with 1585 additions and 23 deletions
@@ -0,0 +1,345 @@
+## 2026-01-19 - NetDisco Web UI Fixed
+## 2026-01-19 - Phase 8 NetDisco Integration COMPLETED
+
+### All Tasks Completed
+- [8.1] SNMP enabled on hAP ax³, cAP ac, CSS326
+- [8.2] NetDisco deployed (backend + web containers)
+- [8.3] Discovery configured (3 devices discovered)
+- [8.4] Traefik ingress with Authentik SSO
+- [8.5] NetBox sync script created and tested
+- [8.6] All functionality verified
+
+### Key Fix
+- [FIX] session_cookie_key missing from database - manually inserted
+
+### Metrics
+- Devices discovered: 3
+- Ports collected: 52
+- MAC addresses tracked: 19
+
+### Access
+- External: https://netdisco.xtrm-lab.org (SSO)
+- Internal: http://192.168.31.2:5000
+
+---
+
+
+### Task 8.4: Traefik Ingress - VERIFIED WORKING
+
+**Root Cause:**
+- [BUG] session_cookie_key was missing from database
+- NetDisco generates this key via netdisco-deploy, but our external PostgreSQL setup skipped this step
+- Error: "The setting session_cookie_key must be defined"
+
+**Fix Applied:**
+- [DB] Manually inserted dancer_session_cookie_key into sessions table:
+  ```sql
+  INSERT INTO sessions (id, a_session) VALUES ('dancer_session_cookie_key', md5(random()::text));
+  ```
+
+**Verification:**
+- [TEST] http://netdisco-web:5000 - WORKING (returns HTML)
+- [TEST] https://netdisco.xtrm-lab.org - WORKING (302 redirect to Authentik)
+
+**Access:**
+- External URL: https://netdisco.xtrm-lab.org (SSO via Authentik)
+- Internal URL: http://192.168.31.2:5000 (direct)
+- Database: session_cookie_key stored in PostgreSQL sessions table
+
+---
+
+# Infrastructure Changelog
+## 2026-01-19 - NetDisco Traefik Integration
+
+### Task 8.4: Traefik Ingress - COMPLETED
+
+- [TRAEFIK] Added netdisco-secure router to dynamic.yml
+- [TRAEFIK] Service: http://netdisco-web:5000
+- [AUTH] Protected with authentik-forward-auth middleware
+- [DNS] netdisco.xtrm-lab.org (configured by user)
+- [URL] https://netdisco.xtrm-lab.org
+
+---
+
+## 2026-01-19 - NetDisco Deployment
+
+### Task 8.2: Deploy NetDisco - COMPLETED
+
+**Database Setup:**
+- [DB] Created netdisco_db database in postgresql17
+- [DB] Created netdisco_user with password
+- [DB] Database schema initialized (96 migrations applied)
+
+**Container Deployment:**
+- [SERVICE] netdisco-web: Web UI on port 5000
+- [SERVICE] netdisco-backend: SNMP polling daemon
+- [CONFIG] deployment.yml configured with SNMP communities
+- [CONFIG] Discover schedule: Every 2 hours
+- [CONFIG] MAC/ARP polling: Hourly
+
+**Device Discovery:**
+- [DISCOVER] 192.168.31.1 (hAP ax³) - Successful
+- [DISCOVER] 192.168.31.6 (cAP ac) - Successful
+- [DISCOVER] 192.168.31.9 (CSS326) - Successful
+- [DATA] MAC address tables collected
+- [DATA] ARP tables collected
+
+**Access:**
+- Web UI: http://192.168.31.2:5000
+- Initial auth: no_auth=true (unauthenticated admin)
+
+---
+
+## 2026-01-19 - SNMP Configuration for NetDisco
+
+### Task 8.1: Enable SNMP on MikroTik Devices - COMPLETED
+
+**hAP ax³ (192.168.31.1):**
+- [SNMP] Enabled SNMPv2c
+- [SNMP] Added `netdisco` community (192.168.31.2 only)
+- [SNMP] Restricted `public` community to 192.168.31.2
+- [SNMP] Contact: admin@xtrm-lab.org, Location: XTRM Home Lab
+
+**cAP ac (192.168.31.6):**
+- [SNMP] Enabled SNMPv2c via REST API
+- [SNMP] Added `netdisco` community (192.168.31.2 only)
+- [SNMP] Restricted `public` community to 192.168.31.2
+
+**CSS326 (192.168.31.9):**
+- [SNMP] Verified SNMP enabled (SwOS default)
+- [SNMP] Using `public` community (SwOS single-community limitation)
+- [NOTE] SwOS does not support address restrictions via API
+
+**Testing:**
+- [TEST] All three devices responding to SNMP queries from Unraid
+- [TEST] sysName.0 verified: HAPax3, CAP XL ac, CSS326-24G-2S+
+
+---
+
+## 2026-01-19 - NetDisco Integration Planning
+
+### Phase 8: NetDisco + NetBox Integration
+- [DOCS] Created 12-PHASE8-NETDISCO-INTEGRATION.md
+- [PLANNING] NetDisco deployment for SNMP-based network discovery
+- [PLANNING] MikroTik SNMP configuration (hAP ax³, CSS326, cAP ac)
+- [PLANNING] NetBox integration via plugin or custom API sync
+- [DECISION] Keep NetAlertX running in parallel during transition
+
+### MikroTik SNMP Support Confirmed
+- hAP ax³ (RouterOS 7.x): Full SNMPv1/v2c/v3 support
+- CSS326 (SwOS): SNMPv1/v2c read-only support
+- cAP ac (RouterOS 7.x): Full SNMPv1/v2c/v3 support
+
+### Resources
+- [NetDisco](https://netdisco.org/)
+- [NetBox Plugin for NetDisco](https://github.com/mksoska/netbox-plugin-netdisco)
+
+---
+
+
+## 2026-01-19 - NetBox IPAM/DCIM Deployment
+
+### NetBox Installation
+- [SERVICE] netbox: Deployed NetBox IPAM/DCIM on port 8090
+- [SERVICE] netbox-worker: Background task worker
+- [SERVICE] netbox-housekeeping: Database maintenance
+- [SERVICE] netbox-postgres: Dedicated PostgreSQL 16 database
+- [SERVICE] netbox-redis: Persistent Redis for queues
+- [SERVICE] netbox-redis-cache: Redis cache instance
+
+### Configuration
+- URL: https://netbox.xtrm-lab.org
+- Local: http://192.168.31.2:8090
+- Network: dockerproxy (for Traefik) + netbox (internal)
+- Data: /mnt/user/appdata/netbox/
+
+### DNS
+- [DNS] Added netbox.xtrm-lab.org to Cloudflare
+
+### Purpose
+- IP Address Management (IPAM)
+- Data Center Infrastructure Management (DCIM)
+- VLAN documentation and planning
+- Network device inventory
+- Cable/connection tracking
+
+---
+
+## 2026-01-18 - MikroTik WiFi & CAPsMAN Configuration
+
+### CAPsMAN Setup
+- [MIKROTIK] Configured CAPsMAN on hAP ax³ (192.168.31.1) as WiFi controller
+- [MIKROTIK] Added cAP ac (192.168.31.6) as managed access point
+- [MIKROTIK] Created provisioning rules for 2.4GHz and 5GHz bands
+- [MIKROTIK] cAP ac radios now managed by CAPsMAN (configuration.manager=capsman)
+
+### WiFi Security Updates
+- [WIFI] XTRM (2.4GHz): Changed from WPA2/WPA3 to WPA/WPA2 for legacy device support
+- [WIFI] XTRM2 (IoT): Changed from WPA2/WPA3 to WPA/WPA2 for legacy device support
+- [WIFI] XTRM (5GHz): Remains WPA2/WPA3
+- [WIFI] Fixed configuration band mismatch (cfg-XTRM5g had 2ghz-n, cfg-XTRM2g had 5ghz-ac)
+
+### SSH Key Configuration
+- [SSH] Added SSH key to cAP ac for xtrm user (key-based auth from Desktop)
+- [SSH] Documented SSH access for both MikroTik devices
+
+### Documentation
+- [DOCS] Created 09-MIKROTIK-WIFI-CAPSMAN.md with full WiFi/CAPsMAN configuration
+- [DOCS] Updated 00-CURRENT-STATE.md with cAP ac device info and WiFi networks
+- [DOCS] Added WiFi passwords and connection details to documentation
+
+### Issue Resolved
+- [FIX] iPad 2 connectivity issue - resolved by enabling WPA-PSK on 2.4GHz networks
+
+---
+
+## 2026-01-18 - Docker Organization & Container Fixes
+
+### FolderView2 Categories Reorganized
+- [UNRAID] Reorganized Docker containers into 10 categories
+- Categories: Infrastructure, Security, Monitoring, DevOps, Media, Storage/Backup, Productivity, Smart Home, Remote Access, Management
+- Icon collection: Dazzle Line Icons (https://www.svgrepo.com/collection/dazzle-line-icons/)
+- Deleted old categories: Cloud, Dashboards and Stats, Finance, Databases, Network, Network Monitoring, Automation, Backup
+
+### Container Icon Fixes
+- [SERVICE] rustfs: Fixed missing icon (was 404), now uses GitHub org avatar
+- [SERVICE] gitea: Fixed webui label (was template syntax, now https://git.xtrm-lab.org)
+- [SERVICE] woodpecker-server: Fixed icon URL (woodpecker.png → woodpecker-ci.png)
+- [SERVICE] woodpecker-agent: Fixed icon URL (woodpecker.png → woodpecker-ci.png)
+
+### Documentation
+- [DOCS] Added Unraid Docker Organization section to 00-CURRENT-STATE.md
+
+## 2026-01-18
+- [INFRA] Added pending task: Static IP assignment for critical services on dockerproxy and bridge networks
+- [SERVICE] postgresql17: Recreated container (was stopped due to port conflict)
+- [SERVICE] authentik + authentik-worker: Restarted after PostgreSQL fix
+- [TEMPLATE] Added RustDesk container templates with icons
+- [TEMPLATE] Updated Pi-hole template with proper Unraid CA metadata
+
+
+Track all changes to services, configurations, and phase progress.
+
+---
+
+## 2026-01-17 - Homarr + Portainer Integration
+
+### Portainer App Added to Homarr
+- [SERVICE] homarr: Added Portainer app to dashboard
+- Section: Monitoring
+- URL: http://100.100.208.70:9002 (Tailscale)
+- Ping URL: http://192.168.31.2:9002 (LAN)
+
+### Docker Integration Added
+- [SERVICE] homarr: Added Docker integration via socket
+- Integration name: Docker (Unraid)
+- Socket: unix:///var/run/docker.sock
+- Linked to Portainer app for container status display
+
+### Database Changes
+- Added app record for Portainer
+- Added item and item_layout for Monitoring section
+- Added integration record for Docker
+- Linked integration to Portainer item
+
+### Access
+- Homarr: https://xtrm-lab.org
+- Portainer visible in Monitoring section
+
+---
+
+## 2026-01-17 - Phase 6.2/6.3 Cancelled: MikroTik Incompatible
+
+### Discovery
+- MikroTik RouterOS containers are NOT Docker-based
+- No `/var/run/docker.sock` exists on MikroTik
+- Portainer cannot connect to MikroTik's container runtime
+
+### What Was Attempted
+- Created veth-socat interface (172.17.0.5)
+- Deployed alpine/socat container
+- Added firewall and NAT rules for port 2375
+- Socat failed: `No such file or directory` for docker.sock
+
+### Cleanup Performed
+- Removed socat container
+- Removed veth-socat interface and bridge port
+- Removed docker_sock mount
+- Removed firewall/NAT rules for port 2375
+
+### Conclusion
+- Phase 6.2 and 6.3 are NOT FEASIBLE
+- MikroTik containers must be managed via RouterOS CLI/WebFig
+- Portainer remains useful for Unraid-only container management
+
+### Status Update
+- [PHASE 6.1] COMPLETED - Portainer managing Unraid
+- [PHASE 6.2] CANCELLED - MikroTik incompatible
+- [PHASE 6.3] CANCELLED - MikroTik incompatible
+
+---
+
+## 2026-01-17 - Unraid Container Labels Fixed
+
+### Containers Updated
+- [SERVICE] unbound: Added Unraid labels (`net.unraid.docker.managed`, `net.unraid.docker.icon`)
+- [SERVICE] portainer: Added Unraid labels + Tailscale labels
+
+### Portainer Labels
+- `net.unraid.docker.managed=dockerman`
+- `net.unraid.docker.icon` - Portainer icon
+- `net.unraid.docker.webui=http://100.100.208.70:9002`
+- `tailscale.expose=true`
+- `tailscale.host=100.100.208.70`
+- `tailscale.port=9002`
+
+### Unbound Labels
+- `net.unraid.docker.managed=dockerman`
+- `net.unraid.docker.icon` - Unbound icon
+
+### Note
+Both containers recreated to apply labels. Services verified working after recreation.
+
+---
+
+## 2026-01-17 - Phase 6.1 Completed: Portainer CE Deployed
+
+### Portainer CE Installation
+- [PHASE 6.1] Portainer CE deployed on Unraid - COMPLETED
+- Container: `portainer/portainer-ce:latest`
+- HTTP Port: **9002** (changed from 9000 due to Authentik conflict)
+- HTTPS Port: **9444**
+- Data: `/mnt/user/appdata/portainer`
+- LAN URL: `http://192.168.31.2:9002`
+- Tailscale URL: `http://100.100.208.70:9002`
+
+### Port Conflict Resolution
+- Original plan: port 9000
+- Conflict: Authentik already using port 9000
+- Resolution: Mapped to port 9002 (HTTP) and 9444 (HTTPS)
+
+### Next Steps
+- Phase 6.2: Deploy Socat proxy on MikroTik (port 2375)
+- Phase 6.3: Connect MikroTik environment to Portainer
+
+### Status
+- [PHASE 6.1] COMPLETED - Portainer running, needs initial setup via web UI
+- [PHASE 6.2] NOT STARTED
+- [PHASE 6.3] NOT STARTED
+
+---
+
+## 2026-01-17 - Phase 6 Added: Multi-Host Docker Management
+
+### New Documentation
+- [PHASE 6] Created 06-PHASE6-PORTAINER-MANAGEMENT.md
+- Portainer CE deployment plan for unified Docker management
+- Covers Unraid local setup and MikroTik remote API via Socat
+
+### Phase 6 Components
+- Phase 6.1: Portainer CE installation on Unraid (port 9002)
+- Phase 6.2: MikroTik Socat proxy for Docker API exposure (port 2375)
+- Phase 6.3: Unified dashboard connection
+
+### Security Considerations
+-