From 1ddfc68192af3153a6027febd6b8a29c6c49577f Mon Sep 17 00:00:00 2001 From: XTRM Admin Date: Mon, 19 Jan 2026 21:28:11 +0200 Subject: [PATCH] Update Phase 8: Tasks 8.1-8.4 completed, document session_cookie_key fix --- docs/12-PHASE8-NETDISCO-INTEGRATION.md | 406 +++++++++---------------- 1 file changed, 152 insertions(+), 254 deletions(-) diff --git a/docs/12-PHASE8-NETDISCO-INTEGRATION.md b/docs/12-PHASE8-NETDISCO-INTEGRATION.md index b6b8fec..3a518fb 100644 --- a/docs/12-PHASE8-NETDISCO-INTEGRATION.md +++ b/docs/12-PHASE8-NETDISCO-INTEGRATION.md @@ -1,8 +1,9 @@ # Phase 8: NetDisco Integration with NetBox -**Status:** IN PROGRESS +**Status:** IN PROGRESS (Tasks 8.1-8.4 Complete) **Priority:** Medium **Created:** 2026-01-19 +**Last Updated:** 2026-01-19 --- @@ -14,99 +15,60 @@ Deploy NetDisco for enterprise-grade network discovery using SNMP, and integrate --- +## Task Status Summary + +| Task | Description | Status | +|------|-------------|--------| +| 8.1 | Enable SNMP on MikroTik Devices | ✅ COMPLETED | +| 8.2 | Deploy NetDisco on Unraid | ✅ COMPLETED | +| 8.3 | Configure NetDisco Discovery | ✅ COMPLETED | +| 8.4 | Traefik Ingress for NetDisco | ✅ COMPLETED | +| 8.5 | NetBox Integration | ⏳ PENDING | +| 8.6 | Testing & Validation | ⏳ PENDING | + +--- + ## Current State | Tool | Status | Purpose | |------|--------|---------| | NetAlertX | Running | ARP/ICMP-based device discovery | | NetBox | Running | IPAM/DCIM (manual data entry) | -| NetDisco | Not deployed | SNMP-based discovery | +| NetDisco | ✅ Running | SNMP-based discovery | **Decision:** Keep NetAlertX running in parallel during transition. --- -## Architecture +## Deployment Details -``` -┌─────────────────────────────────────────────────────────────┐ -│ SNMP-Enabled Devices │ -├─────────────────┬─────────────────┬─────────────────────────┤ -│ hAP ax³ │ CSS326 │ cAP ac │ -│ 192.168.31.1 │ 192.168.31.9 │ 192.168.31.6 │ -│ RouterOS 7.x │ SwOS │ RouterOS 7.x │ -│ SNMPv2c/v3 │ SNMPv1/v2c │ SNMPv2c/v3 │ -└────────┬────────┴────────┬────────┴────────┬────────────────┘ - │ │ │ - └─────────────────┼─────────────────┘ - │ SNMP Polling - ▼ - ┌─────────────────────┐ - │ NetDisco │ - │ 192.168.31.2:5000 │ - │ (Unraid Docker) │ - └──────────┬──────────┘ - │ API Sync - ▼ - ┌─────────────────────┐ - │ NetBox │ - │ 192.168.31.2:8090 │ - │ netbox-plugin │ - └─────────────────────┘ -``` +### Database +- **Host:** postgresql17 +- **Database:** netdisco_db +- **User:** netdisco_user +- **Password:** NetD1sc0_M0stW4nt3d@db + +### Containers +| Container | Image | Port | Status | +|-----------|-------|------|--------| +| netdisco-web | netdisco/netdisco:latest-web | 5000 | Running | +| netdisco-backend | netdisco/netdisco:latest-backend | - | Running | + +### Access URLs +| URL | Purpose | +|-----|---------| +| https://netdisco.xtrm-lab.org | External access (Authentik SSO) | +| http://192.168.31.2:5000 | Internal direct access | + +### Configuration +- **Config Path:** /mnt/user/appdata/netdisco/environments/deployment.yml +- **Logs:** /mnt/user/appdata/netdisco/logs/ --- -## Tasks +## Completed Tasks -### Task 8.1: Enable SNMP on MikroTik Devices - -#### 8.1.1 hAP ax³ (RouterOS) - -```routeros -# SSH to router -ssh -p 2222 xtrm@192.168.31.1 - -# Enable SNMP -/snmp set enabled=yes contact="admin@xtrm-lab.org" location="XTRM Home Lab" - -# Create community (restrict to Unraid only) -/snmp community add name=netdisco addresses=192.168.31.2/32 read-access=yes write-access=no - -# Verify -/snmp print -/snmp community print -``` - -**Status:** [x] COMPLETED (2026-01-19) - -#### 8.1.2 CSS326 (SwOS) - -1. Access SwOS web UI: http://192.168.31.9/index.html -2. Navigate to **SNMP** tab -3. Configure: - - Enable: Yes - - Community: netdisco - - IP Address Filter: 192.168.31.2 - -**Note:** SwOS only supports SNMPv1/v2c read-only. - -**Status:** [x] COMPLETED (2026-01-19) - -#### 8.1.3 cAP ac (RouterOS) - -```routeros -# SSH to cAP -ssh -p 2222 xtrm@192.168.31.6 - -# Enable SNMP (same config as hAP ax³) -/snmp set enabled=yes contact="admin@xtrm-lab.org" location="XTRM Home Lab" -/snmp community add name=netdisco addresses=192.168.31.2/32 read-access=yes write-access=no -``` - -**Status:** [x] COMPLETED (2026-01-19) - -#### SNMP Configuration Applied (2026-01-19) +### Task 8.1: SNMP Configuration (COMPLETED) | Device | IP | Communities | Access Restriction | |--------|-----|-------------|-------------------| @@ -120,242 +82,178 @@ ssh -p 2222 xtrm@192.168.31.6 - Version: SNMPv2c - Write Access: Disabled -**Test Results:** -``` -hAP ax³: SNMPv2-MIB::sysName.0 = STRING: HAPax3 -cAP ac: SNMPv2-MIB::sysName.0 = STRING: CAP XL ac -CSS326: SNMPv2-MIB::sysName.0 = STRING: CSS326-24G-2S+ -``` +### Task 8.2: NetDisco Deployment (COMPLETED) - ---- - -### Task 8.2: Deploy NetDisco on Unraid - -**Status:** [x] COMPLETED (2026-01-19) - -#### Docker Compose +**Docker Compose:** /mnt/user/appdata/netdisco/docker-compose.yml ```yaml -version: '3' services: - netdisco: - image: netdisco/netdisco:latest - container_name: netdisco + netdisco-backend: + container_name: netdisco-backend + image: netdisco/netdisco:latest-backend + hostname: netdisco-backend restart: unless-stopped - ports: - - "5000:5000" - environment: - - NETDISCO_DOMAIN=netdisco.xtrm-lab.org - - NETDISCO_DB_HOST=postgresql17 - - NETDISCO_DB_NAME=netdisco - - NETDISCO_DB_USER=netdisco - - NETDISCO_DB_PASS= - - NETDISCO_ADMIN_USER=admin - - NETDISCO_ADMIN_PASS= volumes: - - /mnt/user/appdata/netdisco/config:/home/netdisco/environments - - /mnt/user/appdata/netdisco/logs:/home/netdisco/logs - /mnt/user/appdata/netdisco/nd-site-local:/home/netdisco/nd-site-local + - /mnt/user/appdata/netdisco/environments:/home/netdisco/environments + - /mnt/user/appdata/netdisco/logs:/home/netdisco/logs + environment: + NETDISCO_DOMAIN: deployment + NETDISCO_DB_HOST: postgresql17 + NETDISCO_DB_PORT: 5432 + NETDISCO_DB_NAME: netdisco_db + NETDISCO_DB_USER: netdisco_user + NETDISCO_DB_PASS: NetD1sc0_M0stW4nt3d@db + networks: + - dockerproxy + + netdisco-web: + container_name: netdisco-web + image: netdisco/netdisco:latest-web + hostname: netdisco-web + restart: unless-stopped + volumes: + - /mnt/user/appdata/netdisco/nd-site-local:/home/netdisco/nd-site-local + - /mnt/user/appdata/netdisco/environments:/home/netdisco/environments + - /mnt/user/appdata/netdisco/logs:/home/netdisco/logs + environment: + NETDISCO_DOMAIN: deployment + NETDISCO_DB_HOST: postgresql17 + NETDISCO_DB_PORT: 5432 + NETDISCO_DB_NAME: netdisco_db + NETDISCO_DB_USER: netdisco_user + NETDISCO_DB_PASS: NetD1sc0_M0stW4nt3d@db + PORT: 5000 networks: - dockerproxy - depends_on: - - postgresql17 networks: dockerproxy: external: true ``` -#### Database Setup - -```bash -# Create NetDisco database in existing PostgreSQL -docker exec -it postgresql17 psql -U postgres -c "CREATE USER netdisco WITH PASSWORD 'secure_password';" -docker exec -it postgresql17 psql -U postgres -c "CREATE DATABASE netdisco OWNER netdisco;" -``` - -**Status:** [x] COMPLETED (2026-01-19) - ---- - -### Task 8.3: Configure NetDisco Discovery - -#### deployment.yml Configuration +### Task 8.3: Discovery Configuration (COMPLETED) +**deployment.yml:** ```yaml -# /mnt/user/appdata/netdisco/config/deployment.yml -snmp_community: - - community: netdisco - read: true - write: false +database: + name: 'netdisco_db' + user: 'netdisco_user' + pass: 'NetD1sc0_M0stW4nt3d@db' + host: 'postgresql17' -device_auth: - - tag: mikrotik - community: netdisco - snmp_version: 2 +community_rw: + - netdisco + - public discover_only: - - 192.168.31.1 # hAP ax³ - - 192.168.31.6 # cAP ac - - 192.168.31.9 # CSS326 + - 192.168.31.1 # hAP ax³ + - 192.168.31.6 # cAP ac + - 192.168.31.9 # CSS326 + +site_local_files: true +no_auth: true schedule: - discover: '0 * * * *' # Hourly discovery - macsuck: '15 * * * *' # MAC address table poll - arpnip: '30 * * * *' # ARP table poll - nbtstat: '45 * * * *' # NetBIOS name poll + discover: '0 */2 * * *' # Every 2 hours + macsuck: '15 * * * *' # MAC poll hourly + arpnip: '30 * * * *' # ARP poll hourly + nbtstat: '45 * * * *' # NetBIOS hourly ``` -**Status:** [x] COMPLETED (2026-01-19) +**Discovery Results:** +| Device | IP | Status | +|--------|-----|--------| +| hAP ax³ | 192.168.31.1 | ✅ Discovered | +| cAP ac | 192.168.31.6 | ✅ Discovered | +| CSS326 | 192.168.31.9 | ✅ Discovered | ---- +### Task 8.4: Traefik Ingress (COMPLETED) -### Task 8.4: Traefik Ingress for NetDisco - -Add to Traefik dynamic configuration: +**Traefik Dynamic Config:** /mnt/user/appdata/traefik/dynamic.yml ```yaml http: routers: - netdisco: - rule: "Host(`netdisco.xtrm-lab.org`)" + netdisco-secure: + rule: "Host(\`netdisco.xtrm-lab.org\`)" entryPoints: - - websecure - service: netdisco + - https + middlewares: + - default-headers + - authentik-forward-auth tls: certResolver: cloudflare + service: netdisco services: netdisco: loadBalancer: servers: - - url: "http://netdisco:5000" + - url: "http://netdisco-web:5000" ``` -**DNS:** Add netdisco.xtrm-lab.org to Cloudflare +**Important Fix Applied:** -**Status:** [x] COMPLETED (2026-01-19) +The web UI failed to start with error: `The setting session_cookie_key must be defined` + +**Root Cause:** When using external PostgreSQL, the `dancer_session_cookie_key` record was missing from the sessions table. + +**Fix:** +```sql +INSERT INTO sessions (id, a_session) VALUES ('dancer_session_cookie_key', md5(random()::text)); +``` --- +## Pending Tasks + ### Task 8.5: NetBox Integration -#### Option A: NetBox Plugin (Prototype) +**Option A: NetBox Plugin (Prototype)** +- Repository: https://github.com/mksoska/netbox-plugin-netdisco +- Status: Prototype stage - may have compatibility issues -Repository: https://github.com/mksoska/netbox-plugin-netdisco - -```bash -# Install in NetBox container -docker exec -it netbox pip install netbox-plugin-netdisco - -# Add to NetBox configuration.py -PLUGINS = ['netbox_netdisco'] -PLUGINS_CONFIG = { - 'netbox_netdisco': { - 'NETDISCO_HOST': 'http://netdisco:5000', - 'NETDISCO_USERNAME': 'admin', - 'NETDISCO_PASSWORD': '', - } -} -``` - -**Note:** Plugin is prototype-stage. Test thoroughly. - -#### Option B: Custom API Sync Script - -Create a sync script using NetDisco API + pynetbox: - -```python -#!/usr/bin/env python3 -# /mnt/user/appdata/netbox/scripts/netdisco_sync.py - -import requests -import pynetbox - -NETDISCO_URL = "http://192.168.31.2:5000" -NETBOX_URL = "http://192.168.31.2:8090" -NETBOX_TOKEN = "" - -nb = pynetbox.api(NETBOX_URL, token=NETBOX_TOKEN) - -# Fetch devices from NetDisco -devices = requests.get(f"{NETDISCO_URL}/api/v1/device").json() - -for device in devices: - # Check if device exists in NetBox - existing = nb.dcim.devices.get(primary_ip=device['ip']) - if not existing: - print(f"New device found: {device['ip']} ({device['name']})") - # Create or flag for review -``` - -**Status:** [x] COMPLETED (2026-01-19) - ---- +**Option B: Custom API Sync Script (Recommended)** +- Use NetDisco API + pynetbox library +- More control over sync behavior +- Can be scheduled via cron ### Task 8.6: Testing & Validation -#### Checklist +- [ ] Verify all SNMP queries working +- [ ] Confirm switch port mapping +- [ ] Test MAC address tracking +- [ ] Validate NetBox sync (once implemented) +- [ ] Test external access via Traefik -- [ ] SNMP reachable from Unraid to all MikroTik devices -- [ ] NetDisco discovers hAP ax³, CSS326, cAP ac -- [ ] Switch port mapping shows connected devices -- [ ] MAC address tracking working -- [ ] NetBox sync shows discovered devices -- [ ] Traefik proxy working (netdisco.xtrm-lab.org) +--- -#### SNMP Test Commands +## Troubleshooting -```bash -# Test SNMP from Unraid -docker run --rm --network dockerproxy netdisco/netdisco snmpwalk -v2c -c netdisco 192.168.31.1 system -docker run --rm --network dockerproxy netdisco/netdisco snmpwalk -v2c -c netdisco 192.168.31.9 system -docker run --rm --network dockerproxy netdisco/netdisco snmpwalk -v2c -c netdisco 192.168.31.6 system +### session_cookie_key Error +If web UI shows "session_cookie_key must be defined": +```sql +-- Check if key exists +SELECT * FROM sessions WHERE id = 'dancer_session_cookie_key'; + +-- Insert if missing +INSERT INTO sessions (id, a_session) VALUES ('dancer_session_cookie_key', md5(random()::text)); ``` -**Status:** [x] COMPLETED (2026-01-19) - ---- - -## Expected Outcomes - -| Metric | Before | After | -|--------|--------|-------| -| Device discovery | ARP scan only | SNMP + ARP | -| Switch port visibility | None | Full mapping | -| MAC tracking | Basic | Full history | -| NetBox data source | Manual | Auto-sync | -| L2 topology | Unknown | Mapped | - ---- - -## Resource Requirements - -| Resource | Estimate | -|----------|----------| -| Memory | ~200-400 MiB | -| CPU | Low (polling-based) | -| Disk | ~500 MiB (logs + DB) | -| Database | Reuse postgresql17 | - ---- - -## Rollback Plan - -If issues occur: -1. Stop NetDisco container -2. SNMP can remain enabled (read-only, no risk) -3. NetAlertX continues to function -4. Remove Traefik route +### Discovery Not Working +1. Test SNMP from netdisco container: + ```bash + docker exec netdisco-backend snmpwalk -v2c -c netdisco 192.168.31.1 system + ``` +2. Check logs: `tail -f /mnt/user/appdata/netdisco/logs/netdisco-daemon.log` +3. Verify community and IP restrictions on MikroTik devices --- ## References - [NetDisco Documentation](https://netdisco.org/) -- [NetDisco API Wiki](https://github.com/netdisco/netdisco/wiki/API) +- [NetDisco GitHub Issues](https://github.com/netdisco/netdisco-docker/issues) - [MikroTik SNMP Documentation](https://help.mikrotik.com/docs/spaces/ROS/pages/8978519/SNMP) -- [CSS326 SwOS SNMP](https://help.mikrotik.com/docs/spaces/SWOS/pages/76415036/CRS3xx+and+CSS326-24G-2S+series+Manual) - [NetBox Plugin for NetDisco](https://github.com/mksoska/netbox-plugin-netdisco) -- [pynetbox Library](https://github.com/netbox-community/pynetbox)