From 0e35142e100ee5cd0fb05a26f27301a8957468e5 Mon Sep 17 00:00:00 2001 From: XTRM Admin Date: Mon, 19 Jan 2026 23:43:55 +0200 Subject: [PATCH] Phase 8: Enhanced network mapping with sync scripts - Added MikroTik DHCP to NetBox sync script - Installed Slurp'it plugin v1.2.7 in NetBox - Enhanced NetDisco to NetBox sync with ARP/MAC data - Enabled SNMP on Unraid server (4 devices now discoverable) - Updated Phase 8 documentation with all scripts and configs Co-Authored-By: Claude Opus 4.5 --- docs/06-CHANGELOG.md | 23 ++ docs/12-PHASE8-NETDISCO-INTEGRATION.md | 363 +++++++++---------------- 2 files changed, 146 insertions(+), 240 deletions(-) create mode 100644 docs/06-CHANGELOG.md diff --git a/docs/06-CHANGELOG.md b/docs/06-CHANGELOG.md new file mode 100644 index 0000000..a74d1cf --- /dev/null +++ b/docs/06-CHANGELOG.md @@ -0,0 +1,23 @@ + +## 2026-01-19 - Phase 8 Enhanced Network Mapping + +### MikroTik DHCP Sync +- [SCRIPT] Created mikrotik_dhcp_to_netbox.sh +- [SYNC] 29 DHCP leases synced to NetBox IPs +- [DATA] Hostname, MAC, comments captured + +### Slurp'it Plugin Installation +- [PLUGIN] Installed slurpit_netbox v1.2.7 +- [BUILD] Created netbox-custom:latest image +- [CONFIG] Plugin configuration at /mnt/user/appdata/netbox/config/plugins.py + +### Enhanced NetDisco Sync +- [SCRIPT] Updated sync_to_netbox.py with additional data +- [SYNC] Device info, IPs, MACs, ARP table entries +- [DATA] 4 devices synced with full metadata + +### Unraid SNMP +- [SERVICE] kubedzero/unraid-snmp plugin installed +- [CONFIG] Communities: netdisco, public +- [DISCOVER] XTRM-Unraid now visible in NetDisco + diff --git a/docs/12-PHASE8-NETDISCO-INTEGRATION.md b/docs/12-PHASE8-NETDISCO-INTEGRATION.md index 3f8a062..ecc2bbf 100644 --- a/docs/12-PHASE8-NETDISCO-INTEGRATION.md +++ b/docs/12-PHASE8-NETDISCO-INTEGRATION.md @@ -1,6 +1,6 @@ # Phase 8: NetDisco Integration with NetBox -**Status:** ✅ COMPLETED +**Status:** ✅ COMPLETED (Enhanced) **Priority:** Medium **Created:** 2026-01-19 **Last Updated:** 2026-01-19 @@ -25,6 +25,7 @@ Deploy NetDisco for enterprise-grade network discovery using SNMP, and integrate | 8.4 | Traefik Ingress for NetDisco | ✅ COMPLETED | | 8.5 | NetBox Integration | ✅ COMPLETED | | 8.6 | Testing & Validation | ✅ COMPLETED | +| 8.7 | Enhanced Network Mapping | ✅ COMPLETED | --- @@ -33,10 +34,10 @@ Deploy NetDisco for enterprise-grade network discovery using SNMP, and integrate | Tool | Status | Purpose | |------|--------|---------| | NetAlertX | Running | ARP/ICMP-based device discovery | -| NetBox | Running | IPAM/DCIM (manual data entry) | +| NetBox | Running | IPAM/DCIM with Slurpit plugin | | NetDisco | ✅ Running | SNMP-based discovery | -**Decision:** Keep NetAlertX running in parallel during transition. +**Discovered Devices:** 4 (hAP ax³, cAP ac, CSS326, Unraid) --- @@ -53,280 +54,164 @@ Deploy NetDisco for enterprise-grade network discovery using SNMP, and integrate |-----------|-------|------|--------| | netdisco-web | netdisco/netdisco:latest-web | 5000 | Running | | netdisco-backend | netdisco/netdisco:latest-backend | - | Running | +| netbox | netbox-custom:latest (with slurpit) | 8090 | Running | ### Access URLs | URL | Purpose | |-----|---------| | https://netdisco.xtrm-lab.org | External access (Authentik SSO) | -| http://192.168.31.2:5000 | Internal direct access | - -### Configuration -- **Config Path:** /mnt/user/appdata/netdisco/environments/deployment.yml -- **Logs:** /mnt/user/appdata/netdisco/logs/ +| http://192.168.31.2:5000 | Internal NetDisco access | +| https://netbox.xtrm-lab.org | External NetBox access | +| http://192.168.31.2:8090 | Internal NetBox access | --- -## Completed Tasks +## Task 8.7: Enhanced Network Mapping (COMPLETED) -### Task 8.1: SNMP Configuration (COMPLETED) +### 1. MikroTik DHCP → NetBox Sync -| Device | IP | Communities | Access Restriction | -|--------|-----|-------------|-------------------| -| hAP ax³ | 192.168.31.1 | `netdisco`, `public` | 192.168.31.2/32 | -| cAP ac | 192.168.31.6 | `netdisco`, `public` | 192.168.31.2/32 | -| CSS326 | 192.168.31.9 | `public` | None (SwOS limitation) | +**Script:** `/mnt/user/appdata/netdisco/scripts/mikrotik_dhcp_to_netbox.sh` -**SNMP Settings:** -- Contact: admin@xtrm-lab.org -- Location: XTRM Home Lab -- Version: SNMPv2c -- Write Access: Disabled +Syncs DHCP leases from MikroTik router to NetBox IP addresses with: +- Hostname +- MAC address +- Comments/notes from DHCP -### Task 8.2: NetDisco Deployment (COMPLETED) - -**Docker Compose:** /mnt/user/appdata/netdisco/docker-compose.yml - -```yaml -services: - netdisco-backend: - container_name: netdisco-backend - image: netdisco/netdisco:latest-backend - hostname: netdisco-backend - restart: unless-stopped - volumes: - - /mnt/user/appdata/netdisco/nd-site-local:/home/netdisco/nd-site-local - - /mnt/user/appdata/netdisco/environments:/home/netdisco/environments - - /mnt/user/appdata/netdisco/logs:/home/netdisco/logs - environment: - NETDISCO_DOMAIN: deployment - NETDISCO_DB_HOST: postgresql17 - NETDISCO_DB_PORT: 5432 - NETDISCO_DB_NAME: netdisco_db - NETDISCO_DB_USER: netdisco_user - NETDISCO_DB_PASS: NetD1sc0_M0stW4nt3d@db - networks: - - dockerproxy - - netdisco-web: - container_name: netdisco-web - image: netdisco/netdisco:latest-web - hostname: netdisco-web - restart: unless-stopped - volumes: - - /mnt/user/appdata/netdisco/nd-site-local:/home/netdisco/nd-site-local - - /mnt/user/appdata/netdisco/environments:/home/netdisco/environments - - /mnt/user/appdata/netdisco/logs:/home/netdisco/logs - environment: - NETDISCO_DOMAIN: deployment - NETDISCO_DB_HOST: postgresql17 - NETDISCO_DB_PORT: 5432 - NETDISCO_DB_NAME: netdisco_db - NETDISCO_DB_USER: netdisco_user - NETDISCO_DB_PASS: NetD1sc0_M0stW4nt3d@db - PORT: 5000 - networks: - - dockerproxy - -networks: - dockerproxy: - external: true +**Run Command:** +```bash +/mnt/user/appdata/netdisco/scripts/mikrotik_dhcp_to_netbox.sh ``` -### Task 8.3: Discovery Configuration (COMPLETED) +**Result:** 29 devices synced (26 created, 3 updated) -**deployment.yml:** -```yaml -database: - name: 'netdisco_db' - user: 'netdisco_user' - pass: 'NetD1sc0_M0stW4nt3d@db' - host: 'postgresql17' +### 2. Slurp'it NetBox Plugin -community_rw: - - netdisco - - public +**Plugin Version:** 1.2.7 +**NetBox Version:** 4.5.0 -discover_only: - - 192.168.31.1 # hAP ax³ - - 192.168.31.6 # cAP ac - - 192.168.31.9 # CSS326 +**Custom Image:** `netbox-custom:latest` -site_local_files: true -no_auth: true - -schedule: - discover: '0 */2 * * *' # Every 2 hours - macsuck: '15 * * * *' # MAC poll hourly - arpnip: '30 * * * *' # ARP poll hourly - nbtstat: '45 * * * *' # NetBIOS hourly +**Plugin Configuration:** `/mnt/user/appdata/netbox/config/plugins.py` +```python +PLUGINS = ["slurpit_netbox"] +PLUGINS_CONFIG = { + "slurpit_netbox": {} +} ``` -**Discovery Results:** -| Device | IP | Status | -|--------|-----|--------| -| hAP ax³ | 192.168.31.1 | ✅ Discovered | -| cAP ac | 192.168.31.6 | ✅ Discovered | -| CSS326 | 192.168.31.9 | ✅ Discovered | +**Note:** Slurp'it requires a separate Slurp'it server for full network discovery. The plugin provides NetBox integration once the server is configured. -### Task 8.4: Traefik Ingress (COMPLETED) +### 3. Enhanced NetDisco → NetBox Sync -**Traefik Dynamic Config:** /mnt/user/appdata/traefik/dynamic.yml +**Script:** `/mnt/user/appdata/netdisco/scripts/sync_to_netbox.py` -```yaml -http: - routers: - netdisco-secure: - rule: "Host(\`netdisco.xtrm-lab.org\`)" - entryPoints: - - https - middlewares: - - default-headers - - authentik-forward-auth - tls: - certResolver: cloudflare - service: netdisco +Enhanced version syncs: +- Device info (serial, OS, model, vendor) +- IP addresses from device_ip table +- ARP table entries (MAC to IP mappings) +- System MAC addresses +- Uptime information - services: - netdisco: - loadBalancer: - servers: - - url: "http://netdisco-web:5000" -``` +**Run Command:** +```bash +PG17_IP=$(docker inspect postgresql17 --format "{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}") -**Important Fix Applied:** - -The web UI failed to start with error: `The setting session_cookie_key must be defined` - -**Root Cause:** When using external PostgreSQL, the `dancer_session_cookie_key` record was missing from the sessions table. - -**Fix:** -```sql -INSERT INTO sessions (id, a_session) VALUES ('dancer_session_cookie_key', md5(random()::text)); +docker run --rm \ + --network dockerproxy \ + -e NETDISCO_DB_HOST=$PG17_IP \ + -e NETBOX_URL=http://192.168.31.2:8090 \ + -v /mnt/user/appdata/netdisco/scripts:/scripts \ + python:3.11-slim python /scripts/sync_to_netbox.py ``` --- -## Pending Tasks +## SNMP-Enabled Devices -### Task 8.5: NetBox Integration +| Device | IP | Type | Communities | +|--------|-----|------|-------------| +| hAP ax³ | 192.168.31.1 | Router | netdisco, public | +| cAP ac | 192.168.31.6 | AP | netdisco, public | +| CSS326 | 192.168.31.9 | Switch | public | +| XTRM-Unraid | 192.168.31.2 | Server | netdisco, public | -**Option A: NetBox Plugin (Prototype)** -- Repository: https://github.com/mksoska/netbox-plugin-netdisco -- Status: Prototype stage - may have compatibility issues +### Unraid SNMP Configuration -**Option B: Custom API Sync Script (Recommended)** -- Use NetDisco API + pynetbox library -- More control over sync behavior -- Can be scheduled via cron +**Config File:** `/etc/snmp/snmpd.conf` +``` +rocommunity netdisco +rocommunity public +syslocation XTRM Home Lab +syscontact admin@xtrm-lab.org +sysname XTRM-Unraid +disk /mnt/disk1 +disk /mnt/cache +extend cpumhz /usr/local/emhttp/plugins/snmp/cpu_mhz.sh +extend diskfree /usr/local/emhttp/plugins/snmp/disk_free_space.sh +extend disktemp /usr/local/emhttp/plugins/snmp/disk_temps.sh +extend meminfo /usr/local/emhttp/plugins/snmp/mem_info.sh +extend sharefree /usr/local/emhttp/plugins/snmp/share_free_space.sh +``` -### Task 8.6: Testing & Validation +**Plugin:** kubedzero/unraid-snmp -- [ ] Verify all SNMP queries working -- [ ] Confirm switch port mapping -- [ ] Test MAC address tracking -- [ ] Validate NetBox sync (once implemented) -- [ ] Test external access via Traefik +--- + +## Sync Scripts Summary + +| Script | Purpose | Trigger | +|--------|---------|---------| +| sync_to_netbox.py | NetDisco → NetBox device sync | Manual / cron | +| mikrotik_dhcp_to_netbox.sh | MikroTik DHCP → NetBox IPs | Manual / cron | + +### Scheduling (Cron Examples) + +```bash +# NetDisco sync - hourly at :45 +45 * * * * docker run --rm --network dockerproxy -e NETDISCO_DB_HOST=172.18.0.13 -e NETBOX_URL=http://192.168.31.2:8090 -v /mnt/user/appdata/netdisco/scripts:/scripts python:3.11-slim python /scripts/sync_to_netbox.py >> /mnt/user/appdata/netdisco/logs/sync.log 2>&1 + +# MikroTik DHCP sync - hourly at :50 +50 * * * * /mnt/user/appdata/netdisco/scripts/mikrotik_dhcp_to_netbox.sh >> /mnt/user/appdata/netdisco/logs/dhcp-sync.log 2>&1 +``` + +--- + +## NetBox Credentials + +| Item | Value | +|------|-------| +| URL | http://192.168.31.2:8090 | +| Admin User | admin | +| Admin Password | M0stW4nt3d@xtrm | +| API Token (v1) | kI2HiUD8Bb53KfdkOTaiWYxZuERjhGU30CgGdYxm | +| Secret Key | 56537576d25051987077c7270c876099fe44ad83bc7695a7f4531904fce5678e | --- ## Troubleshooting ### session_cookie_key Error -If web UI shows "session_cookie_key must be defined": +If NetDisco web UI shows "session_cookie_key must be defined": ```sql --- Check if key exists SELECT * FROM sessions WHERE id = 'dancer_session_cookie_key'; - --- Insert if missing INSERT INTO sessions (id, a_session) VALUES ('dancer_session_cookie_key', md5(random()::text)); ``` -### Discovery Not Working -1. Test SNMP from netdisco container: - ```bash - docker exec netdisco-backend snmpwalk -v2c -c netdisco 192.168.31.1 system - ``` -2. Check logs: `tail -f /mnt/user/appdata/netdisco/logs/netdisco-daemon.log` -3. Verify community and IP restrictions on MikroTik devices - ---- - -## References - -- [NetDisco Documentation](https://netdisco.org/) -- [NetDisco GitHub Issues](https://github.com/netdisco/netdisco-docker/issues) -- [MikroTik SNMP Documentation](https://help.mikrotik.com/docs/spaces/ROS/pages/8978519/SNMP) -- [NetBox Plugin for NetDisco](https://github.com/mksoska/netbox-plugin-netdisco) - -### Task 8.5: NetBox Integration (COMPLETED) - -**Implementation: Custom Sync Script** - -The prototype NetBox plugin for NetDisco was not production-ready, so a custom Python sync script was created. - -**Script Location:** `/mnt/user/appdata/netdisco/scripts/sync_to_netbox.py` - -**What It Does:** -1. Reads device data from NetDisco PostgreSQL database -2. Matches devices in NetBox by name -3. Updates NetBox device comments with NetDisco data: - - IP address - - OS version - - Model - - Vendor - - Last SNMP discovery timestamp - -**NetBox API Token:** -- Token: `kI2HiUD8Bb53KfdkOTaiWYxZuERjhGU30CgGdYxm` -- Type: v1 (plaintext) -- User: admin -- Permissions: Write-enabled - -**Running the Sync:** +### NetBox Plugin Not Loading +Check container logs: ```bash -docker run --rm --network dockerproxy \ - -v /mnt/user/appdata/netdisco/scripts:/scripts \ - python:3.11-slim sh -c \ - 'pip install -q psycopg2-binary requests && python /scripts/sync_to_netbox.py' +docker logs netbox 2>&1 | grep -i plugin ``` -**Sync Results:** -| NetBox Device | NetDisco Device | IP | Status | -|---------------|-----------------|-----|--------| -| HAPax3 | HAPax3 | 192.168.31.1 | ✅ Synced | -| CSS326 | CSS326-24G-2S+ | 192.168.31.9 | ✅ Synced | -| CAP-XL-ac | CAP XL ac | 192.168.31.6 | ✅ Synced | - -**Scheduling (Optional):** -To run automatically, add to cron: +Verify plugins.py is mounted: ```bash -# Run sync hourly at :45 -45 * * * * docker run --rm --network dockerproxy -v /mnt/user/appdata/netdisco/scripts:/scripts python:3.11-slim sh -c 'pip install -q psycopg2-binary requests 2>/dev/null && python /scripts/sync_to_netbox.py' >> /mnt/user/appdata/netdisco/logs/sync.log 2>&1 +docker exec netbox cat /etc/netbox/config/plugins.py ``` - -### Task 8.6: Testing & Validation (COMPLETED) - -**Verification Results:** - -| Test | Result | -|------|--------| -| SNMP to hAP ax³ (192.168.31.1) | ✅ Responding | -| SNMP to cAP ac (192.168.31.6) | ✅ Responding | -| SNMP to CSS326 (192.168.31.9) | ✅ Responding | -| Device discovery | ✅ 3 devices discovered | -| Port mapping | ✅ 52 ports collected | -| MAC address tracking | ✅ 19 nodes tracked | -| NetBox sync | ✅ All 3 devices synced | -| Internal access (port 5000) | ✅ HTTP 200 | -| External access (Traefik) | ✅ SSO redirect working | - -**Data Collected:** -- 3 network devices (MikroTik) -- 52 switch/router ports -- 19 MAC addresses (nodes) -- Automatic discovery every 2 hours -- MAC polling hourly +### SNMP Not Responding +```bash +snmpwalk -v2c -c netdisco 192.168.31.1 system +``` --- @@ -334,18 +219,16 @@ To run automatically, add to cron: **Status: ✅ COMPLETED** -All tasks in Phase 8 have been successfully completed: -- SNMP enabled on all MikroTik devices -- NetDisco deployed and discovering network -- Traefik ingress with Authentik SSO configured -- NetBox integration via custom sync script -- All functionality verified +All tasks completed with enhanced capabilities: +- 4 devices discovered via SNMP (MikroTik + Unraid) +- NetDisco → NetBox sync with device details +- MikroTik DHCP → NetBox sync (29 devices/IPs) +- Slurp'it plugin installed for future discovery expansion +- ARP table sync for MAC-to-IP mapping -**Access URLs:** -- NetDisco Web: https://netdisco.xtrm-lab.org (with SSO) -- NetDisco Internal: http://192.168.31.2:5000 - -**Maintenance:** -- Discovery runs automatically every 2 hours -- MAC/ARP polling runs hourly -- NetBox sync can be run manually or scheduled via cron +**Data Flow:** +``` +MikroTik Devices ─SNMP─> NetDisco ─sync─> NetBox + │ │ ↑ + └──────DHCP──────────sync────────────┘ +```