Phase 8: Enhanced network mapping with sync scripts

- Added MikroTik DHCP to NetBox sync script - Installed Slurp'it plugin v1.2.7 in NetBox - Enhanced NetDisco to NetBox sync with ARP/MAC data - Enabled SNMP on Unraid server (4 devices now discoverable) - Updated Phase 8 documentation with all scripts and configs Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-19 23:43:55 +02:00
parent 1861a34ed8
commit 0e35142e10
2 changed files with 146 additions and 240 deletions
--- a/docs/06-CHANGELOG.md
+++ b/docs/06-CHANGELOG.md
@@ -0,0 +1,23 @@
+
+## 2026-01-19 - Phase 8 Enhanced Network Mapping
+
+### MikroTik DHCP Sync
+- [SCRIPT] Created mikrotik_dhcp_to_netbox.sh
+- [SYNC] 29 DHCP leases synced to NetBox IPs
+- [DATA] Hostname, MAC, comments captured
+
+### Slurp'it Plugin Installation
+- [PLUGIN] Installed slurpit_netbox v1.2.7
+- [BUILD] Created netbox-custom:latest image
+- [CONFIG] Plugin configuration at /mnt/user/appdata/netbox/config/plugins.py
+
+### Enhanced NetDisco Sync
+- [SCRIPT] Updated sync_to_netbox.py with additional data
+- [SYNC] Device info, IPs, MACs, ARP table entries
+- [DATA] 4 devices synced with full metadata
+
+### Unraid SNMP
+- [SERVICE] kubedzero/unraid-snmp plugin installed
+- [CONFIG] Communities: netdisco, public
+- [DISCOVER] XTRM-Unraid now visible in NetDisco
+
--- a/docs/12-PHASE8-NETDISCO-INTEGRATION.md
+++ b/docs/12-PHASE8-NETDISCO-INTEGRATION.md
@@ -1,6 +1,6 @@
 # Phase 8: NetDisco Integration with NetBox

-**Status:** ✅ COMPLETED
+**Status:** ✅ COMPLETED (Enhanced)
 **Priority:** Medium
 **Created:** 2026-01-19
 **Last Updated:** 2026-01-19
@@ -25,6 +25,7 @@ Deploy NetDisco for enterprise-grade network discovery using SNMP, and integrate
 | 8.4 | Traefik Ingress for NetDisco | ✅ COMPLETED |
 | 8.5 | NetBox Integration | ✅ COMPLETED |
 | 8.6 | Testing & Validation | ✅ COMPLETED |
+| 8.7 | Enhanced Network Mapping | ✅ COMPLETED |

 ---

@@ -33,10 +34,10 @@ Deploy NetDisco for enterprise-grade network discovery using SNMP, and integrate
 | Tool | Status | Purpose |
 |------|--------|---------|
 | NetAlertX | Running | ARP/ICMP-based device discovery |
-| NetBox | Running | IPAM/DCIM (manual data entry) |
+| NetBox | Running | IPAM/DCIM with Slurpit plugin |
 | NetDisco | ✅ Running | SNMP-based discovery |

-**Decision:** Keep NetAlertX running in parallel during transition.
+**Discovered Devices:** 4 (hAP ax³, cAP ac, CSS326, Unraid)

 ---

@@ -53,280 +54,164 @@ Deploy NetDisco for enterprise-grade network discovery using SNMP, and integrate
 |-----------|-------|------|--------|
 | netdisco-web | netdisco/netdisco:latest-web | 5000 | Running |
 | netdisco-backend | netdisco/netdisco:latest-backend | - | Running |
+| netbox | netbox-custom:latest (with slurpit) | 8090 | Running |

 ### Access URLs
 | URL | Purpose |
 |-----|---------|
 | https://netdisco.xtrm-lab.org | External access (Authentik SSO) |
-| http://192.168.31.2:5000 | Internal direct access |
-
-### Configuration
- **Config Path:** /mnt/user/appdata/netdisco/environments/deployment.yml
- **Logs:** /mnt/user/appdata/netdisco/logs/
+| http://192.168.31.2:5000 | Internal NetDisco access |
+| https://netbox.xtrm-lab.org | External NetBox access |
+| http://192.168.31.2:8090 | Internal NetBox access |

 ---

-## Completed Tasks
+## Task 8.7: Enhanced Network Mapping (COMPLETED)

-### Task 8.1: SNMP Configuration (COMPLETED)
+### 1. MikroTik DHCP → NetBox Sync

-| Device | IP | Communities | Access Restriction |
-|--------|-----|-------------|-------------------|
-| hAP ax³ | 192.168.31.1 | `netdisco`, `public` | 192.168.31.2/32 |
-| cAP ac | 192.168.31.6 | `netdisco`, `public` | 192.168.31.2/32 |
-| CSS326 | 192.168.31.9 | `public` | None (SwOS limitation) |
+**Script:** `/mnt/user/appdata/netdisco/scripts/mikrotik_dhcp_to_netbox.sh`

-**SNMP Settings:**
- Contact: admin@xtrm-lab.org
- Location: XTRM Home Lab
- Version: SNMPv2c
- Write Access: Disabled
+Syncs DHCP leases from MikroTik router to NetBox IP addresses with:
+- Hostname
+- MAC address
+- Comments/notes from DHCP

-### Task 8.2: NetDisco Deployment (COMPLETED)
-
-**Docker Compose:** /mnt/user/appdata/netdisco/docker-compose.yml
-
-```yaml
-services:
-  netdisco-backend:
-    container_name: netdisco-backend
-    image: netdisco/netdisco:latest-backend
-    hostname: netdisco-backend
-    restart: unless-stopped
-    volumes:
-      - /mnt/user/appdata/netdisco/nd-site-local:/home/netdisco/nd-site-local
-      - /mnt/user/appdata/netdisco/environments:/home/netdisco/environments
-      - /mnt/user/appdata/netdisco/logs:/home/netdisco/logs
-    environment:
-      NETDISCO_DOMAIN: deployment
-      NETDISCO_DB_HOST: postgresql17
-      NETDISCO_DB_PORT: 5432
-      NETDISCO_DB_NAME: netdisco_db
-      NETDISCO_DB_USER: netdisco_user
-      NETDISCO_DB_PASS: NetD1sc0_M0stW4nt3d@db
-    networks:
-      - dockerproxy
-
-  netdisco-web:
-    container_name: netdisco-web
-    image: netdisco/netdisco:latest-web
-    hostname: netdisco-web
-    restart: unless-stopped
-    volumes:
-      - /mnt/user/appdata/netdisco/nd-site-local:/home/netdisco/nd-site-local
-      - /mnt/user/appdata/netdisco/environments:/home/netdisco/environments
-      - /mnt/user/appdata/netdisco/logs:/home/netdisco/logs
-    environment:
-      NETDISCO_DOMAIN: deployment
-      NETDISCO_DB_HOST: postgresql17
-      NETDISCO_DB_PORT: 5432
-      NETDISCO_DB_NAME: netdisco_db
-      NETDISCO_DB_USER: netdisco_user
-      NETDISCO_DB_PASS: NetD1sc0_M0stW4nt3d@db
-      PORT: 5000
-    networks:
-      - dockerproxy
-
-networks:
-  dockerproxy:
-    external: true
+**Run Command:**
+```bash
+/mnt/user/appdata/netdisco/scripts/mikrotik_dhcp_to_netbox.sh
 ```

-### Task 8.3: Discovery Configuration (COMPLETED)
+**Result:** 29 devices synced (26 created, 3 updated)

-**deployment.yml:**
-```yaml
-database:
-  name: 'netdisco_db'
-  user: 'netdisco_user'
-  pass: 'NetD1sc0_M0stW4nt3d@db'
-  host: 'postgresql17'
+### 2. Slurp'it NetBox Plugin

-community_rw:
-  - netdisco
-  - public
+**Plugin Version:** 1.2.7  
+**NetBox Version:** 4.5.0

-discover_only:
-  - 192.168.31.1  # hAP ax³
-  - 192.168.31.6  # cAP ac
-  - 192.168.31.9  # CSS326
+**Custom Image:** `netbox-custom:latest`

-site_local_files: true
-no_auth: true
-
-schedule:
-  discover: '0 */2 * * *'    # Every 2 hours
-  macsuck:  '15 * * * *'     # MAC poll hourly
-  arpnip:   '30 * * * *'     # ARP poll hourly
-  nbtstat:  '45 * * * *'     # NetBIOS hourly
+**Plugin Configuration:** `/mnt/user/appdata/netbox/config/plugins.py`
+```python
+PLUGINS = ["slurpit_netbox"]
+PLUGINS_CONFIG = {
+    "slurpit_netbox": {}
+}
 ```

-**Discovery Results:**
-| Device | IP | Status |
-|--------|-----|--------|
-| hAP ax³ | 192.168.31.1 | ✅ Discovered |
-| cAP ac | 192.168.31.6 | ✅ Discovered |
-| CSS326 | 192.168.31.9 | ✅ Discovered |
+**Note:** Slurp'it requires a separate Slurp'it server for full network discovery. The plugin provides NetBox integration once the server is configured.

-### Task 8.4: Traefik Ingress (COMPLETED)
+### 3. Enhanced NetDisco → NetBox Sync

-**Traefik Dynamic Config:** /mnt/user/appdata/traefik/dynamic.yml
+**Script:** `/mnt/user/appdata/netdisco/scripts/sync_to_netbox.py`

-```yaml
-http:
-  routers:
-    netdisco-secure:
-      rule: "Host(\`netdisco.xtrm-lab.org\`)"
-      entryPoints:
-        - https
-      middlewares:
-        - default-headers
-        - authentik-forward-auth
-      tls:
-        certResolver: cloudflare
-      service: netdisco
+Enhanced version syncs:
+- Device info (serial, OS, model, vendor)
+- IP addresses from device_ip table
+- ARP table entries (MAC to IP mappings)
+- System MAC addresses
+- Uptime information

-  services:
-    netdisco:
-      loadBalancer:
-        servers:
-          - url: "http://netdisco-web:5000"
-```
+**Run Command:**
+```bash
+PG17_IP=$(docker inspect postgresql17 --format "{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}")

-**Important Fix Applied:**
-
-The web UI failed to start with error: `The setting session_cookie_key must be defined`
-
-**Root Cause:** When using external PostgreSQL, the `dancer_session_cookie_key` record was missing from the sessions table.
-
-**Fix:**
-```sql
-INSERT INTO sessions (id, a_session) VALUES ('dancer_session_cookie_key', md5(random()::text));
+docker run --rm \
+  --network dockerproxy \
+  -e NETDISCO_DB_HOST=$PG17_IP \
+  -e NETBOX_URL=http://192.168.31.2:8090 \
+  -v /mnt/user/appdata/netdisco/scripts:/scripts \
+  python:3.11-slim python /scripts/sync_to_netbox.py
 ```

 ---

-## Pending Tasks
+## SNMP-Enabled Devices

-### Task 8.5: NetBox Integration
+| Device | IP | Type | Communities |
+|--------|-----|------|-------------|
+| hAP ax³ | 192.168.31.1 | Router | netdisco, public |
+| cAP ac | 192.168.31.6 | AP | netdisco, public |
+| CSS326 | 192.168.31.9 | Switch | public |
+| XTRM-Unraid | 192.168.31.2 | Server | netdisco, public |

-**Option A: NetBox Plugin (Prototype)**
- Repository: https://github.com/mksoska/netbox-plugin-netdisco
- Status: Prototype stage - may have compatibility issues
+### Unraid SNMP Configuration

-**Option B: Custom API Sync Script (Recommended)**
- Use NetDisco API + pynetbox library
- More control over sync behavior
- Can be scheduled via cron
+**Config File:** `/etc/snmp/snmpd.conf`
+```
+rocommunity netdisco
+rocommunity public
+syslocation XTRM Home Lab
+syscontact admin@xtrm-lab.org
+sysname XTRM-Unraid
+disk /mnt/disk1
+disk /mnt/cache
+extend cpumhz /usr/local/emhttp/plugins/snmp/cpu_mhz.sh
+extend diskfree /usr/local/emhttp/plugins/snmp/disk_free_space.sh
+extend disktemp /usr/local/emhttp/plugins/snmp/disk_temps.sh
+extend meminfo /usr/local/emhttp/plugins/snmp/mem_info.sh
+extend sharefree /usr/local/emhttp/plugins/snmp/share_free_space.sh
+```

-### Task 8.6: Testing & Validation
+**Plugin:** kubedzero/unraid-snmp

- [ ] Verify all SNMP queries working
- [ ] Confirm switch port mapping
- [ ] Test MAC address tracking
- [ ] Validate NetBox sync (once implemented)
- [ ] Test external access via Traefik
+---
+
+## Sync Scripts Summary
+
+| Script | Purpose | Trigger |
+|--------|---------|---------|
+| sync_to_netbox.py | NetDisco → NetBox device sync | Manual / cron |
+| mikrotik_dhcp_to_netbox.sh | MikroTik DHCP → NetBox IPs | Manual / cron |
+
+### Scheduling (Cron Examples)
+
+```bash
+# NetDisco sync - hourly at :45
+45 * * * * docker run --rm --network dockerproxy -e NETDISCO_DB_HOST=172.18.0.13 -e NETBOX_URL=http://192.168.31.2:8090 -v /mnt/user/appdata/netdisco/scripts:/scripts python:3.11-slim python /scripts/sync_to_netbox.py >> /mnt/user/appdata/netdisco/logs/sync.log 2>&1
+
+# MikroTik DHCP sync - hourly at :50
+50 * * * * /mnt/user/appdata/netdisco/scripts/mikrotik_dhcp_to_netbox.sh >> /mnt/user/appdata/netdisco/logs/dhcp-sync.log 2>&1
+```
+
+---
+
+## NetBox Credentials
+
+| Item | Value |
+|------|-------|
+| URL | http://192.168.31.2:8090 |
+| Admin User | admin |
+| Admin Password | M0stW4nt3d@xtrm |
+| API Token (v1) | kI2HiUD8Bb53KfdkOTaiWYxZuERjhGU30CgGdYxm |
+| Secret Key | 56537576d25051987077c7270c876099fe44ad83bc7695a7f4531904fce5678e |

 ---

 ## Troubleshooting

 ### session_cookie_key Error
-If web UI shows "session_cookie_key must be defined":
+If NetDisco web UI shows "session_cookie_key must be defined":
 ```sql
-- Check if key exists
 SELECT * FROM sessions WHERE id = 'dancer_session_cookie_key';
-
-- Insert if missing
 INSERT INTO sessions (id, a_session) VALUES ('dancer_session_cookie_key', md5(random()::text));
 ```

-### Discovery Not Working
-1. Test SNMP from netdisco container:
-   ```bash
-   docker exec netdisco-backend snmpwalk -v2c -c netdisco 192.168.31.1 system
-   ```
-2. Check logs: `tail -f /mnt/user/appdata/netdisco/logs/netdisco-daemon.log`
-3. Verify community and IP restrictions on MikroTik devices
-
---
-
-## References
-
- [NetDisco Documentation](https://netdisco.org/)
- [NetDisco GitHub Issues](https://github.com/netdisco/netdisco-docker/issues)
- [MikroTik SNMP Documentation](https://help.mikrotik.com/docs/spaces/ROS/pages/8978519/SNMP)
- [NetBox Plugin for NetDisco](https://github.com/mksoska/netbox-plugin-netdisco)
-
-### Task 8.5: NetBox Integration (COMPLETED)
-
-**Implementation: Custom Sync Script**
-
-The prototype NetBox plugin for NetDisco was not production-ready, so a custom Python sync script was created.
-
-**Script Location:** `/mnt/user/appdata/netdisco/scripts/sync_to_netbox.py`
-
-**What It Does:**
-1. Reads device data from NetDisco PostgreSQL database
-2. Matches devices in NetBox by name
-3. Updates NetBox device comments with NetDisco data:
-   - IP address
-   - OS version
-   - Model
-   - Vendor
-   - Last SNMP discovery timestamp
-
-**NetBox API Token:**
- Token: `kI2HiUD8Bb53KfdkOTaiWYxZuERjhGU30CgGdYxm`
- Type: v1 (plaintext)
- User: admin
- Permissions: Write-enabled
-
-**Running the Sync:**
+### NetBox Plugin Not Loading
+Check container logs:
 ```bash
-docker run --rm --network dockerproxy \
-  -v /mnt/user/appdata/netdisco/scripts:/scripts \
-  python:3.11-slim sh -c \
-  'pip install -q psycopg2-binary requests && python /scripts/sync_to_netbox.py'
+docker logs netbox 2>&1 | grep -i plugin
 ```

-**Sync Results:**
-| NetBox Device | NetDisco Device | IP | Status |
-|---------------|-----------------|-----|--------|
-| HAPax3 | HAPax3 | 192.168.31.1 | ✅ Synced |
-| CSS326 | CSS326-24G-2S+ | 192.168.31.9 | ✅ Synced |
-| CAP-XL-ac | CAP XL ac | 192.168.31.6 | ✅ Synced |
-
-**Scheduling (Optional):**
-To run automatically, add to cron:
+Verify plugins.py is mounted:
 ```bash
-# Run sync hourly at :45
-45 * * * * docker run --rm --network dockerproxy -v /mnt/user/appdata/netdisco/scripts:/scripts python:3.11-slim sh -c 'pip install -q psycopg2-binary requests 2>/dev/null && python /scripts/sync_to_netbox.py' >> /mnt/user/appdata/netdisco/logs/sync.log 2>&1
+docker exec netbox cat /etc/netbox/config/plugins.py
 ```

-
-### Task 8.6: Testing & Validation (COMPLETED)
-
-**Verification Results:**
-
-| Test | Result |
-|------|--------|
-| SNMP to hAP ax³ (192.168.31.1) | ✅ Responding |
-| SNMP to cAP ac (192.168.31.6) | ✅ Responding |
-| SNMP to CSS326 (192.168.31.9) | ✅ Responding |
-| Device discovery | ✅ 3 devices discovered |
-| Port mapping | ✅ 52 ports collected |
-| MAC address tracking | ✅ 19 nodes tracked |
-| NetBox sync | ✅ All 3 devices synced |
-| Internal access (port 5000) | ✅ HTTP 200 |
-| External access (Traefik) | ✅ SSO redirect working |
-
-**Data Collected:**
- 3 network devices (MikroTik)
- 52 switch/router ports
- 19 MAC addresses (nodes)
- Automatic discovery every 2 hours
- MAC polling hourly
+### SNMP Not Responding
+```bash
+snmpwalk -v2c -c netdisco 192.168.31.1 system
+```

 ---

@@ -334,18 +219,16 @@ To run automatically, add to cron:

 **Status: ✅ COMPLETED**

-All tasks in Phase 8 have been successfully completed:
- SNMP enabled on all MikroTik devices
- NetDisco deployed and discovering network
- Traefik ingress with Authentik SSO configured
- NetBox integration via custom sync script
- All functionality verified
+All tasks completed with enhanced capabilities:
+- 4 devices discovered via SNMP (MikroTik + Unraid)
+- NetDisco → NetBox sync with device details
+- MikroTik DHCP → NetBox sync (29 devices/IPs)
+- Slurp'it plugin installed for future discovery expansion
+- ARP table sync for MAC-to-IP mapping

-**Access URLs:**
- NetDisco Web: https://netdisco.xtrm-lab.org (with SSO)
- NetDisco Internal: http://192.168.31.2:5000
-
-**Maintenance:**
- Discovery runs automatically every 2 hours
- MAC/ARP polling runs hourly
- NetBox sync can be run manually or scheduled via cron
+**Data Flow:**
+```
+MikroTik Devices ─SNMP─> NetDisco ─sync─> NetBox
+       │                     │              ↑
+       └──────DHCP──────────sync────────────┘
+```