jazzymc/infrastructure
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Update Phase 8: Tasks 8.1-8.4 completed, document session_cookie_key fix
All checks were successful
ci/woodpecker/push/woodpecker Pipeline was successful
Details

Browse Source
This commit is contained in:
XTRM Admin
2026-01-19 21:28:11 +02:00
parent a4e589b583
commit 1ddfc68192
1 changed files with 152 additions and 254 deletions
Download Patch File Download Diff File Expand all files Collapse all files

406
docs/12-PHASE8-NETDISCO-INTEGRATION.md
View File

@@ -1,8 +1,9 @@
# Phase 8: NetDisco Integration with NetBox # Phase 8: NetDisco Integration with NetBox
**Status:** IN PROGRESS **Status:** IN PROGRESS (Tasks 8.1-8.4 Complete)
**Priority:** Medium **Priority:** Medium
**Created:** 2026-01-19 **Created:** 2026-01-19
**Last Updated:** 2026-01-19
--- ---
@@ -14,99 +15,60 @@ Deploy NetDisco for enterprise-grade network discovery using SNMP, and integrate
--- ---
## Task Status Summary
| Task | Description | Status |
|------|-------------|--------|
| 8.1 | Enable SNMP on MikroTik Devices | ✅ COMPLETED |
| 8.2 | Deploy NetDisco on Unraid | ✅ COMPLETED |
| 8.3 | Configure NetDisco Discovery | ✅ COMPLETED |
| 8.4 | Traefik Ingress for NetDisco | ✅ COMPLETED |
| 8.5 | NetBox Integration | ⏳ PENDING |
| 8.6 | Testing & Validation | ⏳ PENDING |
---
## Current State ## Current State
| Tool | Status | Purpose | | Tool | Status | Purpose |
|------|--------|---------| |------|--------|---------|
| NetAlertX | Running | ARP/ICMP-based device discovery | | NetAlertX | Running | ARP/ICMP-based device discovery |
| NetBox | Running | IPAM/DCIM (manual data entry) | | NetBox | Running | IPAM/DCIM (manual data entry) |
| NetDisco | Not deployed | SNMP-based discovery | | NetDisco | ✅ Running | SNMP-based discovery |
**Decision:** Keep NetAlertX running in parallel during transition. **Decision:** Keep NetAlertX running in parallel during transition.
--- ---
## Architecture ## Deployment Details
``` ### Database
┌─────────────────────────────────────────────────────────────┐ - **Host:** postgresql17
│ SNMP-Enabled Devices │ - **Database:** netdisco_db
├─────────────────┬─────────────────┬─────────────────────────┤ - **User:** netdisco_user
│ hAP ax³ │ CSS326 │ cAP ac │ - **Password:** NetD1sc0_M0stW4nt3d@db
│ 192.168.31.1 │ 192.168.31.9 │ 192.168.31.6 │
│ RouterOS 7.x │ SwOS │ RouterOS 7.x │ ### Containers
│ SNMPv2c/v3 │ SNMPv1/v2c │ SNMPv2c/v3 │ | Container | Image | Port | Status |
└────────┬────────┴────────┬────────┴────────┬────────────────┘ |-----------|-------|------|--------|
│ │ │ | netdisco-web | netdisco/netdisco:latest-web | 5000 | Running |
└─────────────────┼─────────────────┘ | netdisco-backend | netdisco/netdisco:latest-backend | - | Running |
│ SNMP Polling
▼ ### Access URLs
┌─────────────────────┐ | URL | Purpose |
│ NetDisco │ |-----|---------|
│ 192.168.31.2:5000 │ | https://netdisco.xtrm-lab.org | External access (Authentik SSO) |
│ (Unraid Docker) │ | http://192.168.31.2:5000 | Internal direct access |
└──────────┬──────────┘
│ API Sync ### Configuration
▼ - **Config Path:** /mnt/user/appdata/netdisco/environments/deployment.yml
┌─────────────────────┐ - **Logs:** /mnt/user/appdata/netdisco/logs/
│ NetBox │
│ 192.168.31.2:8090 │
│ netbox-plugin │
└─────────────────────┘
```
--- ---
## Tasks ## Completed Tasks
### Task 8.1: Enable SNMP on MikroTik Devices ### Task 8.1: SNMP Configuration (COMPLETED)
#### 8.1.1 hAP ax³ (RouterOS)
```routeros
# SSH to router
ssh -p 2222 xtrm@192.168.31.1
# Enable SNMP
/snmp set enabled=yes contact="admin@xtrm-lab.org" location="XTRM Home Lab"
# Create community (restrict to Unraid only)
/snmp community add name=netdisco addresses=192.168.31.2/32 read-access=yes write-access=no
# Verify
/snmp print
/snmp community print
```
**Status:** [x] COMPLETED (2026-01-19)
#### 8.1.2 CSS326 (SwOS)
1. Access SwOS web UI: http://192.168.31.9/index.html
2. Navigate to **SNMP** tab
3. Configure:
- Enable: Yes
- Community: netdisco
- IP Address Filter: 192.168.31.2
**Note:** SwOS only supports SNMPv1/v2c read-only.
**Status:** [x] COMPLETED (2026-01-19)
#### 8.1.3 cAP ac (RouterOS)
```routeros
# SSH to cAP
ssh -p 2222 xtrm@192.168.31.6
# Enable SNMP (same config as hAP ax³)
/snmp set enabled=yes contact="admin@xtrm-lab.org" location="XTRM Home Lab"
/snmp community add name=netdisco addresses=192.168.31.2/32 read-access=yes write-access=no
```
**Status:** [x] COMPLETED (2026-01-19)
#### SNMP Configuration Applied (2026-01-19)
| Device | IP | Communities | Access Restriction | | Device | IP | Communities | Access Restriction |
|--------|-----|-------------|-------------------| |--------|-----|-------------|-------------------|
@@ -120,242 +82,178 @@ ssh -p 2222 xtrm@192.168.31.6
- Version: SNMPv2c - Version: SNMPv2c
- Write Access: Disabled - Write Access: Disabled
**Test Results:** ### Task 8.2: NetDisco Deployment (COMPLETED)
```
hAP ax³: SNMPv2-MIB::sysName.0 = STRING: HAPax3
cAP ac: SNMPv2-MIB::sysName.0 = STRING: CAP XL ac
CSS326: SNMPv2-MIB::sysName.0 = STRING: CSS326-24G-2S+
```
**Docker Compose:** /mnt/user/appdata/netdisco/docker-compose.yml
---
### Task 8.2: Deploy NetDisco on Unraid
**Status:** [x] COMPLETED (2026-01-19)
#### Docker Compose
```yaml ```yaml
version: '3'
services: services:
netdisco: netdisco-backend:
image: netdisco/netdisco:latest container_name: netdisco-backend
container_name: netdisco image: netdisco/netdisco:latest-backend
hostname: netdisco-backend
restart: unless-stopped restart: unless-stopped
ports:
- "5000:5000"
environment:
- NETDISCO_DOMAIN=netdisco.xtrm-lab.org
- NETDISCO_DB_HOST=postgresql17
- NETDISCO_DB_NAME=netdisco
- NETDISCO_DB_USER=netdisco
- NETDISCO_DB_PASS=<secure_password>
- NETDISCO_ADMIN_USER=admin
- NETDISCO_ADMIN_PASS=<secure_password>
volumes: volumes:
- /mnt/user/appdata/netdisco/config:/home/netdisco/environments
- /mnt/user/appdata/netdisco/logs:/home/netdisco/logs
- /mnt/user/appdata/netdisco/nd-site-local:/home/netdisco/nd-site-local - /mnt/user/appdata/netdisco/nd-site-local:/home/netdisco/nd-site-local
- /mnt/user/appdata/netdisco/environments:/home/netdisco/environments
- /mnt/user/appdata/netdisco/logs:/home/netdisco/logs
environment:
NETDISCO_DOMAIN: deployment
NETDISCO_DB_HOST: postgresql17
NETDISCO_DB_PORT: 5432
NETDISCO_DB_NAME: netdisco_db
NETDISCO_DB_USER: netdisco_user
NETDISCO_DB_PASS: NetD1sc0_M0stW4nt3d@db
networks:
- dockerproxy
netdisco-web:
container_name: netdisco-web
image: netdisco/netdisco:latest-web
hostname: netdisco-web
restart: unless-stopped
volumes:
- /mnt/user/appdata/netdisco/nd-site-local:/home/netdisco/nd-site-local
- /mnt/user/appdata/netdisco/environments:/home/netdisco/environments
- /mnt/user/appdata/netdisco/logs:/home/netdisco/logs
environment:
NETDISCO_DOMAIN: deployment
NETDISCO_DB_HOST: postgresql17
NETDISCO_DB_PORT: 5432
NETDISCO_DB_NAME: netdisco_db
NETDISCO_DB_USER: netdisco_user
NETDISCO_DB_PASS: NetD1sc0_M0stW4nt3d@db
PORT: 5000
networks: networks:
- dockerproxy - dockerproxy
depends_on:
- postgresql17
networks: networks:
dockerproxy: dockerproxy:
external: true external: true
``` ```
#### Database Setup ### Task 8.3: Discovery Configuration (COMPLETED)
```bash
# Create NetDisco database in existing PostgreSQL
docker exec -it postgresql17 psql -U postgres -c "CREATE USER netdisco WITH PASSWORD 'secure_password';"
docker exec -it postgresql17 psql -U postgres -c "CREATE DATABASE netdisco OWNER netdisco;"
```
**Status:** [x] COMPLETED (2026-01-19)
---
### Task 8.3: Configure NetDisco Discovery
#### deployment.yml Configuration
**deployment.yml:**
```yaml ```yaml
# /mnt/user/appdata/netdisco/config/deployment.yml database:
snmp_community: name: 'netdisco_db'
- community: netdisco user: 'netdisco_user'
read: true pass: 'NetD1sc0_M0stW4nt3d@db'
write: false host: 'postgresql17'
device_auth: community_rw:
- tag: mikrotik - netdisco
community: netdisco - public
snmp_version: 2
discover_only: discover_only:
- 192.168.31.1 # hAP ax³ - 192.168.31.1 # hAP ax³
- 192.168.31.6 # cAP ac - 192.168.31.6 # cAP ac
- 192.168.31.9 # CSS326 - 192.168.31.9 # CSS326
site_local_files: true
no_auth: true
schedule: schedule:
discover: '0 * * * *' # Hourly discovery discover: '0 */2 * * *' # Every 2 hours
macsuck: '15 * * * *' # MAC address table poll macsuck: '15 * * * *' # MAC poll hourly
arpnip: '30 * * * *' # ARP table poll arpnip: '30 * * * *' # ARP poll hourly
nbtstat: '45 * * * *' # NetBIOS name poll nbtstat: '45 * * * *' # NetBIOS hourly
``` ```
**Status:** [x] COMPLETED (2026-01-19) **Discovery Results:**
| Device | IP | Status |
|--------|-----|--------|
| hAP ax³ | 192.168.31.1 | ✅ Discovered |
| cAP ac | 192.168.31.6 | ✅ Discovered |
| CSS326 | 192.168.31.9 | ✅ Discovered |
--- ### Task 8.4: Traefik Ingress (COMPLETED)
### Task 8.4: Traefik Ingress for NetDisco **Traefik Dynamic Config:** /mnt/user/appdata/traefik/dynamic.yml
Add to Traefik dynamic configuration:
```yaml ```yaml
http: http:
routers: routers:
netdisco: netdisco-secure:
rule: "Host(`netdisco.xtrm-lab.org`)" rule: "Host(\`netdisco.xtrm-lab.org\`)"
entryPoints: entryPoints:
- websecure - https
service: netdisco middlewares:
- default-headers
- authentik-forward-auth
tls: tls:
certResolver: cloudflare certResolver: cloudflare
service: netdisco
services: services:
netdisco: netdisco:
loadBalancer: loadBalancer:
servers: servers:
- url: "http://netdisco:5000" - url: "http://netdisco-web:5000"
``` ```
**DNS:** Add netdisco.xtrm-lab.org to Cloudflare **Important Fix Applied:**
**Status:** [x] COMPLETED (2026-01-19) The web UI failed to start with error: `The setting session_cookie_key must be defined`
**Root Cause:** When using external PostgreSQL, the `dancer_session_cookie_key` record was missing from the sessions table.
**Fix:**
```sql
INSERT INTO sessions (id, a_session) VALUES ('dancer_session_cookie_key', md5(random()::text));
```
--- ---
## Pending Tasks
### Task 8.5: NetBox Integration ### Task 8.5: NetBox Integration
#### Option A: NetBox Plugin (Prototype) **Option A: NetBox Plugin (Prototype)**
- Repository: https://github.com/mksoska/netbox-plugin-netdisco
- Status: Prototype stage - may have compatibility issues
Repository: https://github.com/mksoska/netbox-plugin-netdisco **Option B: Custom API Sync Script (Recommended)**
- Use NetDisco API + pynetbox library
```bash - More control over sync behavior
# Install in NetBox container - Can be scheduled via cron
docker exec -it netbox pip install netbox-plugin-netdisco
# Add to NetBox configuration.py
PLUGINS = ['netbox_netdisco']
PLUGINS_CONFIG = {
'netbox_netdisco': {
'NETDISCO_HOST': 'http://netdisco:5000',
'NETDISCO_USERNAME': 'admin',
'NETDISCO_PASSWORD': '<password>',
}
}
```
**Note:** Plugin is prototype-stage. Test thoroughly.
#### Option B: Custom API Sync Script
Create a sync script using NetDisco API + pynetbox:
```python
#!/usr/bin/env python3
# /mnt/user/appdata/netbox/scripts/netdisco_sync.py
import requests
import pynetbox
NETDISCO_URL = "http://192.168.31.2:5000"
NETBOX_URL = "http://192.168.31.2:8090"
NETBOX_TOKEN = "<api_token>"
nb = pynetbox.api(NETBOX_URL, token=NETBOX_TOKEN)
# Fetch devices from NetDisco
devices = requests.get(f"{NETDISCO_URL}/api/v1/device").json()
for device in devices:
# Check if device exists in NetBox
existing = nb.dcim.devices.get(primary_ip=device['ip'])
if not existing:
print(f"New device found: {device['ip']} ({device['name']})")
# Create or flag for review
```
**Status:** [x] COMPLETED (2026-01-19)
---
### Task 8.6: Testing & Validation ### Task 8.6: Testing & Validation
#### Checklist - [ ] Verify all SNMP queries working
- [ ] Confirm switch port mapping
- [ ] Test MAC address tracking
- [ ] Validate NetBox sync (once implemented)
- [ ] Test external access via Traefik
- [ ] SNMP reachable from Unraid to all MikroTik devices ---
- [ ] NetDisco discovers hAP ax³, CSS326, cAP ac
- [ ] Switch port mapping shows connected devices
- [ ] MAC address tracking working
- [ ] NetBox sync shows discovered devices
- [ ] Traefik proxy working (netdisco.xtrm-lab.org)
#### SNMP Test Commands ## Troubleshooting
```bash ### session_cookie_key Error
# Test SNMP from Unraid If web UI shows "session_cookie_key must be defined":
docker run --rm --network dockerproxy netdisco/netdisco snmpwalk -v2c -c netdisco 192.168.31.1 system ```sql
docker run --rm --network dockerproxy netdisco/netdisco snmpwalk -v2c -c netdisco 192.168.31.9 system -- Check if key exists
docker run --rm --network dockerproxy netdisco/netdisco snmpwalk -v2c -c netdisco 192.168.31.6 system SELECT * FROM sessions WHERE id = 'dancer_session_cookie_key';
-- Insert if missing
INSERT INTO sessions (id, a_session) VALUES ('dancer_session_cookie_key', md5(random()::text));
``` ```
**Status:** [x] COMPLETED (2026-01-19) ### Discovery Not Working
1. Test SNMP from netdisco container:
--- ```bash
docker exec netdisco-backend snmpwalk -v2c -c netdisco 192.168.31.1 system
## Expected Outcomes ```
2. Check logs: `tail -f /mnt/user/appdata/netdisco/logs/netdisco-daemon.log`
| Metric | Before | After | 3. Verify community and IP restrictions on MikroTik devices
|--------|--------|-------|
| Device discovery | ARP scan only | SNMP + ARP |
| Switch port visibility | None | Full mapping |
| MAC tracking | Basic | Full history |
| NetBox data source | Manual | Auto-sync |
| L2 topology | Unknown | Mapped |
---
## Resource Requirements
| Resource | Estimate |
|----------|----------|
| Memory | ~200-400 MiB |
| CPU | Low (polling-based) |
| Disk | ~500 MiB (logs + DB) |
| Database | Reuse postgresql17 |
---
## Rollback Plan
If issues occur:
1. Stop NetDisco container
2. SNMP can remain enabled (read-only, no risk)
3. NetAlertX continues to function
4. Remove Traefik route
--- ---
## References ## References
- [NetDisco Documentation](https://netdisco.org/) - [NetDisco Documentation](https://netdisco.org/)
- [NetDisco API Wiki](https://github.com/netdisco/netdisco/wiki/API) - [NetDisco GitHub Issues](https://github.com/netdisco/netdisco-docker/issues)
- [MikroTik SNMP Documentation](https://help.mikrotik.com/docs/spaces/ROS/pages/8978519/SNMP) - [MikroTik SNMP Documentation](https://help.mikrotik.com/docs/spaces/ROS/pages/8978519/SNMP)
- [CSS326 SwOS SNMP](https://help.mikrotik.com/docs/spaces/SWOS/pages/76415036/CRS3xx+and+CSS326-24G-2S+series+Manual)
- [NetBox Plugin for NetDisco](https://github.com/mksoska/netbox-plugin-netdisco) - [NetBox Plugin for NetDisco](https://github.com/mksoska/netbox-plugin-netdisco)
- [pynetbox Library](https://github.com/netbox-community/pynetbox)