jazzymc/infrastructure
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Update Phase 8: Tasks 8.1-8.4 completed, document session_cookie_key fix
All checks were successful
ci/woodpecker/push/woodpecker Pipeline was successful
Details

Browse Source
This commit is contained in:
XTRM Admin
2026-01-19 21:28:11 +02:00
parent a4e589b583
commit 1ddfc68192
1 changed files with 152 additions and 254 deletions
Download Patch File Download Diff File Expand all files Collapse all files

406
docs/12-PHASE8-NETDISCO-INTEGRATION.md
View File

@@ -1,8 +1,9 @@
# Phase 8: NetDisco Integration with NetBox
**Status:** IN PROGRESS
**Status:** IN PROGRESS (Tasks 8.1-8.4 Complete)
**Priority:** Medium
**Created:** 2026-01-19
**Last Updated:** 2026-01-19
---
@@ -14,99 +15,60 @@ Deploy NetDisco for enterprise-grade network discovery using SNMP, and integrate
---
## Task Status Summary
| Task | Description | Status |
|------|-------------|--------|
| 8.1 | Enable SNMP on MikroTik Devices | ✅ COMPLETED |
| 8.2 | Deploy NetDisco on Unraid | ✅ COMPLETED |
| 8.3 | Configure NetDisco Discovery | ✅ COMPLETED |
| 8.4 | Traefik Ingress for NetDisco | ✅ COMPLETED |
| 8.5 | NetBox Integration | ⏳ PENDING |
| 8.6 | Testing & Validation | ⏳ PENDING |
---
## Current State
| Tool | Status | Purpose |
|------|--------|---------|
| NetAlertX | Running | ARP/ICMP-based device discovery |
| NetBox | Running | IPAM/DCIM (manual data entry) |
| NetDisco | Not deployed | SNMP-based discovery |
| NetDisco | ✅ Running | SNMP-based discovery |
**Decision:** Keep NetAlertX running in parallel during transition.
---
## Architecture
## Deployment Details
```
┌─────────────────────────────────────────────────────────────┐
│ SNMP-Enabled Devices │
├─────────────────┬─────────────────┬─────────────────────────┤
│ hAP ax³ │ CSS326 │ cAP ac │
│ 192.168.31.1 │ 192.168.31.9 │ 192.168.31.6 │
│ RouterOS 7.x │ SwOS │ RouterOS 7.x │
│ SNMPv2c/v3 │ SNMPv1/v2c │ SNMPv2c/v3 │
└────────┬────────┴────────┬────────┴────────┬────────────────┘
│ │ │
└─────────────────┼─────────────────┘
│ SNMP Polling
▼
┌─────────────────────┐
│ NetDisco │
│ 192.168.31.2:5000 │
│ (Unraid Docker) │
└──────────┬──────────┘
│ API Sync
▼
┌─────────────────────┐
│ NetBox │
│ 192.168.31.2:8090 │
│ netbox-plugin │
└─────────────────────┘
```
### Database
- **Host:** postgresql17
- **Database:** netdisco_db
- **User:** netdisco_user
- **Password:** NetD1sc0_M0stW4nt3d@db
### Containers
| Container | Image | Port | Status |
|-----------|-------|------|--------|
| netdisco-web | netdisco/netdisco:latest-web | 5000 | Running |
| netdisco-backend | netdisco/netdisco:latest-backend | - | Running |
### Access URLs
| URL | Purpose |
|-----|---------|
| https://netdisco.xtrm-lab.org | External access (Authentik SSO) |
| http://192.168.31.2:5000 | Internal direct access |
### Configuration
- **Config Path:** /mnt/user/appdata/netdisco/environments/deployment.yml
- **Logs:** /mnt/user/appdata/netdisco/logs/
---
## Tasks
## Completed Tasks
### Task 8.1: Enable SNMP on MikroTik Devices
#### 8.1.1 hAP ax³ (RouterOS)
```routeros
# SSH to router
ssh -p 2222 xtrm@192.168.31.1
# Enable SNMP
/snmp set enabled=yes contact="admin@xtrm-lab.org" location="XTRM Home Lab"
# Create community (restrict to Unraid only)
/snmp community add name=netdisco addresses=192.168.31.2/32 read-access=yes write-access=no
# Verify
/snmp print
/snmp community print
```
**Status:** [x] COMPLETED (2026-01-19)
#### 8.1.2 CSS326 (SwOS)
1. Access SwOS web UI: http://192.168.31.9/index.html
2. Navigate to **SNMP** tab
3. Configure:
- Enable: Yes
- Community: netdisco
- IP Address Filter: 192.168.31.2
**Note:** SwOS only supports SNMPv1/v2c read-only.
**Status:** [x] COMPLETED (2026-01-19)
#### 8.1.3 cAP ac (RouterOS)
```routeros
# SSH to cAP
ssh -p 2222 xtrm@192.168.31.6
# Enable SNMP (same config as hAP ax³)
/snmp set enabled=yes contact="admin@xtrm-lab.org" location="XTRM Home Lab"
/snmp community add name=netdisco addresses=192.168.31.2/32 read-access=yes write-access=no
```
**Status:** [x] COMPLETED (2026-01-19)
#### SNMP Configuration Applied (2026-01-19)
### Task 8.1: SNMP Configuration (COMPLETED)
| Device | IP | Communities | Access Restriction |
|--------|-----|-------------|-------------------|
@@ -120,242 +82,178 @@ ssh -p 2222 xtrm@192.168.31.6
- Version: SNMPv2c
- Write Access: Disabled
**Test Results:**
```
hAP ax³: SNMPv2-MIB::sysName.0 = STRING: HAPax3
cAP ac: SNMPv2-MIB::sysName.0 = STRING: CAP XL ac
CSS326: SNMPv2-MIB::sysName.0 = STRING: CSS326-24G-2S+
```
### Task 8.2: NetDisco Deployment (COMPLETED)
---
### Task 8.2: Deploy NetDisco on Unraid
**Status:** [x] COMPLETED (2026-01-19)
#### Docker Compose
**Docker Compose:** /mnt/user/appdata/netdisco/docker-compose.yml
```yaml
version: '3'
services:
netdisco:
image: netdisco/netdisco:latest
container_name: netdisco
netdisco-backend:
container_name: netdisco-backend
image: netdisco/netdisco:latest-backend
hostname: netdisco-backend
restart: unless-stopped
ports:
- "5000:5000"
environment:
- NETDISCO_DOMAIN=netdisco.xtrm-lab.org
- NETDISCO_DB_HOST=postgresql17
- NETDISCO_DB_NAME=netdisco
- NETDISCO_DB_USER=netdisco
- NETDISCO_DB_PASS=<secure_password>
- NETDISCO_ADMIN_USER=admin
- NETDISCO_ADMIN_PASS=<secure_password>
volumes:
- /mnt/user/appdata/netdisco/config:/home/netdisco/environments
- /mnt/user/appdata/netdisco/logs:/home/netdisco/logs
- /mnt/user/appdata/netdisco/nd-site-local:/home/netdisco/nd-site-local
- /mnt/user/appdata/netdisco/environments:/home/netdisco/environments
- /mnt/user/appdata/netdisco/logs:/home/netdisco/logs
environment:
NETDISCO_DOMAIN: deployment
NETDISCO_DB_HOST: postgresql17
NETDISCO_DB_PORT: 5432
NETDISCO_DB_NAME: netdisco_db
NETDISCO_DB_USER: netdisco_user
NETDISCO_DB_PASS: NetD1sc0_M0stW4nt3d@db
networks:
- dockerproxy
netdisco-web:
container_name: netdisco-web
image: netdisco/netdisco:latest-web
hostname: netdisco-web
restart: unless-stopped
volumes:
- /mnt/user/appdata/netdisco/nd-site-local:/home/netdisco/nd-site-local
- /mnt/user/appdata/netdisco/environments:/home/netdisco/environments
- /mnt/user/appdata/netdisco/logs:/home/netdisco/logs
environment:
NETDISCO_DOMAIN: deployment
NETDISCO_DB_HOST: postgresql17
NETDISCO_DB_PORT: 5432
NETDISCO_DB_NAME: netdisco_db
NETDISCO_DB_USER: netdisco_user
NETDISCO_DB_PASS: NetD1sc0_M0stW4nt3d@db
PORT: 5000
networks:
- dockerproxy
depends_on:
- postgresql17
networks:
dockerproxy:
external: true
```
#### Database Setup
```bash
# Create NetDisco database in existing PostgreSQL
docker exec -it postgresql17 psql -U postgres -c "CREATE USER netdisco WITH PASSWORD 'secure_password';"
docker exec -it postgresql17 psql -U postgres -c "CREATE DATABASE netdisco OWNER netdisco;"
```
**Status:** [x] COMPLETED (2026-01-19)
---
### Task 8.3: Configure NetDisco Discovery
#### deployment.yml Configuration
### Task 8.3: Discovery Configuration (COMPLETED)
**deployment.yml:**
```yaml
# /mnt/user/appdata/netdisco/config/deployment.yml
snmp_community:
- community: netdisco
read: true
write: false
database:
name: 'netdisco_db'
user: 'netdisco_user'
pass: 'NetD1sc0_M0stW4nt3d@db'
host: 'postgresql17'
device_auth:
- tag: mikrotik
community: netdisco
snmp_version: 2
community_rw:
- netdisco
- public
discover_only:
- 192.168.31.1 # hAP ax³
- 192.168.31.6 # cAP ac
- 192.168.31.9 # CSS326
- 192.168.31.1 # hAP ax³
- 192.168.31.6 # cAP ac
- 192.168.31.9 # CSS326
site_local_files: true
no_auth: true
schedule:
discover: '0 * * * *' # Hourly discovery
macsuck: '15 * * * *' # MAC address table poll
arpnip: '30 * * * *' # ARP table poll
nbtstat: '45 * * * *' # NetBIOS name poll
discover: '0 */2 * * *' # Every 2 hours
macsuck: '15 * * * *' # MAC poll hourly
arpnip: '30 * * * *' # ARP poll hourly
nbtstat: '45 * * * *' # NetBIOS hourly
```
**Status:** [x] COMPLETED (2026-01-19)
**Discovery Results:**
| Device | IP | Status |
|--------|-----|--------|
| hAP ax³ | 192.168.31.1 | ✅ Discovered |
| cAP ac | 192.168.31.6 | ✅ Discovered |
| CSS326 | 192.168.31.9 | ✅ Discovered |
---
### Task 8.4: Traefik Ingress (COMPLETED)
### Task 8.4: Traefik Ingress for NetDisco
Add to Traefik dynamic configuration:
**Traefik Dynamic Config:** /mnt/user/appdata/traefik/dynamic.yml
```yaml
http:
routers:
netdisco:
rule: "Host(`netdisco.xtrm-lab.org`)"
netdisco-secure:
rule: "Host(\`netdisco.xtrm-lab.org\`)"
entryPoints:
- websecure
service: netdisco
- https
middlewares:
- default-headers
- authentik-forward-auth
tls:
certResolver: cloudflare
service: netdisco
services:
netdisco:
loadBalancer:
servers:
- url: "http://netdisco:5000"
- url: "http://netdisco-web:5000"
```
**DNS:** Add netdisco.xtrm-lab.org to Cloudflare
**Important Fix Applied:**
**Status:** [x] COMPLETED (2026-01-19)
The web UI failed to start with error: `The setting session_cookie_key must be defined`
**Root Cause:** When using external PostgreSQL, the `dancer_session_cookie_key` record was missing from the sessions table.
**Fix:**
```sql
INSERT INTO sessions (id, a_session) VALUES ('dancer_session_cookie_key', md5(random()::text));
```
---
## Pending Tasks
### Task 8.5: NetBox Integration
#### Option A: NetBox Plugin (Prototype)
**Option A: NetBox Plugin (Prototype)**
- Repository: https://github.com/mksoska/netbox-plugin-netdisco
- Status: Prototype stage - may have compatibility issues
Repository: https://github.com/mksoska/netbox-plugin-netdisco
```bash
# Install in NetBox container
docker exec -it netbox pip install netbox-plugin-netdisco
# Add to NetBox configuration.py
PLUGINS = ['netbox_netdisco']
PLUGINS_CONFIG = {
'netbox_netdisco': {
'NETDISCO_HOST': 'http://netdisco:5000',
'NETDISCO_USERNAME': 'admin',
'NETDISCO_PASSWORD': '<password>',
}
}
```
**Note:** Plugin is prototype-stage. Test thoroughly.
#### Option B: Custom API Sync Script
Create a sync script using NetDisco API + pynetbox:
```python
#!/usr/bin/env python3
# /mnt/user/appdata/netbox/scripts/netdisco_sync.py
import requests
import pynetbox
NETDISCO_URL = "http://192.168.31.2:5000"
NETBOX_URL = "http://192.168.31.2:8090"
NETBOX_TOKEN = "<api_token>"
nb = pynetbox.api(NETBOX_URL, token=NETBOX_TOKEN)
# Fetch devices from NetDisco
devices = requests.get(f"{NETDISCO_URL}/api/v1/device").json()
for device in devices:
# Check if device exists in NetBox
existing = nb.dcim.devices.get(primary_ip=device['ip'])
if not existing:
print(f"New device found: {device['ip']} ({device['name']})")
# Create or flag for review
```
**Status:** [x] COMPLETED (2026-01-19)
---
**Option B: Custom API Sync Script (Recommended)**
- Use NetDisco API + pynetbox library
- More control over sync behavior
- Can be scheduled via cron
### Task 8.6: Testing & Validation
#### Checklist
- [ ] Verify all SNMP queries working
- [ ] Confirm switch port mapping
- [ ] Test MAC address tracking
- [ ] Validate NetBox sync (once implemented)
- [ ] Test external access via Traefik
- [ ] SNMP reachable from Unraid to all MikroTik devices
- [ ] NetDisco discovers hAP ax³, CSS326, cAP ac
- [ ] Switch port mapping shows connected devices
- [ ] MAC address tracking working
- [ ] NetBox sync shows discovered devices
- [ ] Traefik proxy working (netdisco.xtrm-lab.org)
---
#### SNMP Test Commands
## Troubleshooting
```bash
# Test SNMP from Unraid
docker run --rm --network dockerproxy netdisco/netdisco snmpwalk -v2c -c netdisco 192.168.31.1 system
docker run --rm --network dockerproxy netdisco/netdisco snmpwalk -v2c -c netdisco 192.168.31.9 system
docker run --rm --network dockerproxy netdisco/netdisco snmpwalk -v2c -c netdisco 192.168.31.6 system
### session_cookie_key Error
If web UI shows "session_cookie_key must be defined":
```sql
-- Check if key exists
SELECT * FROM sessions WHERE id = 'dancer_session_cookie_key';
-- Insert if missing
INSERT INTO sessions (id, a_session) VALUES ('dancer_session_cookie_key', md5(random()::text));
```
**Status:** [x] COMPLETED (2026-01-19)
---
## Expected Outcomes
| Metric | Before | After |
|--------|--------|-------|
| Device discovery | ARP scan only | SNMP + ARP |
| Switch port visibility | None | Full mapping |
| MAC tracking | Basic | Full history |
| NetBox data source | Manual | Auto-sync |
| L2 topology | Unknown | Mapped |
---
## Resource Requirements
| Resource | Estimate |
|----------|----------|
| Memory | ~200-400 MiB |
| CPU | Low (polling-based) |
| Disk | ~500 MiB (logs + DB) |
| Database | Reuse postgresql17 |
---
## Rollback Plan
If issues occur:
1. Stop NetDisco container
2. SNMP can remain enabled (read-only, no risk)
3. NetAlertX continues to function
4. Remove Traefik route
### Discovery Not Working
1. Test SNMP from netdisco container:
```bash
docker exec netdisco-backend snmpwalk -v2c -c netdisco 192.168.31.1 system
```
2. Check logs: `tail -f /mnt/user/appdata/netdisco/logs/netdisco-daemon.log`
3. Verify community and IP restrictions on MikroTik devices
---
## References
- [NetDisco Documentation](https://netdisco.org/)
- [NetDisco API Wiki](https://github.com/netdisco/netdisco/wiki/API)
- [NetDisco GitHub Issues](https://github.com/netdisco/netdisco-docker/issues)
- [MikroTik SNMP Documentation](https://help.mikrotik.com/docs/spaces/ROS/pages/8978519/SNMP)
- [CSS326 SwOS SNMP](https://help.mikrotik.com/docs/spaces/SWOS/pages/76415036/CRS3xx+and+CSS326-24G-2S+series+Manual)
- [NetBox Plugin for NetDisco](https://github.com/mksoska/netbox-plugin-netdisco)
- [pynetbox Library](https://github.com/netbox-community/pynetbox)