infrastructure/docs/12-PHASE8-NETDISCO-INTEGRATION.md

# Phase 8: NetDisco Integration with NetBox

**Status:** ✅ COMPLETED
**Priority:** Medium
**Created:** 2026-01-19
**Last Updated:** 2026-01-19

---

## Overview

Deploy NetDisco for enterprise-grade network discovery using SNMP, and integrate it with NetBox for a unified "Fing-like" network management experience.

**Goal:** Automatic device discovery, switch port mapping, and L2/L3 topology tracking synchronized to NetBox IPAM/DCIM.

---

## Task Status Summary

| Task | Description | Status |
|------|-------------|--------|
| 8.1 | Enable SNMP on MikroTik Devices | ✅ COMPLETED |
| 8.2 | Deploy NetDisco on Unraid | ✅ COMPLETED |
| 8.3 | Configure NetDisco Discovery | ✅ COMPLETED |
| 8.4 | Traefik Ingress for NetDisco | ✅ COMPLETED |
| 8.5 | NetBox Integration | ✅ COMPLETED |
| 8.6 | Testing & Validation | ✅ COMPLETED |

---

## Current State

| Tool | Status | Purpose |
|------|--------|---------|
| NetAlertX | Running | ARP/ICMP-based device discovery |
| NetBox | Running | IPAM/DCIM (manual data entry) |
| NetDisco | ✅ Running | SNMP-based discovery |

**Decision:** Keep NetAlertX running in parallel during transition.

---

## Deployment Details

### Database
- **Host:** postgresql17
- **Database:** netdisco_db
- **User:** netdisco_user
- **Password:** NetD1sc0_M0stW4nt3d@db

### Containers
| Container | Image | Port | Status |
|-----------|-------|------|--------|
| netdisco-web | netdisco/netdisco:latest-web | 5000 | Running |
| netdisco-backend | netdisco/netdisco:latest-backend | - | Running |

### Access URLs
| URL | Purpose |
|-----|---------|
| https://netdisco.xtrm-lab.org | External access (Authentik SSO) |
| http://192.168.31.2:5000 | Internal direct access |

### Configuration
- **Config Path:** /mnt/user/appdata/netdisco/environments/deployment.yml
- **Logs:** /mnt/user/appdata/netdisco/logs/

---

## Completed Tasks

### Task 8.1: SNMP Configuration (COMPLETED)

| Device | IP | Communities | Access Restriction |
|--------|-----|-------------|-------------------|
| hAP ax³ | 192.168.31.1 | `netdisco`, `public` | 192.168.31.2/32 |
| cAP ac | 192.168.31.6 | `netdisco`, `public` | 192.168.31.2/32 |
| CSS326 | 192.168.31.9 | `public` | None (SwOS limitation) |

**SNMP Settings:**
- Contact: admin@xtrm-lab.org
- Location: XTRM Home Lab
- Version: SNMPv2c
- Write Access: Disabled

### Task 8.2: NetDisco Deployment (COMPLETED)

**Docker Compose:** /mnt/user/appdata/netdisco/docker-compose.yml

```yaml
services:
  netdisco-backend:
    container_name: netdisco-backend
    image: netdisco/netdisco:latest-backend
    hostname: netdisco-backend
    restart: unless-stopped
    volumes:
      - /mnt/user/appdata/netdisco/nd-site-local:/home/netdisco/nd-site-local
      - /mnt/user/appdata/netdisco/environments:/home/netdisco/environments
      - /mnt/user/appdata/netdisco/logs:/home/netdisco/logs
    environment:
      NETDISCO_DOMAIN: deployment
      NETDISCO_DB_HOST: postgresql17
      NETDISCO_DB_PORT: 5432
      NETDISCO_DB_NAME: netdisco_db
      NETDISCO_DB_USER: netdisco_user
      NETDISCO_DB_PASS: NetD1sc0_M0stW4nt3d@db
    networks:
      - dockerproxy

  netdisco-web:
    container_name: netdisco-web
    image: netdisco/netdisco:latest-web
    hostname: netdisco-web
    restart: unless-stopped
    volumes:
      - /mnt/user/appdata/netdisco/nd-site-local:/home/netdisco/nd-site-local
      - /mnt/user/appdata/netdisco/environments:/home/netdisco/environments
      - /mnt/user/appdata/netdisco/logs:/home/netdisco/logs
    environment:
      NETDISCO_DOMAIN: deployment
      NETDISCO_DB_HOST: postgresql17
      NETDISCO_DB_PORT: 5432
      NETDISCO_DB_NAME: netdisco_db
      NETDISCO_DB_USER: netdisco_user
      NETDISCO_DB_PASS: NetD1sc0_M0stW4nt3d@db
      PORT: 5000
    networks:
      - dockerproxy

networks:
  dockerproxy:
    external: true
```

### Task 8.3: Discovery Configuration (COMPLETED)

**deployment.yml:**
```yaml
database:
  name: 'netdisco_db'
  user: 'netdisco_user'
  pass: 'NetD1sc0_M0stW4nt3d@db'
  host: 'postgresql17'

community_rw:
  - netdisco
  - public

discover_only:
  - 192.168.31.1  # hAP ax³
  - 192.168.31.6  # cAP ac
  - 192.168.31.9  # CSS326

site_local_files: true
no_auth: true

schedule:
  discover: '0 */2 * * *'    # Every 2 hours
  macsuck:  '15 * * * *'     # MAC poll hourly
  arpnip:   '30 * * * *'     # ARP poll hourly
  nbtstat:  '45 * * * *'     # NetBIOS hourly
```

**Discovery Results:**
| Device | IP | Status |
|--------|-----|--------|
| hAP ax³ | 192.168.31.1 | ✅ Discovered |
| cAP ac | 192.168.31.6 | ✅ Discovered |
| CSS326 | 192.168.31.9 | ✅ Discovered |

### Task 8.4: Traefik Ingress (COMPLETED)

**Traefik Dynamic Config:** /mnt/user/appdata/traefik/dynamic.yml

```yaml
http:
  routers:
    netdisco-secure:
      rule: "Host(\`netdisco.xtrm-lab.org\`)"
      entryPoints:
        - https
      middlewares:
        - default-headers
        - authentik-forward-auth
      tls:
        certResolver: cloudflare
      service: netdisco

  services:
    netdisco:
      loadBalancer:
        servers:
          - url: "http://netdisco-web:5000"
```

**Important Fix Applied:**

The web UI failed to start with error: `The setting session_cookie_key must be defined`

**Root Cause:** When using external PostgreSQL, the `dancer_session_cookie_key` record was missing from the sessions table.

**Fix:**
```sql
INSERT INTO sessions (id, a_session) VALUES ('dancer_session_cookie_key', md5(random()::text));
```

---

## Pending Tasks

### Task 8.5: NetBox Integration

**Option A: NetBox Plugin (Prototype)**
- Repository: https://github.com/mksoska/netbox-plugin-netdisco
- Status: Prototype stage - may have compatibility issues

**Option B: Custom API Sync Script (Recommended)**
- Use NetDisco API + pynetbox library
- More control over sync behavior
- Can be scheduled via cron

### Task 8.6: Testing & Validation

- [ ] Verify all SNMP queries working
- [ ] Confirm switch port mapping
- [ ] Test MAC address tracking
- [ ] Validate NetBox sync (once implemented)
- [ ] Test external access via Traefik

---

## Troubleshooting

### session_cookie_key Error
If web UI shows "session_cookie_key must be defined":
```sql
-- Check if key exists
SELECT * FROM sessions WHERE id = 'dancer_session_cookie_key';

-- Insert if missing
INSERT INTO sessions (id, a_session) VALUES ('dancer_session_cookie_key', md5(random()::text));
```

### Discovery Not Working
1. Test SNMP from netdisco container:
   ```bash
   docker exec netdisco-backend snmpwalk -v2c -c netdisco 192.168.31.1 system
   ```
2. Check logs: `tail -f /mnt/user/appdata/netdisco/logs/netdisco-daemon.log`
3. Verify community and IP restrictions on MikroTik devices

---

## References

- [NetDisco Documentation](https://netdisco.org/)
- [NetDisco GitHub Issues](https://github.com/netdisco/netdisco-docker/issues)
- [MikroTik SNMP Documentation](https://help.mikrotik.com/docs/spaces/ROS/pages/8978519/SNMP)
- [NetBox Plugin for NetDisco](https://github.com/mksoska/netbox-plugin-netdisco)

### Task 8.5: NetBox Integration (COMPLETED)

**Implementation: Custom Sync Script**

The prototype NetBox plugin for NetDisco was not production-ready, so a custom Python sync script was created.

**Script Location:** `/mnt/user/appdata/netdisco/scripts/sync_to_netbox.py`

**What It Does:**
1. Reads device data from NetDisco PostgreSQL database
2. Matches devices in NetBox by name
3. Updates NetBox device comments with NetDisco data:
   - IP address
   - OS version
   - Model
   - Vendor
   - Last SNMP discovery timestamp

**NetBox API Token:**
- Token: `kI2HiUD8Bb53KfdkOTaiWYxZuERjhGU30CgGdYxm`
- Type: v1 (plaintext)
- User: admin
- Permissions: Write-enabled

**Running the Sync:**
```bash
docker run --rm --network dockerproxy \
  -v /mnt/user/appdata/netdisco/scripts:/scripts \
  python:3.11-slim sh -c \
  'pip install -q psycopg2-binary requests && python /scripts/sync_to_netbox.py'
```

**Sync Results:**
| NetBox Device | NetDisco Device | IP | Status |
|---------------|-----------------|-----|--------|
| HAPax3 | HAPax3 | 192.168.31.1 | ✅ Synced |
| CSS326 | CSS326-24G-2S+ | 192.168.31.9 | ✅ Synced |
| CAP-XL-ac | CAP XL ac | 192.168.31.6 | ✅ Synced |

**Scheduling (Optional):**
To run automatically, add to cron:
```bash
# Run sync hourly at :45
45 * * * * docker run --rm --network dockerproxy -v /mnt/user/appdata/netdisco/scripts:/scripts python:3.11-slim sh -c 'pip install -q psycopg2-binary requests 2>/dev/null && python /scripts/sync_to_netbox.py' >> /mnt/user/appdata/netdisco/logs/sync.log 2>&1
```


### Task 8.6: Testing & Validation (COMPLETED)

**Verification Results:**

| Test | Result |
|------|--------|
| SNMP to hAP ax³ (192.168.31.1) | ✅ Responding |
| SNMP to cAP ac (192.168.31.6) | ✅ Responding |
| SNMP to CSS326 (192.168.31.9) | ✅ Responding |
| Device discovery | ✅ 3 devices discovered |
| Port mapping | ✅ 52 ports collected |
| MAC address tracking | ✅ 19 nodes tracked |
| NetBox sync | ✅ All 3 devices synced |
| Internal access (port 5000) | ✅ HTTP 200 |
| External access (Traefik) | ✅ SSO redirect working |

**Data Collected:**
- 3 network devices (MikroTik)
- 52 switch/router ports
- 19 MAC addresses (nodes)
- Automatic discovery every 2 hours
- MAC polling hourly

---

## Phase 8 Summary

**Status: ✅ COMPLETED**

All tasks in Phase 8 have been successfully completed:
- SNMP enabled on all MikroTik devices
- NetDisco deployed and discovering network
- Traefik ingress with Authentik SSO configured
- NetBox integration via custom sync script
- All functionality verified

**Access URLs:**
- NetDisco Web: https://netdisco.xtrm-lab.org (with SSO)
- NetDisco Internal: http://192.168.31.2:5000

**Maintenance:**
- Discovery runs automatically every 2 hours
- MAC/ARP polling runs hourly
- NetBox sync can be run manually or scheduled via cron