163 lines
4.7 KiB
Markdown
163 lines
4.7 KiB
Markdown
# Incident: DNS Outbound Blocked After MikroTik Restart
|
|
|
|
**Date:** 2026-01-25
|
|
**Duration:** ~1 hour
|
|
**Severity:** High (Complete DNS failure)
|
|
**Status:** Resolved
|
|
|
|
---
|
|
|
|
## Summary
|
|
|
|
After MikroTik router restart, AdGuardHome could not reach upstream DNS servers, causing complete DNS resolution failure for all network clients.
|
|
|
|
## Symptoms
|
|
|
|
- AdGuardHome container showing timeout errors to upstream DNS (8.8.8.8, Quad9)
|
|
- All DNS queries failing network-wide
|
|
- ICMP (ping) to external IPs worked, but port 53 (DNS) was unreachable
|
|
- Error pattern: `i/o timeout` when querying any external DNS server
|
|
|
|
## Root Cause
|
|
|
|
MikroTik NAT rules were redirecting ALL DNS traffic from `192.168.31.0/24` to AdGuardHome, including AdGuardHome's own outbound DNS queries. This created a loop where:
|
|
|
|
1. AdGuardHome (192.168.31.4) tries to query 8.8.8.8:53
|
|
2. MikroTik NAT rule 7 intercepts this traffic (src: 192.168.31.0/24, dst-port: 53)
|
|
3. Traffic redirected back to AdGuardHome (172.17.0.5:5355)
|
|
4. Query fails with timeout
|
|
|
|
**Problematic NAT Rules:**
|
|
```
|
|
Rule 7: Force DNS to AdGuard Home
|
|
chain=dstnat action=dst-nat to-addresses=172.17.0.5 to-ports=5355
|
|
protocol=udp src-address=192.168.31.0/24 dst-port=53
|
|
|
|
Rule 9: Force DNS to AdGuard Home TCP
|
|
chain=dstnat action=dst-nat to-addresses=172.17.0.5 to-ports=5355
|
|
protocol=tcp src-address=192.168.31.0/24 dst-port=53
|
|
```
|
|
|
|
## Resolution
|
|
|
|
Added exception rules BEFORE the redirect rules to allow AdGuardHome to reach external DNS:
|
|
|
|
```bash
|
|
# Added via SSH to MikroTik
|
|
/ip firewall nat add chain=dstnat protocol=udp src-address=192.168.31.4 dst-port=53 action=accept comment=Allow AdGuard outbound DNS place-before=7
|
|
/ip firewall nat add chain=dstnat protocol=tcp src-address=192.168.31.4 dst-port=53 action=accept comment=Allow AdGuard outbound DNS TCP place-before=8
|
|
```
|
|
|
|
**Additional Changes:**
|
|
1. Set MikroTik DNS to use only AdGuard: `/ip dns set servers=192.168.31.4`
|
|
2. Disabled ISP DNS from DHCP: `/ip dhcp-client set 0 use-peer-dns=no`
|
|
3. Updated Unraid resolv.conf to use 192.168.31.4 (for Tailscale access)
|
|
|
|
## Final NAT Rule Order (DNS-related)
|
|
|
|
| # | Comment | Action | Details |
|
|
|---|---------|--------|---------|
|
|
| 5 | Allow AdGuard outbound DNS | accept | src=192.168.31.4, udp/53 |
|
|
| 6 | Force DNS to AdGuard Home | dst-nat | src=192.168.31.0/24 → 172.17.0.5:5355 |
|
|
| 7 | Allow AdGuard outbound DNS TCP | accept | src=192.168.31.4, tcp/53 |
|
|
| 9 | Force DNS to AdGuard Home TCP | dst-nat | src=192.168.31.0/24 → 172.17.0.5:5355 |
|
|
|
|
## Verification
|
|
|
|
After fix:
|
|
- ✓ AdGuard can resolve external DNS (google.com, github.com)
|
|
- ✓ MikroTik using only AdGuard as DNS
|
|
- ✓ Internal domains resolving (git.xtrm-lab.org, cloud.xtrm-lab.org)
|
|
- ✓ External access to services working (Gitea, Woodpecker CI)
|
|
|
|
## Lessons Learned
|
|
|
|
1. NAT redirect rules need exceptions for the DNS server itself
|
|
2. After MikroTik restart, verify DNS flow end-to-end
|
|
3. Keep exception rules BEFORE redirect rules in firewall
|
|
|
|
## Related Configuration
|
|
|
|
- AdGuard IP: 192.168.31.4 (macvlan)
|
|
- AdGuard internal: 172.17.0.5:5355
|
|
- MikroTik: 192.168.31.1
|
|
- Upstream DNS: 8.8.8.8 (temporarily, was Quad9 DoH)
|
|
|
|
---
|
|
|
|
*Incident logged by Claude Code via Tailscale connection*
|
|
|
|
---
|
|
|
|
## Additional Issue: Devices No Internet (13:25)
|
|
|
|
### Symptom
|
|
- LG TV and other devices showing no internet connection
|
|
- DNS changes not reaching devices
|
|
|
|
### Root Cause
|
|
DHCP was pushing as DNS to clients, but MikroTik NAT rules redirect all port 53 traffic to AdGuard. This created a mismatch.
|
|
|
|
**Before:**
|
|
```
|
|
DHCP DNS-SERVER: 8.8.8.8
|
|
```
|
|
|
|
### Fix
|
|
Changed DHCP to push MikroTik as DNS (which forwards to AdGuard):
|
|
|
|
```bash
|
|
/ip dhcp-server network set 0 dns-server=192.168.31.1
|
|
```
|
|
|
|
**After:**
|
|
```
|
|
DHCP DNS-SERVER: 192.168.31.1
|
|
```
|
|
|
|
### DNS Flow (Corrected)
|
|
```
|
|
Device → 192.168.31.1 (MikroTik DNS) → 192.168.31.4 (AdGuard) → 8.8.8.8 (upstream)
|
|
```
|
|
|
|
### Device Recovery
|
|
Devices need to renew DHCP lease to get new DNS:
|
|
- Wait for lease expiry (default 10 min)
|
|
- Reconnect to WiFi
|
|
- Reboot device
|
|
|
|
---
|
|
|
|
## Additional Issue: NAT Redirect Wrong IP/Port (13:35)
|
|
|
|
### Symptom
|
|
- TV showing DNS 192.168.31.1 but no internet
|
|
- DNS queries to MikroTik timing out
|
|
|
|
### Root Cause
|
|
NAT rules were redirecting DNS to wrong destination:
|
|
|
|
**Before (WRONG):**
|
|
```
|
|
to-addresses=172.17.0.5 to-ports=5355
|
|
```
|
|
|
|
But AdGuard:
|
|
- Is on macvlan IP: 192.168.31.4 (NOT 172.17.0.5)
|
|
- Listens on port: 53 (NOT 5355)
|
|
|
|
### Fix
|
|
```bash
|
|
/ip firewall nat set [find comment="Force DNS to AdGuard Home"] to-addresses=192.168.31.4 to-ports=53
|
|
/ip firewall nat set [find comment="Force DNS to AdGuard Home TCP"] to-addresses=192.168.31.4 to-ports=53
|
|
```
|
|
|
|
**After (CORRECT):**
|
|
```
|
|
to-addresses=192.168.31.4 to-ports=53
|
|
```
|
|
|
|
### Verification
|
|
- AdGuard container querying 192.168.31.1 → SUCCESS
|
|
- MikroTik resolve command → SUCCESS
|