226 lines
6.3 KiB
Markdown
226 lines
6.3 KiB
Markdown
# Incident: DNS Outbound Blocked After MikroTik Restart
|
|
|
|
**Date:** 2026-01-25
|
|
**Duration:** ~1 hour
|
|
**Severity:** High (Complete DNS failure)
|
|
**Status:** Resolved
|
|
|
|
---
|
|
|
|
## Summary
|
|
|
|
After MikroTik router restart, AdGuardHome could not reach upstream DNS servers, causing complete DNS resolution failure for all network clients.
|
|
|
|
## Symptoms
|
|
|
|
- AdGuardHome container showing timeout errors to upstream DNS (8.8.8.8, Quad9)
|
|
- All DNS queries failing network-wide
|
|
- ICMP (ping) to external IPs worked, but port 53 (DNS) was unreachable
|
|
- Error pattern: `i/o timeout` when querying any external DNS server
|
|
|
|
## Root Cause
|
|
|
|
MikroTik NAT rules were redirecting ALL DNS traffic from `192.168.31.0/24` to AdGuardHome, including AdGuardHome's own outbound DNS queries. This created a loop where:
|
|
|
|
1. AdGuardHome (192.168.31.4) tries to query 8.8.8.8:53
|
|
2. MikroTik NAT rule 7 intercepts this traffic (src: 192.168.31.0/24, dst-port: 53)
|
|
3. Traffic redirected back to AdGuardHome (172.17.0.5:5355)
|
|
4. Query fails with timeout
|
|
|
|
**Problematic NAT Rules:**
|
|
```
|
|
Rule 7: Force DNS to AdGuard Home
|
|
chain=dstnat action=dst-nat to-addresses=172.17.0.5 to-ports=5355
|
|
protocol=udp src-address=192.168.31.0/24 dst-port=53
|
|
|
|
Rule 9: Force DNS to AdGuard Home TCP
|
|
chain=dstnat action=dst-nat to-addresses=172.17.0.5 to-ports=5355
|
|
protocol=tcp src-address=192.168.31.0/24 dst-port=53
|
|
```
|
|
|
|
## Resolution
|
|
|
|
Added exception rules BEFORE the redirect rules to allow AdGuardHome to reach external DNS:
|
|
|
|
```bash
|
|
# Added via SSH to MikroTik
|
|
/ip firewall nat add chain=dstnat protocol=udp src-address=192.168.31.4 dst-port=53 action=accept comment=Allow AdGuard outbound DNS place-before=7
|
|
/ip firewall nat add chain=dstnat protocol=tcp src-address=192.168.31.4 dst-port=53 action=accept comment=Allow AdGuard outbound DNS TCP place-before=8
|
|
```
|
|
|
|
**Additional Changes:**
|
|
1. Set MikroTik DNS to use only AdGuard: `/ip dns set servers=192.168.31.4`
|
|
2. Disabled ISP DNS from DHCP: `/ip dhcp-client set 0 use-peer-dns=no`
|
|
3. Updated Unraid resolv.conf to use 192.168.31.4 (for Tailscale access)
|
|
|
|
## Final NAT Rule Order (DNS-related)
|
|
|
|
| # | Comment | Action | Details |
|
|
|---|---------|--------|---------|
|
|
| 5 | Allow AdGuard outbound DNS | accept | src=192.168.31.4, udp/53 |
|
|
| 6 | Force DNS to AdGuard Home | dst-nat | src=192.168.31.0/24 → 172.17.0.5:5355 |
|
|
| 7 | Allow AdGuard outbound DNS TCP | accept | src=192.168.31.4, tcp/53 |
|
|
| 9 | Force DNS to AdGuard Home TCP | dst-nat | src=192.168.31.0/24 → 172.17.0.5:5355 |
|
|
|
|
## Verification
|
|
|
|
After fix:
|
|
- ✓ AdGuard can resolve external DNS (google.com, github.com)
|
|
- ✓ MikroTik using only AdGuard as DNS
|
|
- ✓ Internal domains resolving (git.xtrm-lab.org, cloud.xtrm-lab.org)
|
|
- ✓ External access to services working (Gitea, Woodpecker CI)
|
|
|
|
## Lessons Learned
|
|
|
|
1. NAT redirect rules need exceptions for the DNS server itself
|
|
2. After MikroTik restart, verify DNS flow end-to-end
|
|
3. Keep exception rules BEFORE redirect rules in firewall
|
|
|
|
## Related Configuration
|
|
|
|
- AdGuard IP: 192.168.31.4 (macvlan)
|
|
- AdGuard internal: 172.17.0.5:5355
|
|
- MikroTik: 192.168.31.1
|
|
- Upstream DNS: 8.8.8.8 (temporarily, was Quad9 DoH)
|
|
|
|
---
|
|
|
|
*Incident logged by Claude Code via Tailscale connection*
|
|
|
|
---
|
|
|
|
## Additional Issue: Devices No Internet (13:25)
|
|
|
|
### Symptom
|
|
- LG TV and other devices showing no internet connection
|
|
- DNS changes not reaching devices
|
|
|
|
### Root Cause
|
|
DHCP was pushing as DNS to clients, but MikroTik NAT rules redirect all port 53 traffic to AdGuard. This created a mismatch.
|
|
|
|
**Before:**
|
|
```
|
|
DHCP DNS-SERVER: 8.8.8.8
|
|
```
|
|
|
|
### Fix
|
|
Changed DHCP to push MikroTik as DNS (which forwards to AdGuard):
|
|
|
|
```bash
|
|
/ip dhcp-server network set 0 dns-server=192.168.31.1
|
|
```
|
|
|
|
**After:**
|
|
```
|
|
DHCP DNS-SERVER: 192.168.31.1
|
|
```
|
|
|
|
### DNS Flow (Corrected)
|
|
```
|
|
Device → 192.168.31.1 (MikroTik DNS) → 192.168.31.4 (AdGuard) → 8.8.8.8 (upstream)
|
|
```
|
|
|
|
### Device Recovery
|
|
Devices need to renew DHCP lease to get new DNS:
|
|
- Wait for lease expiry (default 10 min)
|
|
- Reconnect to WiFi
|
|
- Reboot device
|
|
|
|
---
|
|
|
|
## Additional Issue: NAT Redirect Wrong IP/Port (13:35)
|
|
|
|
### Symptom
|
|
- TV showing DNS 192.168.31.1 but no internet
|
|
- DNS queries to MikroTik timing out
|
|
|
|
### Root Cause
|
|
NAT rules were redirecting DNS to wrong destination:
|
|
|
|
**Before (WRONG):**
|
|
```
|
|
to-addresses=172.17.0.5 to-ports=5355
|
|
```
|
|
|
|
But AdGuard:
|
|
- Is on macvlan IP: 192.168.31.4 (NOT 172.17.0.5)
|
|
- Listens on port: 53 (NOT 5355)
|
|
|
|
### Fix
|
|
```bash
|
|
/ip firewall nat set [find comment="Force DNS to AdGuard Home"] to-addresses=192.168.31.4 to-ports=53
|
|
/ip firewall nat set [find comment="Force DNS to AdGuard Home TCP"] to-addresses=192.168.31.4 to-ports=53
|
|
```
|
|
|
|
**After (CORRECT):**
|
|
```
|
|
to-addresses=192.168.31.4 to-ports=53
|
|
```
|
|
|
|
### Verification
|
|
- AdGuard container querying 192.168.31.1 → SUCCESS
|
|
- MikroTik resolve command → SUCCESS
|
|
|
|
---
|
|
|
|
## Additional Issue: Netflix Blocked by AdGuard Filter (13:45)
|
|
|
|
### Symptom
|
|
- TV reports no internet even though DNS is working
|
|
- Netflix app fails to connect
|
|
|
|
### Root Cause
|
|
AdGuard DNS filter (FilterListID:1) was blocking Netflix operational domains:
|
|
- `ichnaea.netflix.com` - Netflix connectivity check
|
|
- `customerevents.netflix.com` - Netflix events
|
|
- `*.logs.netflix.com` - Netflix logging
|
|
|
|
Netflix requires these domains to function, even though they're flagged as tracking.
|
|
|
|
### Fix
|
|
Added whitelist rules to AdGuard:
|
|
```
|
|
@@||ichnaea.netflix.com^
|
|
@@||customerevents.netflix.com^
|
|
@@||logs.netflix.com^
|
|
```
|
|
|
|
### Verification
|
|
Restarted AdGuard, TV should now connect to Netflix.
|
|
|
|
---
|
|
|
|
## Root Cause Found: Asymmetric Routing (13:55)
|
|
|
|
### Problem
|
|
DNS redirect (DNAT) was causing asymmetric routing:
|
|
1. Client sends DNS to 192.168.31.1:53
|
|
2. MikroTik DNATs to 192.168.31.4:53 (AdGuard)
|
|
3. AdGuard responds DIRECTLY to client (same L2 subnet)
|
|
4. Client receives response from .4 but expected it from .1
|
|
5. Client drops response → "no internet"
|
|
|
|
### Fix
|
|
Added srcnat/masquerade rule so AdGuard sees MikroTik as source:
|
|
|
|
```bash
|
|
/ip firewall nat add chain=srcnat action=masquerade protocol=udp \
|
|
src-address=192.168.31.0/24 dst-address=192.168.31.4 dst-port=53 \
|
|
comment="Masquerade DNS to AdGuard"
|
|
```
|
|
|
|
Now the flow is:
|
|
1. Client → 192.168.31.1:53
|
|
2. DNAT → 192.168.31.4:53 (src masqueraded to MikroTik)
|
|
3. AdGuard responds to MikroTik
|
|
4. MikroTik un-NATs response → Client
|
|
5. Client sees response from .1 ✓
|
|
|
|
### Verification
|
|
```
|
|
$ nslookup google.com 192.168.31.1
|
|
Server: 192.168.31.1
|
|
Name: google.com
|
|
Address: 142.250.187.110
|
|
```
|