jazzymc/infrastructure
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add incident report: DNS outbound blocked after MikroTik restart (2026-01-25)

Browse Source
This commit is contained in:
XTRM-Unraid
2026-01-25 13:17:04 +02:00
parent de42738775
commit 776372a0b9
1 changed files with 88 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

88
docs/incidents/2026-01-25-dns-outbound-blocked-after-mikrotik-restart.md Normal file
View File

@@ -0,0 +1,88 @@
# Incident: DNS Outbound Blocked After MikroTik Restart
**Date:** 2026-01-25
**Duration:** ~1 hour
**Severity:** High (Complete DNS failure)
**Status:** Resolved
---
## Summary
After MikroTik router restart, AdGuardHome could not reach upstream DNS servers, causing complete DNS resolution failure for all network clients.
## Symptoms
- AdGuardHome container showing timeout errors to upstream DNS (8.8.8.8, Quad9)
- All DNS queries failing network-wide
- ICMP (ping) to external IPs worked, but port 53 (DNS) was unreachable
- Error pattern: `i/o timeout` when querying any external DNS server
## Root Cause
MikroTik NAT rules were redirecting ALL DNS traffic from `192.168.31.0/24` to AdGuardHome, including AdGuardHome's own outbound DNS queries. This created a loop where:
1. AdGuardHome (192.168.31.4) tries to query 8.8.8.8:53
2. MikroTik NAT rule 7 intercepts this traffic (src: 192.168.31.0/24, dst-port: 53)
3. Traffic redirected back to AdGuardHome (172.17.0.5:5355)
4. Query fails with timeout
**Problematic NAT Rules:**
```
Rule 7: Force DNS to AdGuard Home
chain=dstnat action=dst-nat to-addresses=172.17.0.5 to-ports=5355
protocol=udp src-address=192.168.31.0/24 dst-port=53
Rule 9: Force DNS to AdGuard Home TCP
chain=dstnat action=dst-nat to-addresses=172.17.0.5 to-ports=5355
protocol=tcp src-address=192.168.31.0/24 dst-port=53
```
## Resolution
Added exception rules BEFORE the redirect rules to allow AdGuardHome to reach external DNS:
```bash
# Added via SSH to MikroTik
/ip firewall nat add chain=dstnat protocol=udp src-address=192.168.31.4 dst-port=53 action=accept comment=Allow AdGuard outbound DNS place-before=7
/ip firewall nat add chain=dstnat protocol=tcp src-address=192.168.31.4 dst-port=53 action=accept comment=Allow AdGuard outbound DNS TCP place-before=8
```
**Additional Changes:**
1. Set MikroTik DNS to use only AdGuard: `/ip dns set servers=192.168.31.4`
2. Disabled ISP DNS from DHCP: `/ip dhcp-client set 0 use-peer-dns=no`
3. Updated Unraid resolv.conf to use 192.168.31.4 (for Tailscale access)
## Final NAT Rule Order (DNS-related)
| # | Comment | Action | Details |
|---|---------|--------|---------|
| 5 | Allow AdGuard outbound DNS | accept | src=192.168.31.4, udp/53 |
| 6 | Force DNS to AdGuard Home | dst-nat | src=192.168.31.0/24 → 172.17.0.5:5355 |
| 7 | Allow AdGuard outbound DNS TCP | accept | src=192.168.31.4, tcp/53 |
| 9 | Force DNS to AdGuard Home TCP | dst-nat | src=192.168.31.0/24 → 172.17.0.5:5355 |
## Verification
After fix:
- ✓ AdGuard can resolve external DNS (google.com, github.com)
- ✓ MikroTik using only AdGuard as DNS
- ✓ Internal domains resolving (git.xtrm-lab.org, cloud.xtrm-lab.org)
- ✓ External access to services working (Gitea, Woodpecker CI)
## Lessons Learned
1. NAT redirect rules need exceptions for the DNS server itself
2. After MikroTik restart, verify DNS flow end-to-end
3. Keep exception rules BEFORE redirect rules in firewall
## Related Configuration
- AdGuard IP: 192.168.31.4 (macvlan)
- AdGuard internal: 172.17.0.5:5355
- MikroTik: 192.168.31.1
- Upstream DNS: 8.8.8.8 (temporarily, was Quad9 DoH)
---
*Incident logged by Claude Code via Tailscale connection*