From 776372a0b9329de73e458a902dbb2d1ec98a118c Mon Sep 17 00:00:00 2001 From: XTRM-Unraid Date: Sun, 25 Jan 2026 13:17:04 +0200 Subject: [PATCH] Add incident report: DNS outbound blocked after MikroTik restart (2026-01-25) --- ...outbound-blocked-after-mikrotik-restart.md | 88 +++++++++++++++++++ 1 file changed, 88 insertions(+) create mode 100644 docs/incidents/2026-01-25-dns-outbound-blocked-after-mikrotik-restart.md diff --git a/docs/incidents/2026-01-25-dns-outbound-blocked-after-mikrotik-restart.md b/docs/incidents/2026-01-25-dns-outbound-blocked-after-mikrotik-restart.md new file mode 100644 index 0000000..fc56b6b --- /dev/null +++ b/docs/incidents/2026-01-25-dns-outbound-blocked-after-mikrotik-restart.md @@ -0,0 +1,88 @@ +# Incident: DNS Outbound Blocked After MikroTik Restart + +**Date:** 2026-01-25 +**Duration:** ~1 hour +**Severity:** High (Complete DNS failure) +**Status:** Resolved + +--- + +## Summary + +After MikroTik router restart, AdGuardHome could not reach upstream DNS servers, causing complete DNS resolution failure for all network clients. + +## Symptoms + +- AdGuardHome container showing timeout errors to upstream DNS (8.8.8.8, Quad9) +- All DNS queries failing network-wide +- ICMP (ping) to external IPs worked, but port 53 (DNS) was unreachable +- Error pattern: `i/o timeout` when querying any external DNS server + +## Root Cause + +MikroTik NAT rules were redirecting ALL DNS traffic from `192.168.31.0/24` to AdGuardHome, including AdGuardHome's own outbound DNS queries. This created a loop where: + +1. AdGuardHome (192.168.31.4) tries to query 8.8.8.8:53 +2. MikroTik NAT rule 7 intercepts this traffic (src: 192.168.31.0/24, dst-port: 53) +3. Traffic redirected back to AdGuardHome (172.17.0.5:5355) +4. Query fails with timeout + +**Problematic NAT Rules:** +``` +Rule 7: Force DNS to AdGuard Home + chain=dstnat action=dst-nat to-addresses=172.17.0.5 to-ports=5355 + protocol=udp src-address=192.168.31.0/24 dst-port=53 + +Rule 9: Force DNS to AdGuard Home TCP + chain=dstnat action=dst-nat to-addresses=172.17.0.5 to-ports=5355 + protocol=tcp src-address=192.168.31.0/24 dst-port=53 +``` + +## Resolution + +Added exception rules BEFORE the redirect rules to allow AdGuardHome to reach external DNS: + +```bash +# Added via SSH to MikroTik +/ip firewall nat add chain=dstnat protocol=udp src-address=192.168.31.4 dst-port=53 action=accept comment=Allow AdGuard outbound DNS place-before=7 +/ip firewall nat add chain=dstnat protocol=tcp src-address=192.168.31.4 dst-port=53 action=accept comment=Allow AdGuard outbound DNS TCP place-before=8 +``` + +**Additional Changes:** +1. Set MikroTik DNS to use only AdGuard: `/ip dns set servers=192.168.31.4` +2. Disabled ISP DNS from DHCP: `/ip dhcp-client set 0 use-peer-dns=no` +3. Updated Unraid resolv.conf to use 192.168.31.4 (for Tailscale access) + +## Final NAT Rule Order (DNS-related) + +| # | Comment | Action | Details | +|---|---------|--------|---------| +| 5 | Allow AdGuard outbound DNS | accept | src=192.168.31.4, udp/53 | +| 6 | Force DNS to AdGuard Home | dst-nat | src=192.168.31.0/24 → 172.17.0.5:5355 | +| 7 | Allow AdGuard outbound DNS TCP | accept | src=192.168.31.4, tcp/53 | +| 9 | Force DNS to AdGuard Home TCP | dst-nat | src=192.168.31.0/24 → 172.17.0.5:5355 | + +## Verification + +After fix: +- ✓ AdGuard can resolve external DNS (google.com, github.com) +- ✓ MikroTik using only AdGuard as DNS +- ✓ Internal domains resolving (git.xtrm-lab.org, cloud.xtrm-lab.org) +- ✓ External access to services working (Gitea, Woodpecker CI) + +## Lessons Learned + +1. NAT redirect rules need exceptions for the DNS server itself +2. After MikroTik restart, verify DNS flow end-to-end +3. Keep exception rules BEFORE redirect rules in firewall + +## Related Configuration + +- AdGuard IP: 192.168.31.4 (macvlan) +- AdGuard internal: 172.17.0.5:5355 +- MikroTik: 192.168.31.1 +- Upstream DNS: 8.8.8.8 (temporarily, was Quad9 DoH) + +--- + +*Incident logged by Claude Code via Tailscale connection*