Files
infrastructure/docs/incidents/2026-01-25-dns-outbound-blocked-after-mikrotik-restart.md

3.1 KiB

Incident: DNS Outbound Blocked After MikroTik Restart

Date: 2026-01-25
Duration: ~1 hour
Severity: High (Complete DNS failure)
Status: Resolved


Summary

After MikroTik router restart, AdGuardHome could not reach upstream DNS servers, causing complete DNS resolution failure for all network clients.

Symptoms

  • AdGuardHome container showing timeout errors to upstream DNS (8.8.8.8, Quad9)
  • All DNS queries failing network-wide
  • ICMP (ping) to external IPs worked, but port 53 (DNS) was unreachable
  • Error pattern: i/o timeout when querying any external DNS server

Root Cause

MikroTik NAT rules were redirecting ALL DNS traffic from 192.168.31.0/24 to AdGuardHome, including AdGuardHome's own outbound DNS queries. This created a loop where:

  1. AdGuardHome (192.168.31.4) tries to query 8.8.8.8:53
  2. MikroTik NAT rule 7 intercepts this traffic (src: 192.168.31.0/24, dst-port: 53)
  3. Traffic redirected back to AdGuardHome (172.17.0.5:5355)
  4. Query fails with timeout

Problematic NAT Rules:

Rule 7: Force DNS to AdGuard Home
  chain=dstnat action=dst-nat to-addresses=172.17.0.5 to-ports=5355 
  protocol=udp src-address=192.168.31.0/24 dst-port=53

Rule 9: Force DNS to AdGuard Home TCP
  chain=dstnat action=dst-nat to-addresses=172.17.0.5 to-ports=5355 
  protocol=tcp src-address=192.168.31.0/24 dst-port=53

Resolution

Added exception rules BEFORE the redirect rules to allow AdGuardHome to reach external DNS:

# Added via SSH to MikroTik
/ip firewall nat add chain=dstnat protocol=udp src-address=192.168.31.4 dst-port=53 action=accept comment=Allow AdGuard outbound DNS place-before=7
/ip firewall nat add chain=dstnat protocol=tcp src-address=192.168.31.4 dst-port=53 action=accept comment=Allow AdGuard outbound DNS TCP place-before=8

Additional Changes:

  1. Set MikroTik DNS to use only AdGuard: /ip dns set servers=192.168.31.4
  2. Disabled ISP DNS from DHCP: /ip dhcp-client set 0 use-peer-dns=no
  3. Updated Unraid resolv.conf to use 192.168.31.4 (for Tailscale access)
# Comment Action Details
5 Allow AdGuard outbound DNS accept src=192.168.31.4, udp/53
6 Force DNS to AdGuard Home dst-nat src=192.168.31.0/24 → 172.17.0.5:5355
7 Allow AdGuard outbound DNS TCP accept src=192.168.31.4, tcp/53
9 Force DNS to AdGuard Home TCP dst-nat src=192.168.31.0/24 → 172.17.0.5:5355

Verification

After fix:

  • ✓ AdGuard can resolve external DNS (google.com, github.com)
  • ✓ MikroTik using only AdGuard as DNS
  • ✓ Internal domains resolving (git.xtrm-lab.org, cloud.xtrm-lab.org)
  • ✓ External access to services working (Gitea, Woodpecker CI)

Lessons Learned

  1. NAT redirect rules need exceptions for the DNS server itself
  2. After MikroTik restart, verify DNS flow end-to-end
  3. Keep exception rules BEFORE redirect rules in firewall
  • AdGuard IP: 192.168.31.4 (macvlan)
  • AdGuard internal: 172.17.0.5:5355
  • MikroTik: 192.168.31.1
  • Upstream DNS: 8.8.8.8 (temporarily, was Quad9 DoH)

Incident logged by Claude Code via Tailscale connection