infrastructure/docs/incidents/2026-01-25-dns-outbound-blocked-after-mikrotik-restart.md

Incident: DNS Outbound Blocked After MikroTik Restart

Date: 2026-01-25
Duration: ~1 hour
Severity: High (Complete DNS failure)
Status: Resolved

---

Summary

After MikroTik router restart, AdGuardHome could not reach upstream DNS servers, causing complete DNS resolution failure for all network clients.

Symptoms

• AdGuardHome container showing timeout errors to upstream DNS (8.8.8.8, Quad9)
• All DNS queries failing network-wide
• ICMP (ping) to external IPs worked, but port 53 (DNS) was unreachable
• Error pattern: i/o timeout when querying any external DNS server

Root Cause

MikroTik NAT rules were redirecting ALL DNS traffic from 192.168.31.0/24 to AdGuardHome, including AdGuardHome's own outbound DNS queries. This created a loop where:

1. AdGuardHome (192.168.31.4) tries to query 8.8.8.8:53
2. MikroTik NAT rule 7 intercepts this traffic (src: 192.168.31.0/24, dst-port: 53)
3. Traffic redirected back to AdGuardHome (172.17.0.5:5355)
4. Query fails with timeout

Problematic NAT Rules:

Rule 7: Force DNS to AdGuard Home
  chain=dstnat action=dst-nat to-addresses=172.17.0.5 to-ports=5355 
  protocol=udp src-address=192.168.31.0/24 dst-port=53

Rule 9: Force DNS to AdGuard Home TCP
  chain=dstnat action=dst-nat to-addresses=172.17.0.5 to-ports=5355 
  protocol=tcp src-address=192.168.31.0/24 dst-port=53

Resolution

Added exception rules BEFORE the redirect rules to allow AdGuardHome to reach external DNS:

# Added via SSH to MikroTik
/ip firewall nat add chain=dstnat protocol=udp src-address=192.168.31.4 dst-port=53 action=accept comment=Allow AdGuard outbound DNS place-before=7
/ip firewall nat add chain=dstnat protocol=tcp src-address=192.168.31.4 dst-port=53 action=accept comment=Allow AdGuard outbound DNS TCP place-before=8

Additional Changes:

1. Set MikroTik DNS to use only AdGuard: /ip dns set servers=192.168.31.4
2. Disabled ISP DNS from DHCP: /ip dhcp-client set 0 use-peer-dns=no
3. Updated Unraid resolv.conf to use 192.168.31.4 (for Tailscale access)

Final NAT Rule Order (DNS-related)

#	Comment	Action	Details
5	Allow AdGuard outbound DNS	accept	src=192.168.31.4, udp/53
6	Force DNS to AdGuard Home	dst-nat	src=192.168.31.0/24 → 172.17.0.5:5355
7	Allow AdGuard outbound DNS TCP	accept	src=192.168.31.4, tcp/53
9	Force DNS to AdGuard Home TCP	dst-nat	src=192.168.31.0/24 → 172.17.0.5:5355

Verification

After fix:

• ✓ AdGuard can resolve external DNS (google.com, github.com)
• ✓ MikroTik using only AdGuard as DNS
• ✓ Internal domains resolving (git.xtrm-lab.org, cloud.xtrm-lab.org)
• ✓ External access to services working (Gitea, Woodpecker CI)

Lessons Learned

1. NAT redirect rules need exceptions for the DNS server itself
2. After MikroTik restart, verify DNS flow end-to-end
3. Keep exception rules BEFORE redirect rules in firewall

Related Configuration

• AdGuard IP: 192.168.31.4 (macvlan)
• AdGuard internal: 172.17.0.5:5355
• MikroTik: 192.168.31.1
• Upstream DNS: 8.8.8.8 (temporarily, was Quad9 DoH)

---

Incident logged by Claude Code via Tailscale connection

---

Additional Issue: Devices No Internet (13:25)

Symptom

• LG TV and other devices showing no internet connection
• DNS changes not reaching devices

Root Cause

DHCP was pushing as DNS to clients, but MikroTik NAT rules redirect all port 53 traffic to AdGuard. This created a mismatch.

Before:

DHCP DNS-SERVER: 8.8.8.8

Fix

Changed DHCP to push MikroTik as DNS (which forwards to AdGuard):

/ip dhcp-server network set 0 dns-server=192.168.31.1

After:

DHCP DNS-SERVER: 192.168.31.1

DNS Flow (Corrected)

Device → 192.168.31.1 (MikroTik DNS) → 192.168.31.4 (AdGuard) → 8.8.8.8 (upstream)

Device Recovery

Devices need to renew DHCP lease to get new DNS:

• Wait for lease expiry (default 10 min)
• Reconnect to WiFi
• Reboot device

---

Additional Issue: NAT Redirect Wrong IP/Port (13:35)

Symptom

• TV showing DNS 192.168.31.1 but no internet
• DNS queries to MikroTik timing out

Root Cause

NAT rules were redirecting DNS to wrong destination:

Before (WRONG):

to-addresses=172.17.0.5 to-ports=5355

But AdGuard:

• Is on macvlan IP: 192.168.31.4 (NOT 172.17.0.5)
• Listens on port: 53 (NOT 5355)

Fix

/ip firewall nat set [find comment="Force DNS to AdGuard Home"] to-addresses=192.168.31.4 to-ports=53
/ip firewall nat set [find comment="Force DNS to AdGuard Home TCP"] to-addresses=192.168.31.4 to-ports=53

After (CORRECT):

to-addresses=192.168.31.4 to-ports=53

Verification

• AdGuard container querying 192.168.31.1 → SUCCESS
• MikroTik resolve command → SUCCESS

---

Additional Issue: Netflix Blocked by AdGuard Filter (13:45)

Symptom

• TV reports no internet even though DNS is working
• Netflix app fails to connect

Root Cause

AdGuard DNS filter (FilterListID:1) was blocking Netflix operational domains:

• ichnaea.netflix.com - Netflix connectivity check
• customerevents.netflix.com - Netflix events
• *.logs.netflix.com - Netflix logging

Netflix requires these domains to function, even though they're flagged as tracking.

Fix

Added whitelist rules to AdGuard:

@@||ichnaea.netflix.com^
@@||customerevents.netflix.com^
@@||logs.netflix.com^

Verification

Restarted AdGuard, TV should now connect to Netflix.

---

Root Cause Found: Asymmetric Routing (13:55)

Problem

DNS redirect (DNAT) was causing asymmetric routing:

1. Client sends DNS to 192.168.31.1:53
2. MikroTik DNATs to 192.168.31.4:53 (AdGuard)
3. AdGuard responds DIRECTLY to client (same L2 subnet)
4. Client receives response from .4 but expected it from .1
5. Client drops response → "no internet"

Fix

Added srcnat/masquerade rule so AdGuard sees MikroTik as source:

/ip firewall nat add chain=srcnat action=masquerade protocol=udp \
  src-address=192.168.31.0/24 dst-address=192.168.31.4 dst-port=53 \
  comment="Masquerade DNS to AdGuard"

Now the flow is:

1. Client → 192.168.31.1:53
2. DNAT → 192.168.31.4:53 (src masqueraded to MikroTik)
3. AdGuard responds to MikroTik
4. MikroTik un-NATs response → Client
5. Client sees response from .1 ✓

Verification

$ nslookup google.com 192.168.31.1
Server:   192.168.31.1
Name:     google.com
Address:  142.250.187.110