Add incident report: DNS outbound blocked after MikroTik restart (2026-01-25)

docs/incidents/2026-01-25-dns-outbound-blocked-after-mikrotik-restart.md

@@ -0,0 +1,88 @@
+# Incident: DNS Outbound Blocked After MikroTik Restart
+
+**Date:** 2026-01-25  
+**Duration:** ~1 hour  
+**Severity:** High (Complete DNS failure)  
+**Status:** Resolved  
+
+---
+
+## Summary
+
+After MikroTik router restart, AdGuardHome could not reach upstream DNS servers, causing complete DNS resolution failure for all network clients.
+
+## Symptoms
+
+- AdGuardHome container showing timeout errors to upstream DNS (8.8.8.8, Quad9)
+- All DNS queries failing network-wide
+- ICMP (ping) to external IPs worked, but port 53 (DNS) was unreachable
+- Error pattern: `i/o timeout` when querying any external DNS server
+
+## Root Cause
+
+MikroTik NAT rules were redirecting ALL DNS traffic from `192.168.31.0/24` to AdGuardHome, including AdGuardHome's own outbound DNS queries. This created a loop where:
+
+1. AdGuardHome (192.168.31.4) tries to query 8.8.8.8:53
+2. MikroTik NAT rule 7 intercepts this traffic (src: 192.168.31.0/24, dst-port: 53)
+3. Traffic redirected back to AdGuardHome (172.17.0.5:5355)
+4. Query fails with timeout
+
+**Problematic NAT Rules:**
+```
+Rule 7: Force DNS to AdGuard Home
+  chain=dstnat action=dst-nat to-addresses=172.17.0.5 to-ports=5355 
+  protocol=udp src-address=192.168.31.0/24 dst-port=53
+
+Rule 9: Force DNS to AdGuard Home TCP
+  chain=dstnat action=dst-nat to-addresses=172.17.0.5 to-ports=5355 
+  protocol=tcp src-address=192.168.31.0/24 dst-port=53
+```
+
+## Resolution
+
+Added exception rules BEFORE the redirect rules to allow AdGuardHome to reach external DNS:
+
+```bash
+# Added via SSH to MikroTik
+/ip firewall nat add chain=dstnat protocol=udp src-address=192.168.31.4 dst-port=53 action=accept comment=Allow AdGuard outbound DNS place-before=7
+/ip firewall nat add chain=dstnat protocol=tcp src-address=192.168.31.4 dst-port=53 action=accept comment=Allow AdGuard outbound DNS TCP place-before=8
+```
+
+**Additional Changes:**
+1. Set MikroTik DNS to use only AdGuard: `/ip dns set servers=192.168.31.4`
+2. Disabled ISP DNS from DHCP: `/ip dhcp-client set 0 use-peer-dns=no`
+3. Updated Unraid resolv.conf to use 192.168.31.4 (for Tailscale access)
+
+## Final NAT Rule Order (DNS-related)
+
+| # | Comment | Action | Details |
+|---|---------|--------|---------|
+| 5 | Allow AdGuard outbound DNS | accept | src=192.168.31.4, udp/53 |
+| 6 | Force DNS to AdGuard Home | dst-nat | src=192.168.31.0/24 → 172.17.0.5:5355 |
+| 7 | Allow AdGuard outbound DNS TCP | accept | src=192.168.31.4, tcp/53 |
+| 9 | Force DNS to AdGuard Home TCP | dst-nat | src=192.168.31.0/24 → 172.17.0.5:5355 |
+
+## Verification
+
+After fix:
+- ✓ AdGuard can resolve external DNS (google.com, github.com)
+- ✓ MikroTik using only AdGuard as DNS
+- ✓ Internal domains resolving (git.xtrm-lab.org, cloud.xtrm-lab.org)
+- ✓ External access to services working (Gitea, Woodpecker CI)
+
+## Lessons Learned
+
+1. NAT redirect rules need exceptions for the DNS server itself
+2. After MikroTik restart, verify DNS flow end-to-end
+3. Keep exception rules BEFORE redirect rules in firewall
+
+## Related Configuration
+
+- AdGuard IP: 192.168.31.4 (macvlan)
+- AdGuard internal: 172.17.0.5:5355
+- MikroTik: 192.168.31.1
+- Upstream DNS: 8.8.8.8 (temporarily, was Quad9 DoH)
+
+---
+
+*Incident logged by Claude Code via Tailscale connection*