infrastructure/docs/incidents/2026-02-06-unraid-flash-drive-failure.md

Incident: Unraid Flash Drive Failure

Date: 2026-02-06 Severity: P1 - Server at risk Status: In Progress Affected: XTRM-U (Unraid NAS)

---

Symptoms

Unraid flash drive experiencing crashes/instability. Risk of complete failure and data loss of boot configuration.

---

Migration Procedure: Replace Flash Drive

Step 1: Retrieve Flash Backup

Try these options in order of preference:

Option A - Fresh backup from WebGUI (if server still boots):

1. Open http://192.168.10.20 in browser
2. Go to Main tab → click on Flash device
3. Under Flash Device Settings, click FLASH BACKUP
4. Download the ZIP file to your Mac

Option B - Google Drive (daily Rclone backup):

# From Mac (if rclone is installed)
rclone copy drive:Backups/unraid-flash ~/Desktop/unraid-flash-backup/

# Or download manually from Google Drive web UI
# Folder: Backups/unraid-flash

Option C - Local backup on Unraid (if server boots but WebGUI broken):

ssh -i ~/.ssh/id_ed25519_unraid root@192.168.10.20 -p 422
# Backup is at:
ls /mnt/user/Backup/unraid-flash/
# Copy it off the server:
scp -P 422 -i ~/.ssh/id_ed25519_unraid root@192.168.10.20:/mnt/user/Backup/unraid-flash/* ~/Desktop/unraid-flash-backup/

Option D - Direct copy from failing drive:

1. Shut down server
2. Remove flash drive, insert into Mac
3. Copy entire contents to ~/Desktop/unraid-flash-backup/

---

Step 2: Prepare New USB Drive

Requirements:

• USB 2.0 recommended (more reliable than USB 3.0 for this purpose)
• Capacity: 4 GB minimum, 32 GB maximum
• Reputable brand (SanDisk, Samsung, Kingston)
• Must have a unique hardware GUID

Write the backup to new drive:

1. Download Unraid USB Flash Creator for macOS
2. Insert new USB drive into Mac
3. Open Flash Creator
4. For Operating System, scroll down and select "Use custom"
5. Browse to your backup ZIP file from Step 1
6. Select the new USB drive as destination
7. Click Write and wait for completion

If you don't have a backup ZIP (only raw files from Option D):

1. In Flash Creator, select the Unraid OS version matching your current install
2. Write a fresh Unraid install to the new drive
3. After writing, mount the drive and copy your backed-up config/ folder onto it, replacing the default one

---

Step 3: Swap Drives and Boot

1. Shut down XTRM-U if still running
2. Remove the old (failing) flash drive
3. Insert the new USB drive
4. Power on the server
5. Wait for boot (1-2 minutes)
6. Try accessing WebGUI at http://192.168.10.20

If WebGUI doesn't load:

• Connect a monitor to the server to check boot messages
• Verify the USB drive is detected in BIOS
• Ensure boot order has USB first

---

Step 4: Transfer License

You will see an "Invalid, missing or expired registration key" message. This is expected.

1. In WebGUI, go to Tools → Registration
2. Click Replace Key
3. Enter the email address associated with your Unraid account
4. Check your email for the confirmation/license key
5. Follow the link or paste the key file URL into the Registration page
6. Click Done

Important warnings:

• Replacing the key permanently blacklists the old USB drive - it can never be used with Unraid again
• First license transfer can be done at any time
• Subsequent transfers: once per 12 months via the automated system
• If you need another transfer within 12 months, contact Unraid support with old GUID, new GUID, license key, and purchase email

If you can't find your license:

• Log into https://account.unraid.net to view your keys
• Check email for original purchase confirmation

---

Step 5: Post-Migration Verification

Run through this checklist after the server is back up:

Array & Storage:

• WebGUI loads at http://192.168.10.20
• Array starts normally (Main tab → Start)
• All disks show healthy status
• Shares are accessible

Docker & Services:

ssh -i ~/.ssh/id_ed25519_unraid root@192.168.10.20 -p 422

# Check all containers
docker ps -a --format 'table {{.Names}}\t{{.Status}}'

# Start any stopped critical containers (in order):
docker start postgresql17    # Wait 30s
docker start Redis           # Wait 10s
docker start traefik
docker start authentik authentik-worker
docker start vaultwarden

Network:

• SSH works: ssh -i ~/.ssh/id_ed25519_unraid root@192.168.10.20 -p 422
• DNS failover AdGuard reachable: http://192.168.10.10:3000
• AdGuard sync working (check docker logs adguardhome-sync --tail 5)
• External URLs working (https://xtrm-lab.org)

Services checklist:

• Traefik reverse proxy (https://xtrm-lab.org)
• Authentik SSO (https://auth.xtrm-lab.org)
• Gitea (https://git.xtrm-lab.org)
• Uptime Kuma (https://uptime.xtrm-lab.org)
• Vaultwarden (https://vault.xtrm-lab.org)
• Plex (https://plex.xtrm-lab.org)

Backup:

• Verify Rclone config still present: rclone listremotes (should show drive:)
• Test flash backup: trigger manual backup from WebGUI or User Scripts
• Verify cron schedule for flash backup is active

---

Step 6: Prevention

After successful migration:

1. Enable Unraid Connect (if not already) for automated cloud flash backup:

   • Settings → Management Access → Unraid Connect
   • Sign in with your unraid.net account
   • Enable Flash Backup

2. Verify Rclone cron is scheduled:

   # Check user scripts plugin for flash backup schedule
   ls /boot/config/plugins/user.scripts/scripts/
   

3. Keep a spare USB drive prepared with a fresh Unraid install - makes future recovery faster

4. Test backup restoration periodically - don't wait for a failure to discover your backup is incomplete

---

References

• Unraid Docs: Changing the Flash Device
• Unraid Docs: Licensing FAQ
• Internal: docs/02-SERVICES-CRITICAL.md (startup order)

---

Resolution

Update this section when migration is complete:

• Date resolved:
• New USB drive:
• License transferred: Yes/No
• Services verified: Yes/No
• Backup reconfigured: Yes/No