diff --git a/nuc/usb4t-dropouts.md b/nuc/usb4t-dropouts.md index 93a83a3..0ae4447 100644 --- a/nuc/usb4t-dropouts.md +++ b/nuc/usb4t-dropouts.md @@ -59,15 +59,20 @@ The scrub also served as a 2.5-hour soak test of the new cable: ## Fixes, in order -1. **Cable + port** — short quality cable, straight into a rear USB-3 - port, no hub *(done 2026-08-28; survived a 2.5 h scrub with zero - disconnects — watch for a week before concluding)*. +1. ~~**Cable + port**~~ — replaced 2026-08-28 with a short quality + cable straight into a rear USB-3 port, no hub. It survived a 2.5 h + scrub, then **dropped again on 2026-08-29 16:22** (pool suspended, + cleared 08-30 00:32). **Cable ruled out.** 2. **Disable USB autosuspend** so power management cannot drop the device: `usbcore.autosuspend=-1` on the kernel command line, or per-device `echo on > /sys/bus/usb/devices//power/control`. -3. **Replace the enclosure** if dropouts continue — JMicron bridges are - the usual suspect in this failure mode; an ASMedia-based enclosure - is the durable fix. *Deferred: watching for a few days first.* +3. **Replace the enclosure — now the prime candidate.** With the cable + ruled out and the disk SMART-clean (0 reallocated/pending/CRC), the + JMicron 152d:0578 bridge is what remains. An ASMedia-based + enclosure (or a direct SATA connection) is the durable fix. + ⚠️ Until this is settled, this pool is not a trustworthy home for + the ks4 off-site replica leg — treat it as a gate before seeding + over FTTH. ## Alerting (why nobody noticed for 11 days) @@ -111,10 +116,28 @@ Two dead ends worth remembering: in the server log) — give SMTP tests ≥30 s before concluding anything. +### ZED does not report everything — hence the watchdog + +The 2026-08-29 suspension produced **no email**, although the mail path +demonstrably worked (the scrub-finish notification arrived the evening +before). Two reasons: + +- a suspended pool keeps its vdev marked **ONLINE**, so + `statechange-notify.sh` never fires; +- the `data`-class events ZED *did* raise were muted because + `ZED_NOTIFY_DATA` was unset (now `=1`). + +Fixed 2026-08-30 by adding +[`zpool-health.sh`](https://git.lutran.fr/julien/scripts/src/branch/main/zpool-health.sh), +run every 15 min from `/etc/cron.d/zpool-health`. It mails only on +`healthy <-> problem` **transitions**, so it is silent in normal +operation and cannot spam; `-t` sends a test. Worth deploying on ks4 +too — its `data` pool is single-disk and has the same blind spot. + Verify: ```sh -echo "test $(date)" | mail -s "nuc alert test" julien@lutran.fr +/root/scripts/zpool-health.sh -m julien@lutran.fr -t tail -2 /var/log/msmtp.log # expect smtpstatus=250 ```