nuc: second dropout rules out the cable; add the pool-health watchdog

Aug 29 16:22 the enclosure dropped again despite the new cable, pool
suspended, and ZED sent nothing (suspended pools keep the vdev ONLINE;
data-class events were muted). Enclosure replacement is now the prime
candidate and a gate before the pool hosts the ks4 off-site leg.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

nuc/usb4t-dropouts.md

@@ -59,15 +59,20 @@ The scrub also served as a 2.5-hour soak test of the new cable:
 
 ## Fixes, in order
 
-1. **Cable + port** — short quality cable, straight into a rear USB-3
+1. ~~**Cable + port**~~ — replaced 2026-08-28 with a short quality
-   port, no hub *(done 2026-08-28; survived a 2.5 h scrub with zero
+   cable straight into a rear USB-3 port, no hub. It survived a 2.5 h
-   disconnects — watch for a week before concluding)*.
+   scrub, then **dropped again on 2026-08-29 16:22** (pool suspended,
+   cleared 08-30 00:32). **Cable ruled out.**
 2. **Disable USB autosuspend** so power management cannot drop the
    device: `usbcore.autosuspend=-1` on the kernel command line, or
    per-device `echo on > /sys/bus/usb/devices/<dev>/power/control`.
-3. **Replace the enclosure** if dropouts continue — JMicron bridges are
+3. **Replace the enclosure — now the prime candidate.** With the cable
-   the usual suspect in this failure mode; an ASMedia-based enclosure
+   ruled out and the disk SMART-clean (0 reallocated/pending/CRC), the
-   is the durable fix. *Deferred: watching for a few days first.*
+   JMicron 152d:0578 bridge is what remains. An ASMedia-based
+   enclosure (or a direct SATA connection) is the durable fix.
+   ⚠️ Until this is settled, this pool is not a trustworthy home for
+   the ks4 off-site replica leg — treat it as a gate before seeding
+   over FTTH.
 
 ## Alerting (why nobody noticed for 11 days)
 
@@ -111,10 +116,28 @@ Two dead ends worth remembering:
   in the server log) — give SMTP tests ≥30 s before concluding
   anything.
 
+### ZED does not report everything — hence the watchdog
+
+The 2026-08-29 suspension produced **no email**, although the mail path
+demonstrably worked (the scrub-finish notification arrived the evening
+before). Two reasons:
+
+- a suspended pool keeps its vdev marked **ONLINE**, so
+  `statechange-notify.sh` never fires;
+- the `data`-class events ZED *did* raise were muted because
+  `ZED_NOTIFY_DATA` was unset (now `=1`).
+
+Fixed 2026-08-30 by adding
+[`zpool-health.sh`](https://git.lutran.fr/julien/scripts/src/branch/main/zpool-health.sh),
+run every 15 min from `/etc/cron.d/zpool-health`. It mails only on
+`healthy <-> problem` **transitions**, so it is silent in normal
+operation and cannot spam; `-t` sends a test. Worth deploying on ks4
+too — its `data` pool is single-disk and has the same blind spot.
+
 Verify:
 
 ```sh
-echo "test $(date)" | mail -s "nuc alert test" julien@lutran.fr
+/root/scripts/zpool-health.sh -m julien@lutran.fr -t
 tail -2 /var/log/msmtp.log          # expect smtpstatus=250
 ```