# usb4t: USB dropouts suspend the pool (2026-08) Symptom seen first in the nightly backup log (`/var/log/incus-copy.log`, job in `/etc/cron.d/incus-copy`): every instance fails with ``` Error: Refresh instance: Create instance volume from copy failed: Volume exists in database but not on storage ``` That message is misleading — incus is fine. The `usb4t` pool underneath was **SUSPENDED**, so the replica volumes incus has in its database had no storage behind them. ## Diagnosis (2026-08-28) The disk is healthy, the USB link is not: - `smartctl -a -d sat /dev/sdc`: **PASSED**, 0 reallocated, 0 pending, 0 offline-uncorrectable, **0 UDMA CRC errors**, 463 power-on hours. - `dmesg`: `usb 4-3: USB disconnect` followed by immediate re-enumeration — the enclosure drops off the bus and comes back as a new device. Bridge is **JMicron 152d:0578** (the kernel already disables UAS for it and applies quirks). - **61 USB disconnects in 30 days**; pool suspensions on Aug 17 (resilver), 19, 21, 27 (×2) and 28. - While suspended, anything touching the pool hangs — the journal shows tasks blocked for 1000+ seconds. Blast radius each time: nuc's own replicas (`usb4t/backup/nuc`) stop being refreshed, `/srv/media` disappears from the Jellyfin containers, and — once seeded — the ks4 pull leg (`usb4t/backup/ks4`) would stop too. **The pool holding the off-site copy of ks4 must not be the least reliable device in the setup.** ## Recovery ```sh zpool clear usb4t # device is back; ZFS resumes IO zpool status -v usb4t # lists files damaged by the interrupted writes zpool scrub usb4t # validate the whole pool ``` If `clear` hangs: `zpool export usb4t && zpool import usb4t`, reboot as a last resort. Datasets remount by themselves once the pool resumes. Damage from the 2026-08-28 incident, as first reported: two pool-metadata objects (`:<0x0>`, `:<0x3d>`) and one media file. **The scrub cleared all of it** — 1.08 T verified in 2 h 29 m, `0 B repaired, 0 errors`, and the error list is now `No known data errors`. Those entries were artefacts of the interrupted writes, not real corruption; nothing had to be restored. (One CKSUM count remains on the vdev — a single checksum ZFS fixed from its own redundant metadata copy.) The scrub also served as a 2.5-hour soak test of the new cable: **zero USB disconnects** during it. ## Fixes, in order 1. **Cable + port** — short quality cable, straight into a rear USB-3 port, no hub *(done 2026-08-28; survived a 2.5 h scrub with zero disconnects — watch for a week before concluding)*. 2. **Disable USB autosuspend** so power management cannot drop the device: `usbcore.autosuspend=-1` on the kernel command line, or per-device `echo on > /sys/bus/usb/devices//power/control`. 3. **Replace the enclosure** if dropouts continue — JMicron bridges are the usual suspect in this failure mode; an ASMedia-based enclosure is the durable fix. *Deferred: watching for a few days first.* ## Alerting (why nobody noticed for 11 days) `zfs-zed` was installed, enabled and running — but the host had **no MTA**, so its notifications went nowhere for 11 days. Fixed 2026-08-28; the working chain is: ``` zed → mail (bsd-mailx) → /usr/sbin/sendmail = msmtp → mail.lutran.fr:587 (auth, STARTTLS) ``` - `apt install msmtp msmtp-mta bsd-mailx` - **`/etc/msmtprc` (mode 600) holds the SMTP account** — host `mail.lutran.fr`, port **587 (submission)**, user `zed@lutran.fr`, STARTTLS, `aliases /etc/aliases`. - `/etc/zfs/zed.d/zed.rc` (mode 600): `ZED_EMAIL_ADDR="julien@lutran.fr"`, `ZED_EMAIL_PROG="mail"`, `ZED_EMAIL_OPTS="-s '@SUBJECT@' @ADDRESS@"`, `ZED_NOTIFY_VERBOSE=1` (so scrub/resilver results are mailed too, not only failures), `ZED_NOTIFY_INTERVAL_SECS=3600`. Original kept as `zed.rc.orig`. **No credentials in this file** — they live only in `/etc/msmtprc`. **Why nuc differs from ks2/ks4** (which mail via s-nail straight to port 25, and work fine): it is not the zed config, it is the source address. ks4 *is* the mail server's host (internal client, `mynetworks`); ks2 is a datacenter IP with a PTR. nuc sends from a **dynamic home IP with no PTR and no SPF alignment**, so the inbound MX path treats it like any stranger. Hence submission + auth here, deliberately different from the other two hosts. Two dead ends worth remembering: - **Port 25 is the wrong port for alerts.** Authentication succeeds, then rspamd rejects the message: `554 5.7.1 Spam message rejected` — a short mail from a dynamic home IP with no SPF alignment scores badly on the inbound MX path. Submission (587) does not apply those rules. - **The home IP has no PTR**, so postfix's reverse lookup delays the 587 greeting by several seconds. Probes with short timeouts look like a firewall block (`lost connection after CONNECT, commands=0/0` in the server log) — give SMTP tests ≥30 s before concluding anything. Verify: ```sh echo "test $(date)" | mail -s "nuc alert test" julien@lutran.fr tail -2 /var/log/msmtp.log # expect smtpstatus=250 ``` End-to-end proof: with `ZED_NOTIFY_VERBOSE=1`, the next scrub completion arrives as a mail by itself.