Julien LutranandClaude Fable 5
5ceb2ab742
Nightly replication had been failing since Aug 17 behind a misleading incus error; root cause is a JMicron USB bridge dropping off the bus (61 disconnects in 30 days) and suspending the pool — the disk itself is SMART-clean. Records the recovery (clear/scrub), the damage, the ordered fixes, and why nothing alerted: zed was running without an MTA on the host. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
86 lines
3.4 KiB
Markdown
86 lines
3.4 KiB
Markdown
# usb4t: USB dropouts suspend the pool (2026-08)
|
||
|
||
Symptom seen first in the nightly backup log
|
||
(`/var/log/incus-copy.log`, job in `/etc/cron.d/incus-copy`): every
|
||
instance fails with
|
||
|
||
```
|
||
Error: Refresh instance: Create instance volume from copy failed:
|
||
Volume exists in database but not on storage
|
||
```
|
||
|
||
That message is misleading — incus is fine. The `usb4t` pool underneath
|
||
was **SUSPENDED**, so the replica volumes incus has in its database had
|
||
no storage behind them.
|
||
|
||
## Diagnosis (2026-08-28)
|
||
|
||
The disk is healthy, the USB link is not:
|
||
|
||
- `smartctl -a -d sat /dev/sdc`: **PASSED**, 0 reallocated, 0 pending,
|
||
0 offline-uncorrectable, **0 UDMA CRC errors**, 463 power-on hours.
|
||
- `dmesg`: `usb 4-3: USB disconnect` followed by immediate
|
||
re-enumeration — the enclosure drops off the bus and comes back as a
|
||
new device. Bridge is **JMicron 152d:0578** (the kernel already
|
||
disables UAS for it and applies quirks).
|
||
- **61 USB disconnects in 30 days**; pool suspensions on Aug 17
|
||
(resilver), 19, 21, 27 (×2) and 28.
|
||
- While suspended, anything touching the pool hangs — the journal
|
||
shows tasks blocked for 1000+ seconds.
|
||
|
||
Blast radius each time: nuc's own replicas (`usb4t/backup/nuc`) stop
|
||
being refreshed, `/srv/media` disappears from the Jellyfin containers,
|
||
and — once seeded — the ks4 pull leg (`usb4t/backup/ks4`) would stop
|
||
too. **The pool holding the off-site copy of ks4 must not be the least
|
||
reliable device in the setup.**
|
||
|
||
## Recovery
|
||
|
||
```sh
|
||
zpool clear usb4t # device is back; ZFS resumes IO
|
||
zpool status -v usb4t # lists files damaged by the interrupted writes
|
||
zpool scrub usb4t # validate the whole pool
|
||
```
|
||
|
||
If `clear` hangs: `zpool export usb4t && zpool import usb4t`, reboot as
|
||
a last resort. Datasets remount by themselves once the pool resumes.
|
||
|
||
Damage from the 2026-08-28 incident: two pool-metadata objects
|
||
(`<metadata>:<0x0>`, `<metadata>:<0x3d>`) and one file —
|
||
`/srv/media/videos/David Gilmour - Live In Gdansk (2008)/…D3.ISO`
|
||
(re-rippable). Nothing in the backup datasets.
|
||
|
||
## Fixes, in order
|
||
|
||
1. **Cable + port** — short quality cable, straight into a rear USB-3
|
||
port, no hub *(done 2026-08-28)*.
|
||
2. **Disable USB autosuspend** so power management cannot drop the
|
||
device: `usbcore.autosuspend=-1` on the kernel command line, or
|
||
per-device `echo on > /sys/bus/usb/devices/<dev>/power/control`.
|
||
3. **Replace the enclosure** if dropouts continue — JMicron bridges are
|
||
the usual suspect in this failure mode; an ASMedia-based enclosure
|
||
is the durable fix. *Deferred: watching for a few days first.*
|
||
|
||
## Alerting (why nobody noticed for 11 days)
|
||
|
||
`zfs-zed` was installed, enabled and running — but the host had **no
|
||
MTA**, so its notifications went nowhere. Fixed 2026-08-28:
|
||
|
||
- `apt install msmtp msmtp-mta bsd-mailx` — `sendmail` is now msmtp,
|
||
`mail` (bsd-mailx) is what ZED calls.
|
||
- `/etc/msmtprc` (mode 600): SMTP relay account. **Contains
|
||
`CHANGEME` placeholders — fill host/user/password/from.**
|
||
It also resolves `/etc/aliases`, where `root:` points at the real
|
||
mailbox (placeholder too).
|
||
- `/etc/zfs/zed.d/zed.rc`: `ZED_EMAIL_ADDR`, `ZED_EMAIL_PROG="mail"`,
|
||
`ZED_EMAIL_OPTS`, `ZED_NOTIFY_VERBOSE=1` (so scrub results and
|
||
resilvers are reported, not just failures),
|
||
`ZED_NOTIFY_INTERVAL_SECS=3600`. Original kept as `zed.rc.orig`.
|
||
|
||
Test once the credentials are in:
|
||
|
||
```sh
|
||
echo "zed test $(date)" | mail -s "nuc alert test" root
|
||
tail -5 /var/log/msmtp.log
|
||
```
|