nas+nuc: cold boot verified; record what clearing the metadata errors took

Both hosts were powered off and brought back from a cold start, which is
the first real test of everything built on 2026-08-30.

nas: both pools imported from /etc/zfs/zpool.cache, all instances
autostarted, NFS exports republished, 0 failed units. nuc: /dev/dri
present with 0 i915 warnings (booted with the display unplugged), NFS
auto-remounted, VAAPI transcode at 9.5x realtime.

Records the ordering that actually cleared tank's inherited
<metadata>:<0x0> and <0x3d>: a scrub alone found 0 errors and repaired 0B
but left them, and a plain `zpool clear` afterwards did not drop them —
ZFS flushes the persistent error log on a scrub run *after* the clear.
That matters beyond tidiness, because while those entries stand
`zpool status -x` reports the pool unhealthy forever and zpool-health.sh
cannot signal anything new.

Two kiosk corrections, both from observed behaviour:

- the Pioneer DAC being switched off is the most likely cause of
  "video, no sound" — asound.conf pins the ALSA default to it by card
  name, so `default` fails to open outright and mpv falls back to null
  silently. Adds the one-line aplay check.
- hotplugging the display makes cage exit once and Restart=on-failure
  recovers it ~5s later. Do NOT restart it by hand; check
  ActiveEnterTimestamp against the hotplug time first.

Also flags that the OS mirror is still untested with a disk physically
unplugged — it is a guess until then.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Julien Lutran
2026-08-31 10:16:44 +02:00
co-authored by Claude Opus 5
parent ef88ef9342
commit 81d22c5daf
2 changed files with 50 additions and 6 deletions
+33 -6
View File
@@ -416,7 +416,30 @@ instances depend on that host surviving.
Once `tank` has run a week on SATA with **zero** pool suspensions and no Once `tank` has run a week on SATA with **zero** pool suspensions and no
CRC errors, the `usb4t-dropouts` gate is cleared — record that in CRC errors, the `usb4t-dropouts` gate is cleared — record that in
[ks2/plan.md](../ks2/plan.md). [ks2/plan.md](../ks2/plan.md). Day 1 was clean (2026-08-31).
### Clearing the inherited `<metadata>` errors — order matters
The pool imported carrying `<metadata>:<0x0>` and `<metadata>:<0x3d>` from
the 2026-08-29 USB dropout. A scrub found **0 errors and repaired 0B**,
yet the entries stayed, and a plain `zpool clear` afterwards did not drop
them either. ZFS flushes its persistent error log on a scrub that runs
**after** the clear — so the working order is:
```sh
zpool clear tank
zpool scrub tank # this is the run that flushes the log
```
Result 2026-08-31: `scrub repaired 0B in 02:34:24 with 0 errors`,
`errors: No known data errors`, `all pools are healthy`. They were
artefacts of interrupted writes, not corruption — matching the
[2026-08-28 incident](../nuc/usb4t-dropouts.md).
⚠️ This matters for monitoring, not just tidiness: while those entries
stand, `zpool status -x` reports the pool unhealthy permanently, so
`zpool-health.sh` sits in the alarm state and **cannot signal a new
problem**. Clear them before trusting the watchdog.
## 6. Incus ## 6. Incus
@@ -804,11 +827,15 @@ trace can be recut against the meter's series afterwards.
- [x] IPMI **left unplugged by decision** (2026-08-30) — no BMC on the - [x] IPMI **left unplugged by decision** (2026-08-30) — no BMC on the
LAN, therefore no out-of-band console either LAN, therefore no out-of-band console either
- [ ] `cat /proc/mdstat` — all three arrays `[UU]`; GRUB written to **both** - [x] `cat /proc/mdstat` — all three arrays `[UU]`; GRUB written to
ESPs (`efibootmgr -v` shows two entries), **both** ESPs, both mounted, `debian` + `debian-b` boot entries
and boot tested with one disk unplugged present (verified across a cold boot 2026-08-31)
- [ ] `zpool status` healthy for `incus` and `tank`; weekly scrubs - [ ] boot still untested with **one disk physically unplugged** — the
scheduled; both import cleanly after a reboot (`cachefile` set) mirror is a guess until that is done
- [x] `zpool status` healthy for `incus` and `tank`; weekly scrubs
scheduled; **cold boot verified 2026-08-31** — both pools imported
from `/etc/zfs/zpool.cache`, all instances autostarted, NFS exports
republished, 0 failed units
- [ ] **7 days with zero pool suspensions and zero CRC errors** — the - [ ] **7 days with zero pool suspensions and zero CRC errors** — the
gate that closes [usb4t-dropouts.md](../nuc/usb4t-dropouts.md) gate that closes [usb4t-dropouts.md](../nuc/usb4t-dropouts.md)
- [x] `smartd` monitoring all 3 disks; `zpool-health.sh -t` mail - [x] `smartd` monitoring all 3 disks; `zpool-health.sh -t` mail
+17
View File
@@ -447,6 +447,23 @@ incus exec jellyfin-client -- udevadm info /dev/input/event0 # udev db visibl
visible (#2); check `run-udev.mount` is active. visible (#2); check `run-udev.mount` is active.
- **Video OK, no sound; JMP log shows `AO: [null]`** → ALSA default broken - **Video OK, no sound; JMP log shows `AO: [null]`** → ALSA default broken
(#3); check `/etc/asound.conf` and the card name in `aplay -l`. (#3); check `/etc/asound.conf` and the card name in `aplay -l`.
**Most common cause: the Pioneer DAC is simply switched off.**
`/etc/asound.conf` pins the default to it *by card name* (`Device`), so
with the amp off the name does not exist and `default` fails to open —
mpv then falls back to null silently: picture, no sound, no error.
One-line check before blaming anything else:
```sh
incus exec jellyfin-client -- su -s /bin/bash kiosk -c 'aplay -D default -d 1 /usr/share/sounds/alsa/Front_Center.wav'
```
`audio open error: No such device` = amp is off. Power it on; no
restart needed, JMP opens the device per playback.
- **Display hotplugged after boot → cage exits once, then recovers by
itself.** If the kiosk started with no outputs, plugging the HDMI in
makes cage fail (`Failed with result 'exit-code'`); the unit's
`Restart=on-failure` / `RestartSec=5` restarts it ~5 s later, this time
with the display present. **Do not restart it by hand** — check
`systemctl show jellyfin-kiosk -p ActiveEnterTimestamp` first and only
intervene if the timestamp predates the hotplug. Verified 2026-08-31.
- **Keyboard plugged in after boot isn't seen** — the host udev db is live - **Keyboard plugged in after boot isn't seen** — the host udev db is live
through the bind, but udev hotplug *events* don't cross the container's through the bind, but udev hotplug *events* don't cross the container's
network namespace, so cage only enumerates at startup. Re-plugging the network namespace, so cage only enumerates at startup. Re-plugging the