diff --git a/nuc/README.md b/nuc/README.md index 8da65b1..2895bc1 100644 --- a/nuc/README.md +++ b/nuc/README.md @@ -1,31 +1,45 @@ # Homelab: nuc -Incus host on the LAN. +Incus host on the LAN — **on-demand**: since 2026-08-30 it only needs to +run when watching Jellyfin or using the Spotify Connect kiosk. Everything +always-on (blocky/DNS, privoxy, transmission-bt) moved to +[`nas`](../nas/README.md), which is why nuc can now be powered off. + +⚠️ Powering nuc off has backup consequences — its 03:30 replication to +nas only runs while it is up. See [nas-install.md](../nas/nas-install.md) §9. - Debian 13, Intel Alder Lake-N (iGPU `i915`, shared by both Jellyfin containers) — bare-metal reinstall: [nuc-install.md](nuc-install.md) - Instances are bridged onto the LAN (192.168.0.0/24) -- USB 4 TB WD Red: ZFS pool `usb4t` — `usb4t/backup` → `/backup` - (incus exports, `nuc/` + `ks4/` subdatasets, 1 TB quota) and - `usb4t/media` → `/srv/media` (media library, shared into containers - via `shift=true` disk devices; works because ZFS ≥ 2.2 supports - idmapped mounts) -- ⚠️ The USB enclosure drops off the bus periodically and suspends the - pool, which silently breaks the nightly replication — symptoms, - recovery and fixes: [usb4t-dropouts.md](usb4t-dropouts.md) -- Backups: all local instances replicated to the USB pool - (`/root/scripts/incus-copy.sh -p backup -s nucbackup`; replicas - stopped, autostart off) — see - [nuc-install.md](nuc-install.md); ks4 replicas pulled into - pool `ks4backup` — see [ks4/incus-copy.md](../ks4/incus-copy.md) +- Storage: ZFS pool `data` on the SSD (all instance disks). The USB + 4 TB and its pool **left nuc on 2026-08-30** — the enclosure's JMicron + bridge was suspending the pool ([usb4t-dropouts.md](usb4t-dropouts.md)); + the disk now sits on direct SATA in [`nas`](../nas/nas-install.md) as + pool `tank`. +- Media library: `/srv/media` is now an **NFSv4 mount from nas** + (`192.168.0.4:/media`, read-only). `jellyfin-server` stays here for + the iGPU and reads it with `shift=false` + ([jellyfin-server.md](jellyfin-server.md)). +- Backups: nuc and nas **cross-replicate** — nuc pushes all its + instances to `nas:nucbackup` at 03:30, nas pushes its own to + `nuc:nasbackup` at 04:00, so neither host's instances depend on that + host surviving ([nas/nas-install.md](../nas/nas-install.md) §9). + The ks4 pull leg moved to nas as well + ([ks2/nas-seed.md](../ks2/nas-seed.md)), so nuc no longer runs a + WireGuard tunnel. + +- ⚠️ **Never reboot nuc with the display powered on** — the i915 probe + dies and takes every container with it (including LAN DNS). Symptoms + and fix: [jellyfin-client.md](jellyfin-client.md#-never-boot-nuc-with-the-display-active-2026-08-30) ## Instances | Name | IP | Doc | Features | |---|---|---|---| -| [jellyfin-server](jellyfin-server.md) | 192.168.0.5 | ✅ | unprivileged, autostart; iGPU render node (`gpu` device, render gid) for QSV/VAAPI transcoding; `/srv/media` disk device (`shift=true`); proxy device → host :8096 | +| [jellyfin-server](jellyfin-server.md) | 192.168.0.5 | ✅ | unprivileged, autostart; iGPU render node (`gpu` device, render gid) for QSV/VAAPI transcoding; `/srv/media` disk device (NFS from nas, `shift=false`, read-only); proxy device → host :8096 | | [jellyfin-client](jellyfin-client.md) | 192.168.0.6 | ✅ | **privileged**, autostart; full iGPU (`gpu` device, gid 44) → HDMI kiosk (cage + Jellyfin Media Player); custom `raw.lxc` (bind `/dev/snd`, `/dev/input`, host `/run/udev`); Pioneer USB audio as ALSA default; FR keymap; go-librespot Spotify Connect ("Pioneer A-70") | -| [transmission-bt](transmission-bt.md) | 192.168.0.7 | ✅ | unprivileged, autostart; always-on WireGuard full tunnel → ks4 (egress = 193.70.35.17, kill switch: no default route); `/srv/media` disk device (`shift=true`), downloads to `/media/downloads`; web UI :9091 (LAN only) | -| blocky | 192.168.0.254 | — | unprivileged, autostart; DNS ad-blocker | -| privoxy | 192.168.0.11 | — | unprivileged, autostart; filtering HTTP proxy | | homeassistant | (stopped) | — | **virtual machine**, 50 GiB root disk on pool `data` | + +Moved to [`nas`](../nas/README.md) on 2026-08-30: `transmission-bt` +(next to the media dataset it writes to), plus `blocky` and `privoxy` +so LAN DNS and the proxy survive nuc being shut down. diff --git a/nuc/jellyfin-client.md b/nuc/jellyfin-client.md index f9a5be8..bcd640b 100644 --- a/nuc/jellyfin-client.md +++ b/nuc/jellyfin-client.md @@ -116,6 +116,15 @@ EOF' # --- Kiosk user + seat management ---------------------------------------------- incus exec "$CNAME" -- bash -c 'id kiosk >/dev/null 2>&1 || useradd -m -G video,render,input,audio kiosk' +# ⚠️ Group names are NOT enough. The host (Debian) and the container +# (Ubuntu) allocate dynamic system gids independently, so the container's +# `input` group does not necessarily have the same gid as the group that +# owns /dev/input/* on the host. Bind the kiosk user to the *numeric* +# host gid, whatever it is called inside: +HOST_INPUT_GID="$(stat -c %g /dev/input/event0)" +incus exec "$CNAME" -- usermod -aG "$HOST_INPUT_GID" kiosk +incus exec "$CNAME" -- id kiosk # must list $HOST_INPUT_GID + # In a container seatd must NOT bind the seat to a VT (there is no usable VT; # it would try to open the host's active tty and hang the compositor forever). incus exec "$CNAME" -- mkdir -p /etc/systemd/system/seatd.service.d @@ -245,10 +254,17 @@ it works regardless of which USB port the dongle lands on. `/etc/udev/rules.d/99-jellyfin-kiosk-recover.rules`: ``` -# Logitech Unifying receiver (re)plugged -> recover the jellyfin-client kiosk +# A Logitech receiver was (re)plugged -> recover the jellyfin-client kiosk. +# c52b = Unifying receiver (K400); c539 = Lightspeed receiver (G603). ACTION=="add", SUBSYSTEM=="usb", ENV{DEVTYPE}=="usb_device", ATTR{idVendor}=="046d", ATTR{idProduct}=="c52b", RUN+="/usr/bin/systemctl --no-block start jellyfin-kiosk-recover.service" +ACTION=="add", SUBSYSTEM=="usb", ENV{DEVTYPE}=="usb_device", ATTR{idVendor}=="046d", ATTR{idProduct}=="c539", RUN+="/usr/bin/systemctl --no-block start jellyfin-kiosk-recover.service" ``` +⚠️ The match is per product ID, so **a receiver not listed here will not +auto-recover** — the kiosk must be restarted by hand after plugging it in +(`incus exec jellyfin-client -- systemctl restart jellyfin-kiosk`). Add +the new id here when introducing different input hardware. + `/etc/systemd/system/jellyfin-kiosk-recover.service` — oneshot, so a burst of udev events during one plug merges into a single restart (natural debounce): @@ -285,6 +301,134 @@ udevadm control --reload-rules && systemctl daemon-reload journalctl -t jellyfin-kiosk-recover -f ``` +## ⚠️ Never boot nuc with the display active (2026-08-30) + +**Symptom:** after a reboot, *no* Incus container starts. `incus list` +answers but `incus start ` hangs forever, `systemctl status +incus` sits in `activating (start-post)`, and LAN DNS is down because +blocky never came up. Nothing in the incus logs explains it. + +**Cause — nothing to do with incus.** If the TV/projector is connected +**and powered on** when nuc boots, firmware hands i915 an already-lit +pipe. The driver's state readback then trips a series of warnings and +the probe never completes: + +``` +drm_WARN_ON(!pll_active) intel_ddi.c:4019 intel_ddi_get_clock +drm_WARN_ON(p0 == 0 || p1 == 0 || p2 == 0) intel_dpll_mgr.c:2878 +drm_WARN_ON(pixel_rate == 0) skl_watermark.c:1729 +``` +(all inside `intel_modeset_setup_hw_state` ← `intel_display_driver_probe_nogem`) + +The cascade: + +1. i915 probe dies → **`/dev/dri` never appears** (no GPU at all) +2. `snd_hda_intel` waits forever for i915's audio component → + permanent **deferred probe** holding the PCI device lock on `0000:00:1f.3` +3. incusd reads that device's `sriov_numvfs` while enumerating + resources → blocks in **D state** → the daemon never signals ready, + so nothing autostarts and every `incus start` hangs + +**Reproduced identically on 6.12.107 and 6.12.105** — it is the display +path, not a kernel regression. Do not waste time pinning kernels. + +**Diagnosis, in order:** + +```sh +ls /dev/dri/ # empty = i915 probe failed +cat /sys/kernel/debug/devices_deferred # snd_hda_intel entry = the deadlock +ps -eLo pid,tid,stat,wchan:26,comm | awk '$3 ~ /D/' # incusd in sriov_numvfs_show +dmesg -T | grep -E 'drm_WARN_ON|deferred probe pending' +``` + +**Fix:** disconnect HDMI (or power the display fully off — not standby), +reboot, then **hotplug the cable back in**. Connecting after boot goes +through normal connector detection instead of firmware state readback +and works fine. + +In-place recovery is *not* possible: `modprobe -r i915` fails (module in +use by the wedged probe) and `modprobe i915` times out. A reboot is the +only way out. + +**After hotplugging, restart the kiosk** — `cage` started with zero +outputs and will not pick the display up on its own: + +```sh +incus exec jellyfin-client -- systemctl restart jellyfin-kiosk +cat /sys/class/drm/card0-HDMI-A-2/status # expect: connected +``` + +Note the HDA controller this wedges is only used for **HDMI audio**, +which this setup does not use — audio goes to the Pioneer USB DAC via +`/etc/asound.conf`. It is pure collateral damage, but it takes the whole +host down with it. + +## Input gid mismatch — latent, fix it anyway (2026-08-30) + +> ⚠️ **This was not the cause of the 2026-08-30 outage.** That turned out +> to be a flat/switched-off K400 — a G603 on the same port and the same +> `event0` worked immediately. The mismatch below is real and worth +> correcting, but with `LIBSEAT_BACKEND=seatd` it is **seatd (running as +> root) that opens input devices** and passes the fd to cage, so the +> kiosk user's group membership is not on the critical path for input. +> Fix it for the direct-open fallback path, not as a debugging lead. +> +> **Before suspecting software, prove the hardware emits anything:** +> ```sh +> timeout 60 cat /dev/input/eventN | wc -c # press keys; 0 bytes = nothing reached the kernel +> ``` +> That one check would have saved an hour. +> +> **Dead K400 batteries are invisible from the host.** This K400 exposes +> no `hidpp_battery_*` node under `/sys/class/power_supply/`, so charge +> cannot be read. Worse, the receiver still lists the keyboard as a paired +> peer (`0003:046D:4024.*` under the `C52B` receiver) whether or not it is +> awake, and `/dev/input/event0` plus a `Logitech K400` entry in +> `/proc/bus/input/devices` are present either way — so every software +> check looks perfectly healthy. **Zero bytes from the raw capture is the +> only signal.** Swapping in a different receiver on the same port is the +> quickest A/B confirmation. + +**Symptom (if it ever does bite):** kiosk renders but input does nothing, +with no errors — `WLR_LIBINPUT_NO_DEVICES=1` keeps cage alive rather than +failing loudly. + +**Cause:** `/dev/input/*` is `crw-rw---- root:`. The +install script adds `kiosk` to the group *named* `input` inside the +container, but Debian (host) and Ubuntu (container) allocate dynamic +system gids independently: + +``` +host /dev/input/event0 gid 996 -> "input" +container "input" group gid 995 <- kiosk was here +container gid 996 -> "systemd-timesync" +``` + +So the kiosk user was in the wrong group and could not open any input +device. The udev side was fine — check it first to rule it out: +`cat /run/udev/data/c13:64` should show `E:ID_INPUT=1` etc. + +**Diagnose:** + +```sh +stat -c '%n %a %u:%g' /dev/input/event0 # host gid +incus exec jellyfin-client -- id kiosk # does it include that gid? +# open() test — do NOT use `head`/`cat`, reading an event device blocks +# with no pending events and looks like a permission failure: +incus exec jellyfin-client -- su -s /bin/bash kiosk -c 'exec 3< /dev/input/event0 && echo OPEN_OK' +``` + +**Fix** (persists in the container's `/etc/group`): + +```sh +HOST_INPUT_GID="$(stat -c %g /dev/input/event0)" +incus exec jellyfin-client -- usermod -aG "$HOST_INPUT_GID" kiosk +incus exec jellyfin-client -- systemctl restart jellyfin-kiosk +``` + +Re-check this after any host reinstall — the host's `input` gid is +dynamically allocated and can come back different. + ## Troubleshooting ```sh diff --git a/nuc/jellyfin-server.md b/nuc/jellyfin-server.md index e87ab43..c2f6ba2 100644 --- a/nuc/jellyfin-server.md +++ b/nuc/jellyfin-server.md @@ -5,8 +5,10 @@ Jellyfin media **server** in an unprivileged Incus container on `nuc`. - Image: `images:ubuntu/24.04`, Jellyfin from the official repo (repo.jellyfin.org) - IP: `192.168.0.5` (LAN bridge) — web UI/API on `http://192.168.0.5:8096` - iGPU render node (`/dev/dri/renderD128`) passed for QSV/VAAPI hardware transcoding -- Media library: host `/srv/media` (ZFS dataset `usb4t/media`, USB 4 TB) - mounted at `/media` with `shift=true` (needs ZFS ≥ 2.2 for idmapped mounts) +- Media library: host `/srv/media` — since 2026-08-30 an **NFSv4 mount + from `nas`** (`192.168.0.4:/media`, dataset `tank/media`), mounted at + `/media` in the container with **`shift=false`** and `readonly=true`. + See [nas/nas-install.md](../nas/nas-install.md) §8. - Port 8096 additionally proxied to the host address (`web` proxy device) ## Install script @@ -77,6 +79,43 @@ incus restart "$CNAME" echo "Done. Open http://:8096 to run the setup wizard." ``` +## Media over NFS (2026-08-30) + +The library moved to `nas` when the 4 TB left nuc's USB enclosure. The +container keeps the same path, so everything below still applies — only +the mount underneath `/srv/media` changed. + +```sh +# nuc host: /etc/fstab +192.168.0.4:/media /srv/media nfs4 ro,_netdev,soft,timeo=100,retrans=3 0 0 +``` + +`shift=true` **cannot** be used: idmapped mounts are not supported on +NFS (nor CIFS). Per the troubleshooting note below, dropping the shift is +enough for a read-only library — the export uses `all_squash` so files +carry synthetic world-readable ownership: + +```sh +incus stop jellyfin-server # shift cannot be hot-applied +incus config device set jellyfin-server media shift=false +incus config device set jellyfin-server media readonly=true +incus start jellyfin-server +``` + +⚠️ Two traps: + +- **Boot ordering.** Add `remote-fs.target` to nuc's + `/etc/systemd/system/incus.service.d/after-zfs.conf`, or the container + starts against an empty mountpoint and Jellyfin shows an empty library + (and may prune the library metadata). +- **`soft` is deliberate.** A hung nas should fail Jellyfin's reads, not + wedge nuc's processes in uninterruptible sleep the way the suspended + `usb4t` pool did ([usb4t-dropouts.md](usb4t-dropouts.md)). + +`transmission-bt` is no longer on nuc — it moved to nas and writes to +the dataset locally ([nas/transmission-bt.md](../nas/transmission-bt.md)), +so nuc's mount is read-only and there is exactly one writer. + ## First-run configuration 1. Run the setup wizard; add libraries pointing at `/media/...`. diff --git a/nuc/nuc-install.md b/nuc/nuc-install.md index b311f88..1c53293 100644 --- a/nuc/nuc-install.md +++ b/nuc/nuc-install.md @@ -265,6 +265,13 @@ Both logs rotate monthly (`/etc/logrotate.d/incus-*`). the install scripts in the per-container docs include it) - [ ] Host boots to `multi-user.target`, nothing grabs the GPU (required by the jellyfin-client kiosk) +- [ ] ⚠️ **Reboot with the TV/projector disconnected or powered off.** + Booting with the display active kills the i915 probe, which wedges + `snd_hda_intel` in a deferred probe, which blocks incusd in + `sriov_numvfs_show` — **no container starts at all, including + blocky/DNS**. Hotplug the cable back after boot, then + `incus exec jellyfin-client -- systemctl restart jellyfin-kiosk`. + Full diagnosis: [jellyfin-client.md](jellyfin-client.md) - [ ] Jellyfin web at `http://192.168.0.5:8096`, kiosk UI on HDMI, sound on the Pioneer, "Pioneer A-70" visible in Spotify Connect - [ ] LAN DNS: clients use blocky at `192.168.0.254` (host itself uses