Compare commits
3
Commits
a2003a11e8
...
203728674f
| Author | SHA1 | Date | |
|---|---|---|---|
  Julien LutranandClaude Opus 5
|
203728674f | ||
|
Julien LutranandClaude Opus 5
|
66c7bca28f | ||
|
Julien LutranandClaude Opus 5
|
58dcaa5d41 |
@@ -12,7 +12,9 @@ backed up, by which tool, on what schedule, and how to restore.
|
||||
service to ks4 (and getting it backed up automatically).
|
||||
|
||||
**Tech notes**
|
||||
- [`nuc/`](nuc/README.md) — home lab on `nuc`: host, storage, instances
|
||||
- [`nuc/`](nuc/README.md) — home lab on `nuc`: host, iGPU instances
|
||||
- [`nas/`](nas/README.md) — storage + backup host `nas` (Supermicro
|
||||
A1SAi-2750F): the 4 TB on direct SATA, media over NFS, backup pools
|
||||
- [`ks4/`](ks4/README.md) — prod server `ks4` at OVH: host, services, network flows
|
||||
- [`ks2/`](ks2/) — legacy backup server being [decommissioned](ks2/plan.md)
|
||||
- [`archer-c7/`](archer-c7/) — home router (TP-Link Archer C7 v5 running on OpenWrt)
|
||||
|
||||
+2
-2
@@ -8,8 +8,8 @@ few days of margin.
|
||||
|
||||
- [ ] restic S3 leg running nightly for ≥ a week, `done (rc=0)`,
|
||||
restore test passed ([restic-backup.md](../ks4/restic-backup.md) §7)
|
||||
- [ ] nuc pull leg seeded and one instance test-restored
|
||||
([nuc-seed.md](nuc-seed.md))
|
||||
- [ ] **nas** pull leg seeded and one instance test-restored
|
||||
([nas-seed.md](nas-seed.md))
|
||||
- [ ] local leg (sdb5) cron green in `/var/log/incus-copy.log`
|
||||
|
||||
## 1. Cut the last flows to ks2 (root on ks4)
|
||||
|
||||
@@ -0,0 +1,96 @@
|
||||
# ks4 pull leg — seed after FTTH
|
||||
|
||||
Status: **prepared, waiting on the FTTH link.**
|
||||
|
||||
⚠️ **Changed 2026-08-30: this leg lands on `nas`, not on nuc.** It was
|
||||
originally designed for nuc's USB pool `usb4t`, but that pool proved to
|
||||
be the least reliable device in the setup
|
||||
([nuc/usb4t-dropouts.md](../nuc/usb4t-dropouts.md)) — which is exactly
|
||||
what an off-site copy of ks4 must not be. The 4 TB disk moved to direct
|
||||
SATA on the new host `nas` (`192.168.0.4`,
|
||||
[nas/nas-install.md](../nas/nas-install.md)), and the target pool
|
||||
`ks4backup` moved with it.
|
||||
|
||||
Consequences versus the original plan:
|
||||
|
||||
- Target pool `ks4backup` is now backed by `tank/backup/ks4` on nas.
|
||||
- The **WireGuard tunnel moves too**: nas becomes peer `10.8.0.22`;
|
||||
nuc's `wg-ks4` (`10.8.0.20`) is retired once this works. nuc no longer
|
||||
needs a tunnel at all — `transmission-bt`, its only other user, now
|
||||
runs on nas and carries its own in-container tunnel (`10.8.0.21`,
|
||||
unchanged, ks4 needs no edit for it).
|
||||
- ks4's ufw rule is unchanged: traffic arrives masqueraded as the
|
||||
`wireguard` container (`192.168.1.18`) whichever peer sent it.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- [ ] `tank` running on SATA for ≥ 7 days with zero pool suspensions —
|
||||
the gate that replaces "fix the USB enclosure"
|
||||
- [ ] FTTH up (the first pass moves ~1.75 TiB)
|
||||
|
||||
## Setup (root on nas)
|
||||
|
||||
```sh
|
||||
# 1. peer nas on ks4's wireguard container
|
||||
# (run on ks4) — <nas-pubkey> from /etc/wireguard/wg-ks4.key on nas
|
||||
incus exec wireguard -- wg set wg0 peer <nas-pubkey> allowed-ips 10.8.0.22/32
|
||||
incus exec wireguard -- wg-quick save wg0
|
||||
|
||||
# 2. tunnel on nas: /etc/wireguard/wg-ks4.conf, modelled on nuc's
|
||||
# Address = 10.8.0.22/32, peer pubkey TVs6d7…,
|
||||
# Endpoint = 193.70.35.17:51845,
|
||||
# AllowedIPs = 10.8.0.0/24, 192.168.1.1/32, keepalive 25
|
||||
systemctl enable --now wg-quick@wg-ks4
|
||||
|
||||
# 3. incus remote over the tunnel
|
||||
incus remote add ks4 https://192.168.1.1:8443 --accept-certificate --token '…'
|
||||
incus list ks4: | head # sanity: remote reachable
|
||||
```
|
||||
|
||||
## Seed
|
||||
|
||||
```sh
|
||||
# full pull of every ks4 instance into pool ks4backup (tmux — first pass
|
||||
# moves ~1.75 TiB through the WG tunnel)
|
||||
/root/scripts/incus-copy.sh -r ks4 -s ks4backup 2>&1 | tee -a /var/log/incus-copy-ks4.log
|
||||
```
|
||||
|
||||
Notes:
|
||||
|
||||
- First pass is a full send per instance; later refreshes are
|
||||
ZFS-incremental **as long as they run at least every
|
||||
`snapshots.expiry` (7 d on ks4)** — same caveat as
|
||||
[ks4's local leg](../ks4/local-backup-cron.md).
|
||||
- Replicas arrive stopped with `boot.autostart=false` (the script does
|
||||
this) — they must never come up on the LAN with ks4's proxy devices.
|
||||
|
||||
## Cron (after the seed)
|
||||
|
||||
Add to nas's root crontab, offset from the 03:30 nuc→nas push, the
|
||||
04:00 nas→nuc push and ks4's own 01:00/05:00 jobs:
|
||||
|
||||
```cron
|
||||
0 5 * * * /root/scripts/incus-copy.sh -r ks4 -s ks4backup >> /var/log/incus-copy-ks4.log 2>&1
|
||||
```
|
||||
|
||||
## Verification (release gate for ks2)
|
||||
|
||||
```sh
|
||||
incus list --project backup -c ns -f csv # all ks4 instances present
|
||||
# test-restore one instance: copy a replica to the local pool,
|
||||
# start it isolated, check the service answers, then delete it
|
||||
incus copy solar solar-restoretest -s incus
|
||||
incus start solar-restoretest && incus exec solar-restoretest -- systemctl is-system-running
|
||||
incus delete -f solar-restoretest
|
||||
```
|
||||
|
||||
Once verified, tick the nas gate in the [ks2 plan](plan.md) and retire
|
||||
nuc's tunnel:
|
||||
|
||||
```sh
|
||||
# on nuc
|
||||
systemctl disable --now wg-quick@wg-ks4 && rm /etc/wireguard/wg-ks4.conf
|
||||
# on ks4
|
||||
incus exec wireguard -- wg set wg0 peer <nuc-pubkey> remove
|
||||
incus exec wireguard -- wg-quick save wg0
|
||||
```
|
||||
@@ -1,49 +0,0 @@
|
||||
# nuc pull leg — seed after FTTH
|
||||
|
||||
Status: **prepared, waiting on the FTTH link.** Everything is already
|
||||
configured on nuc (see the main [README](../README.md)): incus remote
|
||||
`ks4` over the WireGuard tunnel (`wg-ks4`, 10.8.0.20 → 10.8.0.1),
|
||||
target pool `ks4backup` on the USB ZFS pool (`usb4t/backup/ks4`) —
|
||||
only the seed itself waited on bandwidth.
|
||||
|
||||
## Seed (root on nuc)
|
||||
|
||||
```sh
|
||||
# sanity: remote reachable through the tunnel
|
||||
incus list ks4: | head
|
||||
|
||||
# full pull of every ks4 instance into pool ks4backup (screen/tmux —
|
||||
# first pass moves ~1.7 T through the WG tunnel)
|
||||
/root/scripts/incus-copy.sh -r ks4 -s ks4backup 2>&1 | tee -a /var/log/incus-copy-ks4.log
|
||||
```
|
||||
|
||||
Notes:
|
||||
|
||||
- First pass is a full send per instance; later refreshes are
|
||||
ZFS-incremental **as long as they run at least every
|
||||
`snapshots.expiry` (7 d on ks4)** — same caveat as
|
||||
[ks4's local leg](../ks4/local-backup-cron.md).
|
||||
- Replicas arrive stopped with `boot.autostart=false` (script does
|
||||
this) — they must never come up on the LAN with ks4's proxy devices.
|
||||
|
||||
## Cron (after the seed)
|
||||
|
||||
Add to nuc's root crontab, offset from the 03:30 local nucbackup copy
|
||||
and ks4's own 01:00/05:00 jobs:
|
||||
|
||||
```cron
|
||||
0 5 * * * /root/scripts/incus-copy.sh -r ks4 -s ks4backup >> /var/log/incus-copy-ks4.log 2>&1
|
||||
```
|
||||
|
||||
## Verification (release gate for ks2)
|
||||
|
||||
```sh
|
||||
incus list --project backup 2>/dev/null || incus list | grep -c . # all ks4 instances present
|
||||
# test-restore one instance: copy a replica to the default pool,
|
||||
# start it isolated, check the service answers, then delete it
|
||||
incus copy solar solar-restoretest -s default
|
||||
incus start solar-restoretest && incus exec solar-restoretest -- systemctl is-system-running
|
||||
incus delete -f solar-restoretest
|
||||
```
|
||||
|
||||
Once verified, tick the nuc gate in the [ks2 plan](plan.md).
|
||||
+8
-7
@@ -17,10 +17,11 @@ What remains on the box is **cold history**: instance replicas on pool
|
||||
`/backup/ns3061243` on pool `backup` (last refreshed 2026-08-28),
|
||||
snapshotted daily there (`zfs-auto-snapshot.sh`, 2-month expiry).
|
||||
|
||||
⚠️ While the nuc leg waits for FTTH, instances have no *fresh*
|
||||
⚠️ While the nas leg waits for FTTH, instances have no *fresh*
|
||||
off-site copy — the ks2 push is to be re-enabled as soon as the
|
||||
initial restic sync finishes (decided 2026-08-28), and retired again
|
||||
when nuc takes over.
|
||||
when nas takes over. (The leg moved from nuc to the new host `nas`
|
||||
on 2026-08-30 — see [nas-seed.md](nas-seed.md).)
|
||||
|
||||
## Inventory findings (2026-08-22)
|
||||
|
||||
@@ -51,8 +52,8 @@ when nuc takes over.
|
||||
| local, 2nd disk | `incus-copy.sh -p backup -s backup` → sdb5 pool | **live** (01:00) |
|
||||
| off-site, S3 (data+DB) | **restic** → bucket `restic-data` ([restic-backup.md](../ks4/restic-backup.md)) | **live** (05:00) |
|
||||
| off-site, S3 (instances) | restic over `incus file mount` of the `backup`-project replicas | **shelved** 2026-08-28 (replication covers instances) |
|
||||
| off-site, nuc | nuc pulls `ks4:*` → pool `ks4backup` over WG ([nuc-seed.md](nuc-seed.md)) | waiting FTTH (< Sep 30) |
|
||||
| off-site, ks2 (interim) | `incus-copy.sh -d ks2 -m push` at 02:00 until the nuc leg seeds | to re-enable once the restic seed finishes |
|
||||
| off-site, nas | **nas** pulls `ks4:*` → pool `ks4backup` over WG ([nas-seed.md](nas-seed.md)) | waiting FTTH (< Sep 30); moved off nuc 2026-08-30 |
|
||||
| off-site, ks2 (interim) | `incus-copy.sh -d ks2 -m push` at 02:00 until the nas leg seeds | to re-enable once the restic seed finishes |
|
||||
|
||||
## Actions (backup work documented in [`../ks4/`](../ks4/), ks2-only tasks here)
|
||||
|
||||
@@ -65,9 +66,9 @@ when nuc takes over.
|
||||
([restic-backup.md](../ks4/restic-backup.md), live 2026-08-28; the
|
||||
predecessor's doc is kept as reference)
|
||||
4. ~~instance leg to S3~~ — **shelved 2026-08-28**: instances are
|
||||
protected by replication (sdb + nuc/ks2), their data and configs by
|
||||
protected by replication (sdb + nas/ks2), their data and configs by
|
||||
`restic-data`
|
||||
5. [nuc-seed.md](nuc-seed.md) — **prepared**; after FTTH: seed nuc
|
||||
5. [nas-seed.md](nas-seed.md) — **prepared**; after FTTH: seed the nas
|
||||
pull leg, verify all instances, test-restore one
|
||||
6. [decommission.md](decommission.md) — **prepared**; cut flows,
|
||||
final diff of `/backup/ns3061243`, wipe pools, terminate at OVH
|
||||
@@ -77,7 +78,7 @@ when nuc takes over.
|
||||
- [x] biwiki + spot consciously abandoned (2026-08-22)
|
||||
- [x] local leg cron running since 2026-08-22, **18/18 instances**
|
||||
replicated (verified 2026-08-28)
|
||||
- [ ] nuc leg fully seeded **and** one instance test-restored
|
||||
- [ ] **nas** leg fully seeded **and** one instance test-restored
|
||||
- [x] restic S3 backups live (05:00) **and** restore drill passed
|
||||
2026-08-28: tree restored byte-identical to live, dump restored
|
||||
and loaded into a scratch MariaDB (12/12 tables)
|
||||
|
||||
@@ -0,0 +1,50 @@
|
||||
# Storage: nas
|
||||
|
||||
Storage + backup host on the LAN, added 2026-08.
|
||||
|
||||
- **Always-on host.** nuc is now an on-demand media box (see
|
||||
[nuc/README.md](../nuc/README.md)), so everything that must stay up —
|
||||
LAN DNS, the HTTP proxy, torrents — lives here.
|
||||
- Debian 13, Supermicro **A1SAi-2750F** / Intel Atom C2750 (8 c, 20 W,
|
||||
ECC DDR3) — build procedure: [nas-install.md](nas-install.md)
|
||||
- SSH: `ssh -i id_rsa_claude root@192.168.0.4`
|
||||
- **No iGPU** (Avoton is headless; video is the AST2400 BMC). Anything
|
||||
needing hardware transcoding stays on nuc.
|
||||
- Pools:
|
||||
- `incus` — ZFS mirror across the last partition of both 120 GB SSDs;
|
||||
holds this host's container roots. OS itself is on mdraid RAID1 +
|
||||
ext4 across the same disks (rationale in
|
||||
[nas-install.md](nas-install.md) §2).
|
||||
- `tank` — the 4 TB WD Red, **on direct SATA**. This is the whole
|
||||
point of the box: the disk used to hang off a JMicron USB bridge on
|
||||
nuc that suspended the pool 61 times in 30 days
|
||||
([nuc/usb4t-dropouts.md](../nuc/usb4t-dropouts.md)). Single vdev,
|
||||
accepted — nothing on it is irreplaceable.
|
||||
- `tank/media` → `/export/media`: the media library. Exported
|
||||
**read-only over NFSv4 to nuc**, where `jellyfin-server` reads it;
|
||||
written locally only by `transmission-bt`.
|
||||
- Backups: nuc ↔ nas **cross-replication** (each host's instances live
|
||||
on the other), plus the ks4 pull leg —
|
||||
[nas-install.md](nas-install.md) §9,
|
||||
[ks2/nas-seed.md](../ks2/nas-seed.md).
|
||||
|
||||
## Instances
|
||||
|
||||
| Name | IP | Doc | Features |
|
||||
|---|---|---|---|
|
||||
| blocky | 192.168.0.254 | — | unprivileged, autostart; DNS ad-blocker for the LAN. Moved from nuc 2026-08-30 so it survives nuc being powered off |
|
||||
| privoxy | 192.168.0.11 | — | unprivileged, autostart; filtering HTTP proxy, listens on **:3128** (not privoxy's default 8118); static config in `/etc/systemd/network/eth0.network`, `DNS=192.168.0.254`. Moved from nuc 2026-08-30 |
|
||||
| [transmission-bt](transmission-bt.md) | 192.168.0.7 | ✅ | unprivileged, autostart; always-on WireGuard full tunnel → ks4 (egress = 193.70.35.17, kill switch: no default route); `/export/media` disk device (`shift=true`), downloads to `/media/downloads`; web UI :9091 (LAN only). Moved from nuc 2026-08-30 |
|
||||
|
||||
## Backup pools hosted here
|
||||
|
||||
| incus pool | dataset | receives |
|
||||
|---|---|---|
|
||||
| `nucbackup` | `tank/backup/nuc` | nuc's instances (pushed nightly, 03:30) |
|
||||
| `ks4backup` | `tank/backup/ks4` | ks4's instances (pulled over WG, 05:00 — after FTTH) |
|
||||
| `nasbackup` | `tank/backup/nas` | **nas's own** instances (local copy, 04:00) |
|
||||
|
||||
nas's own instances are replicated **locally** rather than to nuc: nuc is
|
||||
an on-demand box and usually powered off, so it is not a usable backup
|
||||
target. `nasbackup` lives on `tank`, a different pool from the `incus`
|
||||
SSD mirror the instances run on.
|
||||
@@ -0,0 +1,838 @@
|
||||
# nas — build procedure
|
||||
|
||||
New storage host on the LAN (`192.168.0.4`), built 2026-08 from a
|
||||
Supermicro A1SAi-2750F. It exists to solve one specific problem:
|
||||
[usb4t-dropouts.md](../nuc/usb4t-dropouts.md) ruled out the cable and
|
||||
USB power management and left the **JMicron 152d:0578 bridge** as the
|
||||
cause of 61 disconnects in 30 days. The durable fix named there is a
|
||||
direct SATA connection — this box provides six of them.
|
||||
|
||||
Why it matters beyond the media library: `usb4t` is the intended home
|
||||
of the ks4 off-site replicas, and *"the pool holding the off-site copy
|
||||
of ks4 must not be the least reliable device in the setup"*. That makes
|
||||
this build a **release gate for the ks2 decommission**
|
||||
([ks2/plan.md](../ks2/plan.md), deadline Sep 30, 2026).
|
||||
|
||||
It also takes over `transmission-bt` from nuc, so nuc keeps only what
|
||||
needs its iGPU.
|
||||
|
||||
## Hardware
|
||||
|
||||
- Supermicro **A1SAi-2750F** mini-ITX, Intel Atom **C2750** (8 cores,
|
||||
2.4 GHz Silvermont, 20 W SoC), AES-NI, no AVX
|
||||
- RAM: **2× 4 GB DDR3-1600 ECC SO-DIMM fitted = 8 GB** (`Single-bit ECC`
|
||||
confirmed), in DIMMA1/DIMMB1; **2 slots free**. Board takes 32 GB
|
||||
officially, 64 GB with 16 GB modules.
|
||||
⚠️ 8 GB is modest for a 3.6 TB pool — ARC lands around 4 GB. Fine for
|
||||
streaming and replication (neither benefits much from cache), but the
|
||||
first thing to raise if metadata-heavy operations feel slow.
|
||||
- SATA: **2× SATA3 + 4× SATA2** (6 total)
|
||||
- BIOS **2.2** (2019-11-22) as shipped by the RMA
|
||||
- NIC: 4× GbE (Intel i354) + dedicated IPMI LAN
|
||||
- Video: **ASPEED AST2400 BMC only — there is no iGPU.** Avoton is a
|
||||
headless server SoC; `/dev/dri` is empty. That is why `jellyfin-server`
|
||||
stays on nuc (§8).
|
||||
|
||||
### ⚠️ AVR54 — already handled
|
||||
|
||||
The C2750 is on the list of Atom C2000 parts affected by Intel's
|
||||
**AVR54** erratum: the SoC's `LPC_CLKOUT0/1` signals degrade and stop,
|
||||
after which the board never boots again — typically after ~18 months of
|
||||
power-on, i.e. exactly an always-on duty cycle. **This board was RMA'd
|
||||
by Supermicro for that issue and replaced**, so it carries the fix
|
||||
(C0 stepping or the LPC pull-up rework). Recorded here so a dead C2000
|
||||
board is not re-diagnosed from scratch later.
|
||||
|
||||
### Disk plan
|
||||
|
||||
| Port | Device | Role |
|
||||
|---|---|---|
|
||||
| SATA3-0 | Intel `SSDSC2CT120A3` (120 GB) | md mirror + `incus` pool |
|
||||
| SATA3-1 | Toshiba `Q300.` (120 GB) | md mirror + `incus` pool |
|
||||
| SATA2-0 | WD Red 4 TB (moved off the USB enclosure) | pool `tank` |
|
||||
| SATA2-1/2/3 | free | a second 4 TB to mirror `tank`, later |
|
||||
|
||||
The SSDs get the SATA3 ports because they are the only devices that can
|
||||
use them: both negotiate 6 Gb/s and do ~450–500 MB/s, while the WD Red
|
||||
tops out near 180 MB/s and cannot saturate SATA2's ~270 MB/s. A future
|
||||
SLOG would also be fine on SATA2 — it is latency-bound on small sync
|
||||
writes, not bandwidth-bound.
|
||||
|
||||
⚠️ **Disconnect the 4 TB before partitioning the SSDs.** It carries the
|
||||
media library and nuc's replicas, nothing in the OS install needs it,
|
||||
and it keeps the two 120 GB disks unambiguous in the installer's list.
|
||||
Reconnect it before §5b.
|
||||
|
||||
Device letters shift with enumeration order and mean nothing — mdraid
|
||||
assembles from superblock UUIDs, ZFS imports by GUID, fstab and GRUB use
|
||||
UUIDs. **Identify disks by model and serial**
|
||||
(`lsblk -o NAME,SIZE,MODEL,SERIAL`), never by letter: both SSDs are
|
||||
120 GB, so size alone does not tell them apart.
|
||||
|
||||
SSD health measured 2026-08-30 (both read over the JMicron bridge on
|
||||
nuc, `smartctl -d sat`), before deployment:
|
||||
|
||||
| | Intel 330 (25 nm MLC) | Toshiba Q300 (15 nm TLC) |
|
||||
|---|---|---|
|
||||
| host writes | 16.0 TiB | 7.5 TiB |
|
||||
| endurance consumed | **0 %** (`Media_Wearout_Indicator` 100) | **7 %** (`Percentage Used Endurance Indicator`) |
|
||||
| power-on hours | **unreadable** — attr 9 decodes to 914,563 h on this family | 3,877 |
|
||||
| power cycles | 98 | 432 |
|
||||
| defects | 0 reallocated / program-fail / erase-fail | 0 reported uncorrectable |
|
||||
| SMART error log | **not supported** | supported, clean |
|
||||
| device statistics log | absent | full ACS-2 set |
|
||||
| short self-test | passed | passed |
|
||||
| interface CRC errors | n/a | 16 (baseline — watch for growth) |
|
||||
|
||||
Both are healthy and far from wear-out; at OS-disk write rates endurance
|
||||
is not the binding constraint for either. They were previously a
|
||||
**matched pair** — identical layouts, `bpool` / `rpool` / `ubuntu:0`
|
||||
labels from an Ubuntu ZFS-on-root mirror — which is why they go back
|
||||
into a mirror here.
|
||||
|
||||
⚠️ **The Intel is effectively unmonitorable**: no error log, no device
|
||||
statistics, no temperature, no usable hours counter. `smartd` can watch
|
||||
the Toshiba properly and can only ask the Intel "are you still there".
|
||||
Expect the Intel to be found dead rather than found degrading. That
|
||||
asymmetry is the reason for the mirror.
|
||||
|
||||
### `tank` is a single vdev — accepted
|
||||
|
||||
The 4 TB holds the media library, transmission's downloads and the ks4
|
||||
replicas, with no redundancy. A single vdev can *detect* corruption but
|
||||
only self-heal metadata, not data — as seen in the usb4t incident.
|
||||
|
||||
**Decision 2026-08-30: accepted, nothing on `tank` is irreplaceable.**
|
||||
Media and instances are re-fetchable from their sources; the ks4
|
||||
replicas are leg 3 of ks4's 3-2-1 (local `sdb5` + restic/S3 remain —
|
||||
[backup-strategy.md](../backup-strategy.md)). Losing `tank` costs time,
|
||||
not data.
|
||||
|
||||
Three SATA ports stay free, so `zpool attach tank <existing> <new>`
|
||||
turns it into a mirror whenever a spare 4 TB turns up. Not a
|
||||
prerequisite for anything. Scrub weekly regardless — on a single vdev
|
||||
the scrub is the only thing that *tells* you a file has rotted.
|
||||
|
||||
## 1. IPMI and BIOS first
|
||||
|
||||
The AST2400 stack is old and has known vulnerabilities.
|
||||
|
||||
- **Decision 2026-08-30: the BMC is never cabled.** The dedicated IPMI
|
||||
port stays unplugged, so the AST2400's default credentials and its
|
||||
known vulnerabilities are not reachable from anything. This is the
|
||||
simplest correct answer for a box that sits on a flat home LAN — no
|
||||
management VLAN needed, nothing to harden, nothing to patch.
|
||||
- ⚠️ Consequence: **there is no out-of-band console.** A boot that fails
|
||||
before sshd needs a physical monitor and keyboard. Worth knowing before
|
||||
changing anything that affects booting (GRUB, the md arrays, fstab).
|
||||
- BIOS: enable **restore-on-AC-loss**, enable C-states, and disable the
|
||||
three unused i354 NICs (each costs about a watt).
|
||||
|
||||
### Switching this board to UEFI
|
||||
|
||||
There is **no "Boot Mode Select" entry** in this BIOS — that option
|
||||
exists on later Supermicro generations, not here. What works
|
||||
(verified 2026-08-30):
|
||||
|
||||
- **CSM → Disabled**
|
||||
- **All OpROM policies → UEFI** (storage, video *and* network)
|
||||
|
||||
Setting the **video** OpROM to UEFI is safe despite the console being
|
||||
the AST2400 BMC framebuffer — output survives, both on the BMC console
|
||||
and over IPMI KVM.
|
||||
|
||||
⚠️ **Confirm the mode before you partition anything**, from the
|
||||
installer (`Ctrl+Alt+F2`):
|
||||
|
||||
```sh
|
||||
ls /sys/firmware/efi # directory exists = UEFI. Missing = legacy
|
||||
```
|
||||
|
||||
This single check is what makes the difference between a working
|
||||
install and an afternoon lost — see the trap in §3.
|
||||
|
||||
## 2. Prepare the SSDs
|
||||
|
||||
Both report `ATA Security is: Disabled, NOT FROZEN`, so a real secure
|
||||
erase is available — do that rather than just repartitioning. It
|
||||
restores the full spare-block pool on ten-year-old NAND.
|
||||
|
||||
```sh
|
||||
# per disk, from a live system where the disk is NOT the running OS
|
||||
hdparm --user-master u --security-set-pass Eins /dev/sdX
|
||||
hdparm --user-master u --security-erase Eins /dev/sdX
|
||||
hdparm -I /dev/sdX | grep -A2 Security # expect "not enabled"
|
||||
```
|
||||
|
||||
Identical GPT layout on both, sized so ~27 GB (22 %) stays
|
||||
**unallocated** as over-provisioning:
|
||||
|
||||
| Part | Size | Type | Device | Use |
|
||||
|------|------|------|--------|-----|
|
||||
| `sdX1` | 1 GB | EFI System Partition | — | `/boot/efi` (two independent ESPs) |
|
||||
| `sdX2` | 2 GB | Linux RAID | `md0` | `/boot` ext4 |
|
||||
| `sdX3` | 24 GB | Linux RAID | `md1` | `/` ext4 |
|
||||
| `sdX4` | 2 GB | Linux RAID | `md2` | swap |
|
||||
| `sdX5` | 64 GB | Solaris root (bf00) | — | **zpool `incus`** (mirror) |
|
||||
| — | ~27 GB | **unallocated** | — | over-provisioning |
|
||||
|
||||
- ⚠️ **The ESP is deliberately not a RAID1 array.** ks4 has `md1 →
|
||||
/boot/efi` because OVH's installer builds it with mdadm metadata
|
||||
**1.0** (superblock at the *end*, so firmware still sees plain FAT).
|
||||
`debian-installer` only creates metadata **1.2** arrays, whose
|
||||
superblock sits at the start and makes the ESP unreadable to
|
||||
firmware. So: one plain ESP per disk, only one mounted, the second
|
||||
filled by hand (§3). Deviation from ks4 is intentional.
|
||||
- A legacy-BIOS variant of this layout was tried first — 1 MB
|
||||
`bios_grub` instead of the ESP, which makes the mirror simpler
|
||||
(`grub-install` to both disks, nothing to keep in sync). It was
|
||||
abandoned because this board's firmware has no way to prefer legacy
|
||||
targets once CSM is off, and it kept falling through to the UEFI
|
||||
shell. Recorded so it is not retried: **UEFI is the working path
|
||||
here.**
|
||||
- `/` at 24 GB matches the other hosts (nuc 46 GB, ks4 40 GB) — this box
|
||||
has no desktop and no container roots on `/`.
|
||||
- Mixing md partitions and a ZFS partition on the same disks is exactly
|
||||
what ks4 does (`md1/2/3` + ZFS on `sda5`/`sdb5`).
|
||||
|
||||
### Why not full root-on-ZFS
|
||||
|
||||
Decision 2026-08-30. These SSDs previously ran Ubuntu 20.04's
|
||||
experimental ZFS-root installer (hence the leftover `bpool` / `rpool`
|
||||
labels), so the option was on the table. Rejected because:
|
||||
|
||||
- **Debian's installer cannot do it.** ZFS is CDDL, shipped only in
|
||||
`contrib` as `zfs-dkms`; `debian-installer` can neither partition nor
|
||||
boot from ZFS. Root-on-ZFS means the manual
|
||||
[OpenZFS Debian Trixie HOWTO](https://openzfs.github.io/openzfs-docs/Getting%20Started/Debian/Debian%20Trixie%20Root%20on%20ZFS.html)
|
||||
— ~60 steps from a live ISO. Per the repo convention the doc *is* the
|
||||
rebuild procedure, and that is a bad thing to be executing during an
|
||||
actual failure.
|
||||
- **DKMS failure mode.** `zfs-dkms` rebuilds on every kernel upgrade.
|
||||
If that build fails, root-on-ZFS means the box **does not boot**;
|
||||
with an ext4 md root it boots normally and only the pools are
|
||||
missing — recoverable over IPMI with a shell.
|
||||
- GRUB's ZFS support lags OpenZFS, which is why every root-on-ZFS guide
|
||||
needs a separate feature-limited `bpool`; `zpool upgrade bpool` is a
|
||||
known way to make a machine unbootable.
|
||||
- Ubuntu's version of this is a dead end anyway: the installer option
|
||||
was nearly dropped in 22.04 and `zsys`, which made boot environments
|
||||
useful, is abandoned.
|
||||
|
||||
What root-on-ZFS would buy — snapshot and roll back a bad upgrade — is
|
||||
already covered where the state actually lives: container roots get
|
||||
incus snapshots plus nightly replication (§9). The host is 24 GB of
|
||||
packages reproducible from this file. Accepted cost: no pre-upgrade
|
||||
rollback of the host itself (`etckeeper` covers `/etc` if wanted).
|
||||
|
||||
## 3. Install Debian 13 (trixie)
|
||||
|
||||
Netinst ISO (burned 2026-08-30, sha256 `65273bee…664e7`, verified
|
||||
against `cdimage.debian.org/debian-cd/13.6.0/amd64/iso-cd/SHA256SUMS`).
|
||||
|
||||
- Manual partitioning per the table above: `sdX1` as ESP, `sdX2..4` as
|
||||
RAID1 members (three arrays), `sdX5` left untouched.
|
||||
- Tasks: **SSH server + standard utilities only**. No desktop — there is
|
||||
no GPU and the console is a BMC framebuffer.
|
||||
- Sources: `main contrib non-free-firmware` (`contrib` is required by
|
||||
`zfs-dkms`; the installer does not offer it — add it after first boot).
|
||||
|
||||
Confirm d-i picked the right bootloader once installed:
|
||||
`dpkg -l | grep grub-efi` — `grub-efi-amd64`, not `grub-pc`.
|
||||
|
||||
### ⚠️ Trap: installer boot mode decides the bootloader (hit 2026-08-30)
|
||||
|
||||
`debian-installer` chooses `grub-pc` or `grub-efi-amd64` from **how the
|
||||
installer itself booted**, not from what the disks look like. Booting the
|
||||
USB stick in legacy mode while the firmware prefers UEFI produces:
|
||||
|
||||
1. d-i installs `grub-pc`, targeting the MBR;
|
||||
2. on a **GPT** disk that needs a 1 MB `bios_grub` partition — absent
|
||||
here, so `grub-install` fails, easy to click past;
|
||||
3. the firmware then tries the SSDs as UEFI targets, finds no `.efi`
|
||||
binary, and drops to the **UEFI shell**.
|
||||
|
||||
Nothing is corrupt; the halves simply disagree. Symptoms and checks:
|
||||
|
||||
```sh
|
||||
[ -d /sys/firmware/efi ] && echo UEFI || echo legacy # in the installer
|
||||
lsblk -no PTTYPE,PARTTYPENAME /dev/sdX # gpt + "EFI System"?
|
||||
dd if=/dev/sdX bs=440 count=1 2>/dev/null | od -c | head -3 # all \0 = no boot code
|
||||
dpkg -l | grep -E '^ii.*grub-(pc|efi)' # which flavour got installed
|
||||
```
|
||||
|
||||
The board's hybrid ISO offers both paths, so the F11 boot menu usually
|
||||
lists the stick twice — picking the **`UEFI:`** entry avoids the whole
|
||||
thing. Checking `/sys/firmware/efi` before partitioning is the one step
|
||||
that prevents it.
|
||||
|
||||
### Second ESP — do this before trusting the mirror
|
||||
|
||||
The installer populates only the ESP it mounted. Until the second one is
|
||||
written, losing that disk means the box does not boot, mirror or no
|
||||
mirror.
|
||||
|
||||
⚠️ **Use `/dev/disk/by-id/`, never `/dev/sdX`.** Reconnecting the 4 TB
|
||||
after the install shifts every letter — observed 2026-08-30: the HDD on
|
||||
SATA2-0 takes `sda` even with the SSDs on SATA3, because the SATA2
|
||||
controller enumerates first on this SoC. A bare `/dev/sdb1` written
|
||||
during the install then points at a *different disk*, and here that
|
||||
would mean reformatting the ESP the system actually boots from.
|
||||
|
||||
```sh
|
||||
ESP2=/dev/disk/by-id/ata-TOSHIBA_Q300._36OB318OK1KU-part1 # the one NOT at /boot/efi
|
||||
mkfs.vfat -F32 "$ESP2"
|
||||
mkdir -p /boot/efi2 && mount "$ESP2" /boot/efi2
|
||||
grub-install --target=x86_64-efi --efi-directory=/boot/efi2 \
|
||||
--bootloader-id=debian-b --recheck
|
||||
efibootmgr -v # expect: debian, debian-b
|
||||
echo "UUID=$(blkid -s UUID -o value $ESP2) /boot/efi2 vfat umask=0077 0 1" >> /etc/fstab
|
||||
mount -a && findmnt /boot/efi2
|
||||
```
|
||||
|
||||
Re-run the `grub-install` after any GRUB or kernel change — the second
|
||||
ESP is **not** kept in sync automatically. **Verify by pulling one disk
|
||||
and booting**; the checklist item exists because an untested mirror is a
|
||||
guess, and this is the component most likely to be silently wrong.
|
||||
|
||||
Do not fix the letter ordering by moving cables: the only arrangement
|
||||
that makes the SSDs `sda`/`sdb` puts them on SATA2 and the HDD on SATA3,
|
||||
which caps the only devices that can use 6 Gb/s and gives the bandwidth
|
||||
to a disk that tops out near 180 MB/s.
|
||||
|
||||
## 4. Base system
|
||||
|
||||
```sh
|
||||
apt update && apt full-upgrade -y
|
||||
apt install -y \
|
||||
linux-headers-amd64 zfs-dkms zfsutils-linux zfs-zed \
|
||||
mdadm smartmontools nfs-kernel-server \
|
||||
msmtp msmtp-mta bsd-mailx \
|
||||
curl vim htop ripgrep sysstat dmidecode pciutils usbutils \
|
||||
stress-ng fio
|
||||
```
|
||||
|
||||
Static network — `/etc/network/interfaces` (ifupdown, matching nuc):
|
||||
|
||||
```
|
||||
source /etc/network/interfaces.d/*
|
||||
|
||||
auto lo
|
||||
iface lo inet loopback
|
||||
|
||||
allow-hotplug enp0s20f0
|
||||
iface enp0s20f0 inet static
|
||||
address 192.168.0.4
|
||||
netmask 255.255.255.0
|
||||
gateway 192.168.0.2
|
||||
dns-nameservers 1.1.1.1 9.9.9.9
|
||||
```
|
||||
|
||||
(Gateway is **`192.168.0.2`** — `.1` is gone since 2026-08. The host
|
||||
uses public resolvers, never blocky, to avoid a bootstrap loop.
|
||||
Interface name is a guess until the board is up — check `ip -br link`.)
|
||||
|
||||
Restore `/root/.ssh/authorized_keys` (incl. `id_rsa_claude.pub`) and
|
||||
`timedatectl set-timezone Europe/Paris`.
|
||||
|
||||
## 5. Pools
|
||||
|
||||
### 5a. `incus` — SSD mirror
|
||||
|
||||
```sh
|
||||
zpool create -o ashift=12 \
|
||||
-O compression=zstd -O atime=off -O xattr=sa -O acltype=posixacl \
|
||||
incus mirror \
|
||||
/dev/disk/by-id/ata-INTEL_SSDSC2CT120A3_CVMP250400ES120BGN-part5 \
|
||||
/dev/disk/by-id/ata-TOSHIBA_Q300._36OB318OK1KU-part5
|
||||
zpool set autotrim=on incus
|
||||
```
|
||||
|
||||
Weekly scrubs come from the packaged systemd timers rather than cron:
|
||||
|
||||
```sh
|
||||
systemctl enable --now zfs-scrub-weekly@tank.timer zfs-scrub-weekly@incus.timer
|
||||
systemctl list-timers 'zfs-scrub*'
|
||||
```
|
||||
|
||||
`autotrim` matters on ten-year-old NAND — it is what keeps the
|
||||
unallocated 22 % actually available to the controller as spare.
|
||||
|
||||
### 5b. `tank` — move the 4 TB off USB onto SATA
|
||||
|
||||
The risky step. The pool holds `usb4t/backup/nuc` (nuc's replicas),
|
||||
`usb4t/backup/ks4` (empty, awaiting the FTTH seed) and `usb4t/media`.
|
||||
|
||||
⚠️ **Between export and import, nuc has no replica target and Jellyfin
|
||||
has no media.** Plan a maintenance window and disable nuc's 03:30 cron
|
||||
first, so it fails loudly rather than half-running.
|
||||
|
||||
```sh
|
||||
# --- on nuc, first ---
|
||||
zpool scrub usb4t # start clean; wait for it
|
||||
zpool status usb4t
|
||||
zpool export usb4t
|
||||
```
|
||||
|
||||
Move the disk to SATA2-0 (reconnect it now if you unplugged it for the
|
||||
install), then:
|
||||
|
||||
```sh
|
||||
# --- on nas ---
|
||||
zpool import # confirm it is seen
|
||||
zpool import usb4t tank # rename: it is not USB any more
|
||||
zpool set cachefile=/etc/zfs/zpool.cache tank
|
||||
zpool status tank
|
||||
```
|
||||
|
||||
Properties survive from creation (`ashift=12`, `compression=zstd`,
|
||||
`atime=off`, `xattr=sa`, `acltype=posixacl`). Re-point the mountpoints
|
||||
and add the two new backup datasets:
|
||||
|
||||
```sh
|
||||
zfs set mountpoint=/export/media tank/media # NFSv4 export root (§8)
|
||||
zfs set mountpoint=none tank/backup
|
||||
zfs list -o name,used,avail,mountpoint
|
||||
```
|
||||
|
||||
Resulting layout:
|
||||
|
||||
```
|
||||
incus mirror, 2× SSD — nas's own container roots
|
||||
tank 4 TB, single vdev
|
||||
├── tank/media → /export/media NFS ro → nuc; local device → transmission-bt
|
||||
└── tank/backup
|
||||
├── tank/backup/nuc → incus pool `nucbackup` (nuc pushes here)
|
||||
└── tank/backup/ks4 → incus pool `ks4backup` (nas pulls from ks4)
|
||||
|
||||
nas's own replicas live on **nuc** (`data/backup/nas` → pool `nasbackup`),
|
||||
not here — see §9.
|
||||
```
|
||||
|
||||
Backup pools are named after the **source** host, matching
|
||||
[ks4/incus-copy.md](../ks4/incus-copy.md). nas's *own* instances are not
|
||||
backed up here — they cross-replicate to nuc (§9b), so neither host's
|
||||
instances depend on that host surviving.
|
||||
|
||||
Once `tank` has run a week on SATA with **zero** pool suspensions and no
|
||||
CRC errors, the `usb4t-dropouts` gate is cleared — record that in
|
||||
[ks2/plan.md](../ks2/plan.md).
|
||||
|
||||
## 6. Incus
|
||||
|
||||
Same Zabbly stable repo as nuc and ks4:
|
||||
|
||||
```sh
|
||||
mkdir -p /etc/apt/keyrings
|
||||
curl -fsSL https://pkgs.zabbly.com/key.asc -o /etc/apt/keyrings/zabbly.asc
|
||||
cat > /etc/apt/sources.list.d/zabbly-incus-stable.sources <<EOF
|
||||
Enabled: yes
|
||||
Types: deb
|
||||
URIs: https://pkgs.zabbly.com/incus/stable
|
||||
Suites: trixie
|
||||
Components: main
|
||||
Architectures: amd64
|
||||
Signed-By: /etc/apt/keyrings/zabbly.asc
|
||||
EOF
|
||||
apt update && apt install -y incus
|
||||
```
|
||||
|
||||
macvlan like nuc, so instances get real LAN addresses — `transmission-bt`
|
||||
keeps `192.168.0.7` when it moves:
|
||||
|
||||
```sh
|
||||
cat <<EOF | incus admin init --preseed
|
||||
config:
|
||||
core.https_address: :8443
|
||||
storage_pools:
|
||||
- name: incus
|
||||
driver: zfs
|
||||
config:
|
||||
source: incus
|
||||
networks:
|
||||
- name: macvlan
|
||||
type: macvlan
|
||||
config:
|
||||
parent: enp0s20f0
|
||||
profiles:
|
||||
- name: default
|
||||
devices:
|
||||
eth0: {name: eth0, network: macvlan, type: nic}
|
||||
root: {path: /, pool: incus, type: disk}
|
||||
EOF
|
||||
|
||||
incus profile set default snapshots.schedule="0 3 * * *" snapshots.expiry=7d
|
||||
```
|
||||
|
||||
Same macvlan quirk as nuc: **the host cannot talk to its own instances**,
|
||||
and vice versa. Test container services from another LAN host or from
|
||||
inside the container, never from `nas`.
|
||||
|
||||
Backup pools and the replica project:
|
||||
|
||||
```sh
|
||||
incus storage create nucbackup zfs source=tank/backup/nuc
|
||||
incus storage create ks4backup zfs source=tank/backup/ks4
|
||||
incus project create backup -c features.images=false -c features.profiles=false
|
||||
```
|
||||
|
||||
⚠️ **`tank/backup/nuc` already contains nuc's replicas** — they came
|
||||
across with the pool. Re-register them so refreshes stay
|
||||
ZFS-incremental instead of re-sending everything (the homeassistant VM
|
||||
alone is a 50 GiB volume):
|
||||
|
||||
```sh
|
||||
incus admin recover # point it at pool nucbackup; project backup
|
||||
incus list --project backup
|
||||
```
|
||||
|
||||
Same call nuc-install.md §5 uses after a rebuild. If `recover` is
|
||||
skipped, the first push in §9a silently becomes a full re-send of every
|
||||
nuc instance.
|
||||
|
||||
Order incus after the ZFS mounts, as on nuc —
|
||||
`/etc/systemd/system/incus.service.d/after-zfs.conf`:
|
||||
|
||||
```ini
|
||||
[Unit]
|
||||
After=zfs-mount.service zfs.target
|
||||
```
|
||||
|
||||
## 7. Move `transmission-bt` from nuc
|
||||
|
||||
Its WireGuard tunnel is **entirely inside the container** (`wg0`,
|
||||
`10.8.0.21`, `wg-quick@wg0`, `BindsTo=` on the daemon), so the container
|
||||
carries its own keys and **ks4 needs no change at all** — the peer stays
|
||||
`10.8.0.21/32`. The kill-switch `/32` route points at the gateway
|
||||
`192.168.0.2`, which is the same from here.
|
||||
|
||||
```sh
|
||||
# on nuc — remote already added in §9a
|
||||
incus stop transmission-bt
|
||||
incus move transmission-bt nas: --storage incus
|
||||
```
|
||||
|
||||
Then on nas, re-point the media device at the local dataset — this is a
|
||||
plain `shift=true` device again, because the data is local ZFS:
|
||||
|
||||
```sh
|
||||
incus config device remove transmission-bt media
|
||||
incus config device add transmission-bt media disk \
|
||||
source=/export/media path=/media shift=true
|
||||
incus config set transmission-bt boot.autostart=true
|
||||
incus start transmission-bt
|
||||
```
|
||||
|
||||
Verify the tunnel and the kill switch before trusting it:
|
||||
|
||||
```sh
|
||||
incus exec transmission-bt -- wg show
|
||||
incus exec transmission-bt -- curl -s ifconfig.me # must print 193.70.35.17
|
||||
incus exec transmission-bt -- ip route # must have NO default route
|
||||
```
|
||||
|
||||
⚠️ **The watch-folder workflow moves with it** —
|
||||
[transmission-bt.md](transmission-bt.md) says
|
||||
`scp some.torrent root@192.168.0.3:/srv/media/.watchdir/`; it is now
|
||||
`root@192.168.0.4:/export/media/.watchdir/`.
|
||||
|
||||
## 8. Media over NFS — `jellyfin-server` stays on nuc
|
||||
|
||||
`jellyfin-server` needs the Alder Lake-N iGPU for QSV/VAAPI; the C2750
|
||||
has no render device at all, and software transcoding on Silvermont
|
||||
manages 1–2 concurrent 1080p H.264 streams at best. So the container
|
||||
stays on nuc and reaches the library over NFS, **read-only** —
|
||||
`transmission-bt` is the only writer and it now lives here.
|
||||
|
||||
```sh
|
||||
# --- nas: NFSv4 export, read-only, nuc only ---
|
||||
cat >> /etc/exports <<'EOF'
|
||||
/export 192.168.0.3(ro,fsid=0,crossmnt,no_subtree_check)
|
||||
/export/media 192.168.0.3(ro,no_subtree_check,all_squash,anonuid=65534,anongid=65534)
|
||||
EOF
|
||||
exportfs -ra && exportfs -v
|
||||
```
|
||||
|
||||
```sh
|
||||
# --- nuc: mount at the SAME path, so jellyfin-server.md still applies ---
|
||||
mkdir -p /srv/media
|
||||
echo '192.168.0.4:/media /srv/media nfs4 ro,_netdev,soft,timeo=100,retrans=3 0 0' >> /etc/fstab
|
||||
mount /srv/media && ls /srv/media
|
||||
```
|
||||
|
||||
The container device changes only in losing the shift — per
|
||||
[nuc/jellyfin-server.md](../nuc/jellyfin-server.md)'s own troubleshooting
|
||||
note, *"on CIFS files are world-readable synthetic ownership, enough for
|
||||
a read-only library"*; the same holds for NFS with `all_squash`:
|
||||
|
||||
```sh
|
||||
incus stop jellyfin-server # shift cannot be hot-applied
|
||||
incus config device set jellyfin-server media shift=false
|
||||
incus config device set jellyfin-server media readonly=true
|
||||
incus start jellyfin-server
|
||||
incus exec jellyfin-server -- ls /media # must list the library
|
||||
```
|
||||
|
||||
Requirements for that to work: media files must be **world-readable**
|
||||
(`find /export/media -type f ! -perm -o=r`), and transmission must keep
|
||||
creating them that way (it does — `umask`/`0775` per its doc).
|
||||
|
||||
⚠️ Boot ordering on nuc: incus is already ordered after
|
||||
`zfs-mount.service`; add `remote-fs.target` to that drop-in, or
|
||||
`jellyfin-server` starts against an empty mountpoint and shows an empty
|
||||
library.
|
||||
|
||||
⚠️ `soft` mount is deliberate: a hung NAS should fail Jellyfin's reads,
|
||||
not wedge nuc's processes in uninterruptible sleep the way the suspended
|
||||
`usb4t` pool did.
|
||||
|
||||
## 9. Backup legs
|
||||
|
||||
The driver is unchanged —
|
||||
[`incus-copy.sh`](https://git.lutran.fr/julien/scripts/src/branch/main/incus-copy.sh),
|
||||
deployed to `/root/scripts` as everywhere else.
|
||||
|
||||
### 9a. nuc's instances -> nas (systemd timer, not cron)
|
||||
|
||||
⚠️ **nuc is an on-demand media box** (see [nuc/README.md](../nuc/README.md)):
|
||||
since 2026-08-30 it only runs when watching Jellyfin or using the Spotify
|
||||
kiosk, so it is usually powered off at 03:30. Cron cannot catch up a
|
||||
missed window; a **systemd timer with `Persistent=true`** runs the job
|
||||
shortly after the next boot. That is the whole reason this is a timer.
|
||||
|
||||
`/etc/systemd/system/incus-copy.service` on nuc:
|
||||
|
||||
```ini
|
||||
[Unit]
|
||||
Description=Replicate all nuc instances to nas (project backup, pool nucbackup)
|
||||
After=network-online.target incus.service
|
||||
Wants=network-online.target
|
||||
Requires=incus.service
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
ExecStart=/bin/sh -c "/root/scripts/incus-copy.sh -d nas -m push -s nucbackup -p backup >> /var/log/incus-copy.log 2>&1"
|
||||
```
|
||||
|
||||
`/etc/systemd/system/incus-copy.timer`:
|
||||
|
||||
```ini
|
||||
[Timer]
|
||||
OnCalendar=03:30
|
||||
Persistent=true
|
||||
RandomizedDelaySec=300
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
```
|
||||
|
||||
```sh
|
||||
systemctl enable --now incus-copy.timer && systemctl list-timers incus-copy
|
||||
```
|
||||
|
||||
⚠️ **`-p backup` is not optional.** Without it the replicas land in
|
||||
`default` on nas and collide with nas's *live* instances — both hosts are
|
||||
on the same macvlan LAN and the replicas carry the same static IPs.
|
||||
|
||||
### 9b. nas's own instances -> local pool `nasbackup`
|
||||
|
||||
**Decision 2026-08-30: local, not cross-replicated.** nuc is powered off
|
||||
most of the time, so it is not a usable backup target — a nightly push to
|
||||
it would fail noisily and, once mail works, alarm every morning. The
|
||||
replicas instead go to `tank/backup/nas`, which is a **different pool**
|
||||
from the instances themselves (`incus`, the SSD mirror), so it survives
|
||||
losing that mirror. All three instances here are rebuildable from their
|
||||
docs, so same-host is proportionate.
|
||||
|
||||
```sh
|
||||
zfs create tank/backup/nas
|
||||
zfs set mountpoint=legacy tank/backup/nas # see the trap below
|
||||
incus storage create nasbackup zfs source=tank/backup/nas
|
||||
```
|
||||
|
||||
```cron
|
||||
# nas, /etc/cron.d/incus-copy
|
||||
0 4 * * * root /root/scripts/incus-copy.sh -p backup -s nasbackup >> /var/log/incus-copy.log 2>&1
|
||||
```
|
||||
|
||||
⚠️ **Trap: `incus storage create` hangs forever on a `mountpoint=none`
|
||||
dataset.** `tank/backup` is set to `mountpoint=none`, so any child
|
||||
created afterwards inherits it, and the pool create then blocks with no
|
||||
error and no entry in `incus operation list` — it looks exactly like I/O
|
||||
contention (a scrub was running, which sent me down that path for 20
|
||||
minutes). Set the child to `legacy` to match its siblings first, and it
|
||||
completes instantly.
|
||||
|
||||
### 9c. Cleanup on nuc after the pool move
|
||||
|
||||
|
||||
Exporting `usb4t` leaves nuc with an incus storage pool whose backing
|
||||
dataset is gone, plus replica records in the `backup` project pointing
|
||||
at it. Remove them once §6's `incus admin recover` has re-registered the
|
||||
same volumes on nas — **verify there first, then delete here**:
|
||||
|
||||
```sh
|
||||
# on nas: confirm the replicas are registered
|
||||
incus list --project backup -c ns -f csv
|
||||
|
||||
# on nuc: only then
|
||||
incus delete --project backup --force <each-replica>
|
||||
incus storage delete nucbackup
|
||||
zpool status # only `data` should remain
|
||||
```
|
||||
|
||||
### 9d. ks4 pull leg moves from nuc to nas
|
||||
|
||||
The leg [ks2/nas-seed.md](../ks2/nas-seed.md) prepared. The target pool
|
||||
moves, so the **host** WireGuard tunnel moves too — nuc drops `wg-ks4`
|
||||
entirely once this works (its only other tunnel user, `transmission-bt`,
|
||||
is now here and carries its own).
|
||||
|
||||
- **ks4**: add nas as a peer on the `wireguard` container —
|
||||
`wg set wg0 peer <nas-pubkey> allowed-ips 10.8.0.22/32 && wg-quick save wg0`.
|
||||
The existing ufw rule (`allow in on incusbr0 from 192.168.1.18 to any
|
||||
port 8443 proto tcp`) already covers it: traffic arrives masqueraded as
|
||||
the WG container whichever peer sent it.
|
||||
- **nas**: `/etc/wireguard/wg-ks4.conf` modelled on nuc's —
|
||||
`Address = 10.8.0.22/32`, peer pubkey `TVs6d7…`,
|
||||
`Endpoint = 193.70.35.17:51845`,
|
||||
`AllowedIPs = 10.8.0.0/24, 192.168.1.1/32`, keepalive 25 — then
|
||||
`systemctl enable --now wg-quick@wg-ks4`.
|
||||
- `incus remote add ks4 https://192.168.1.1:8443 --accept-certificate --token '…'`
|
||||
(cross-check the fingerprint against the token).
|
||||
- **nuc, after verification**: `systemctl disable --now wg-quick@wg-ks4`,
|
||||
remove `/etc/wireguard/wg-ks4.conf`, and drop the `10.8.0.20/32` peer
|
||||
on ks4.
|
||||
|
||||
Seed manually in tmux — the first pass is **~1.75 TiB** over the home
|
||||
downlink and only makes sense once FTTH is up:
|
||||
|
||||
```sh
|
||||
/root/scripts/incus-copy.sh -r ks4 -s ks4backup 2>&1 | tee -a /var/log/incus-copy-ks4.log
|
||||
```
|
||||
|
||||
Then test-restore one instance before ticking the gate in
|
||||
[ks2/plan.md](../ks2/plan.md).
|
||||
|
||||
### Resulting schedule
|
||||
|
||||
| When | Host | What |
|
||||
|---|---|---|
|
||||
| 03:00 | nuc, nas | instance snapshots (profile) |
|
||||
| 03:30 | nuc | push all instances → `nas:nucbackup` (**systemd timer**, `Persistent=true` — catches up after boot) |
|
||||
| 04:00 | nas | local copy of nas instances → `nasbackup` (`tank/backup/nas`) |
|
||||
| 05:00 | nas | pull `ks4:*` → `ks4backup` (after FTTH) |
|
||||
| 05:30 | nuc | apt upgrade all containers |
|
||||
| Mon ~00:12 | nas | `zfs-scrub-weekly@tank.timer` / `@incus.timer` (systemd, not cron) |
|
||||
|
||||
Staggered around ks4's own 01:00 / 05:00 jobs.
|
||||
|
||||
⚠️ Keep `snapshots.schedule` on every source. A refresh with no common
|
||||
snapshot degrades to a **full re-send** — the failure mode that cost
|
||||
933 G on ks4 ([ks4/local-backup-cron.md](../ks4/local-backup-cron.md)).
|
||||
The homeassistant **VM** on nuc re-sends its whole volume without them.
|
||||
Note `incus-copy.sh` takes a **global** `flock`: an overrunning ks4 pull
|
||||
aborts that night's other run loudly rather than racing it.
|
||||
|
||||
## 10. Monitoring
|
||||
|
||||
Do not repeat nuc's 11-day blind spot
|
||||
([usb4t-dropouts.md](../nuc/usb4t-dropouts.md) — zed was running, but the
|
||||
host had no MTA).
|
||||
|
||||
**Status: live since 2026-08-30**, verified end to end (`smtpstatus=250`).
|
||||
|
||||
- **msmtp**, same shape as nuc: this box shares the dynamic home IP with
|
||||
no PTR and no SPF alignment, so it must use **submission (587) with
|
||||
auth**, not port 25 — rspamd rejects the port-25 path as spam.
|
||||
`/etc/msmtprc` mode 600, host `mail.lutran.fr`, STARTTLS. Give SMTP
|
||||
tests ≥ 30 s: the missing PTR delays the greeting.
|
||||
The `zed@lutran.fr` account is the **same credential as nuc** — it is
|
||||
SMTP-AUTH, not IP-bound, so `/etc/msmtprc` can simply be copied between
|
||||
hosts (mode 600, root:root). The password lives in the password
|
||||
manager; no host-specific setup is needed.
|
||||
- **zed**: `/etc/zfs/zed.d/zed.rc` mode 600 with
|
||||
`ZED_EMAIL_ADDR="julien@lutran.fr"`, `ZED_EMAIL_PROG="mail"`,
|
||||
`ZED_NOTIFY_VERBOSE=1`, **`ZED_NOTIFY_DATA=1`**,
|
||||
`ZED_NOTIFY_INTERVAL_SECS=3600`.
|
||||
- **[`zpool-health.sh`](https://git.lutran.fr/julien/scripts/src/branch/main/zpool-health.sh)**
|
||||
every 15 min via `/etc/cron.d/zpool-health` — zed does *not* report a
|
||||
suspended pool (the vdev stays `ONLINE`, so `statechange-notify.sh`
|
||||
never fires). That watchdog is the only thing that catches the exact
|
||||
failure this box was built to prevent.
|
||||
- **smartd** — `/etc/smartd.conf`. The Toshiba carries the useful
|
||||
attributes; the Intel is a liveness check only:
|
||||
|
||||
```
|
||||
# Toshiba Q300 — endurance + temperature are real here
|
||||
/dev/disk/by-id/ata-TOSHIBA_Q300._36OB318OK1KU -a -o on -S on -s (S/../.././02|L/../../6/03) -W 4,50,55 -m julien@lutran.fr
|
||||
# Intel 330 — no error log, no devstat, no temperature: liveness only
|
||||
/dev/disk/by-id/ata-INTEL_SSDSC2CT120A3_CVMP250400ES120BGN -H -s (S/../.././03|L/../../6/04) -m julien@lutran.fr
|
||||
# WD Red
|
||||
/dev/disk/by-id/ata-WDC_WD40EFRX-68WT0N0_WD-WCC4E6NLPJJE -a -o on -S on -s (S/../.././04|L/../../6/05) -W 4,45,50 -m julien@lutran.fr
|
||||
```
|
||||
|
||||
Verify the whole chain the day you build it, not the day you need it:
|
||||
|
||||
```sh
|
||||
/root/scripts/zpool-health.sh -m julien@lutran.fr -t
|
||||
tail -2 /var/log/msmtp.log # expect smtpstatus=250
|
||||
smartctl -d sat -l devstat /dev/sdX | grep -i endurance
|
||||
```
|
||||
|
||||
## 11. Power baseline
|
||||
|
||||
Take the measurement **before** the box goes into service, so later
|
||||
readings mean something. Expect roughly 25–35 W idle with three disks:
|
||||
the 20 W SoC plus a BMC drawing several watts continuously, even at
|
||||
soft-off.
|
||||
|
||||
```sh
|
||||
# baseline: 10 min idle, everything settled
|
||||
# CPU in steps — the interesting curve for an always-on box
|
||||
for l in 25 50 75 100; do echo "=== ${l}% $(date +%s)"; stress-ng --cpu 0 --cpu-load $l --timeout 120s; done
|
||||
# disk: random I/O is what moves an HDD's power, not throughput
|
||||
fio --name=rr --directory=/export/media --size=20G --rw=randread --bs=4k \
|
||||
--iodepth=32 --numjobs=4 --ioengine=libaio --direct=1 --runtime=300 --time_based
|
||||
```
|
||||
|
||||
Measure at the wall (PDU or smart plug) — RAPL is unreliable on Avoton
|
||||
and sees neither disks nor fans. Log epoch timestamps per step so the
|
||||
trace can be recut against the meter's series afterwards.
|
||||
|
||||
## 12. Post-install checklist
|
||||
|
||||
- [x] IPMI **left unplugged by decision** (2026-08-30) — no BMC on the
|
||||
LAN, therefore no out-of-band console either
|
||||
- [ ] `cat /proc/mdstat` — all three arrays `[UU]`; GRUB written to **both**
|
||||
ESPs (`efibootmgr -v` shows two entries),
|
||||
and boot tested with one disk unplugged
|
||||
- [ ] `zpool status` healthy for `incus` and `tank`; weekly scrubs
|
||||
scheduled; both import cleanly after a reboot (`cachefile` set)
|
||||
- [ ] **7 days with zero pool suspensions and zero CRC errors** — the
|
||||
gate that closes [usb4t-dropouts.md](../nuc/usb4t-dropouts.md)
|
||||
- [x] `smartd` monitoring all 3 disks; `zpool-health.sh -t` mail
|
||||
delivered (`smtpstatus=250` in `/var/log/msmtp.log`) — done
|
||||
2026-08-30
|
||||
- [ ] `transmission-bt` on nas: egress is `193.70.35.17`, **no default
|
||||
route**, downloads land in `/export/media/downloads`, watch folder
|
||||
works from the new path
|
||||
- [ ] `jellyfin-server` on nuc lists the library over NFS after a **cold
|
||||
reboot of both hosts** (the boot-ordering trap)
|
||||
- [ ] `incus admin recover` ran on nas **before** the first push, so the
|
||||
inherited `tank/backup/nuc` replicas refresh incrementally instead
|
||||
of re-sending (check the first run's duration, not just `rc=0`)
|
||||
- [ ] nuc's 03:30 leg → `nas:nucbackup`, `rc=0`, all nuc instances
|
||||
present (`incus list --project backup -c ns -f csv` on nas)
|
||||
- [ ] nas's 04:00 local leg → `nasbackup`, `rc=0`, and **blocky,
|
||||
privoxy and transmission-bt all appear by name** in
|
||||
`incus list --project backup`
|
||||
- [ ] stale `nucbackup` pool removed from nuc (§9c), `zpool status`
|
||||
shows only `data`
|
||||
- [ ] replicas are **stopped** with `boot.autostart=false` — they hold
|
||||
the live containers' LAN addresses
|
||||
- [ ] ks4 pull leg seeded + one instance test-restored → tick the gate
|
||||
in [ks2/plan.md](../ks2/plan.md); then retire nuc's `wg-ks4`
|
||||
- [ ] power baseline recorded above, with the meter reading
|
||||
- [ ] this file updated with what was built (RAM fitted, NIC name, WD Red
|
||||
serial)
|
||||
@@ -1,6 +1,13 @@
|
||||
# transmission-bt
|
||||
|
||||
BitTorrent client in an unprivileged Incus container on `nuc`, with an
|
||||
|
||||
> Moved from nuc to `nas` on 2026-08-30, together with the media
|
||||
> dataset ([nas-install.md](nas-install.md) §7). Its WireGuard tunnel is
|
||||
> entirely in-container, so ks4 needed no change — the peer is still
|
||||
> `10.8.0.21`. The watch folder moved with it:
|
||||
> `/export/media/.watchdir` on nas, not `/srv/media/.watchdir` on nuc.
|
||||
|
||||
BitTorrent client in an unprivileged Incus container on `nas`, with an
|
||||
**always-on VPN**: all peer traffic exits via ks4's public IP through a
|
||||
WireGuard tunnel to the `wireguard` container on ks4. Kill switch by
|
||||
construction — `eth0` has **no default route**, so with the tunnel down
|
||||
@@ -12,10 +19,10 @@ the container simply has no path to the internet.
|
||||
whitelist (`192.168.0.*` only)
|
||||
- Egress: WG peer `10.8.0.21` → `193.70.35.17:51845`, `AllowedIPs 0.0.0.0/0`
|
||||
(verified: `curl ifconfig.me` from the container returns ks4's IP)
|
||||
- Downloads: `/media/downloads` (= `usb4t/media`, same dataset Jellyfin
|
||||
- Downloads: `/media/downloads` (= `tank/media`, same dataset Jellyfin
|
||||
reads); in-progress files in `/media/.incomplete` so Jellyfin never
|
||||
scans partials
|
||||
- Watch folder: `scp` a `.torrent` into `/srv/media/.watchdir` on the host
|
||||
- Watch folder: `scp` a `.torrent` into `/export/media/.watchdir` on the host
|
||||
and it auto-downloads (see "Watch folder" below)
|
||||
- `transmission-daemon` is `BindsTo=wg-quick@wg0.service` and binds
|
||||
peer traffic to `10.8.0.21` — three independent layers against leaks
|
||||
@@ -90,7 +97,7 @@ EOF'
|
||||
incus exec "$CNAME" -- systemctl enable --now wg-quick@wg0
|
||||
|
||||
# media share (same dataset as jellyfin-server)
|
||||
incus config device add "$CNAME" media disk source=/srv/media path=/media shift=true
|
||||
incus config device add "$CNAME" media disk source=/export/media path=/media shift=true
|
||||
incus exec "$CNAME" -- mkdir -p /media/downloads /media/.incomplete
|
||||
incus exec "$CNAME" -- chown debian-transmission:debian-transmission \
|
||||
/media/downloads /media/.incomplete
|
||||
@@ -129,9 +136,9 @@ incus exec wireguard -- wg-quick save wg0
|
||||
|
||||
## Watch folder (auto-add torrents)
|
||||
|
||||
Drop a `.torrent` into `/srv/media/.watchdir` on the host and Transmission
|
||||
Drop a `.torrent` into `/export/media/.watchdir` on the host and Transmission
|
||||
auto-adds it and starts downloading — no web UI needed. The folder lives on
|
||||
the shared `usb4t/media` dataset (`/media/.watchdir` inside the container).
|
||||
the shared `tank/media` dataset (`/media/.watchdir` inside the container).
|
||||
|
||||
Edit the **active** config only while the daemon is stopped (it rewrites
|
||||
`settings.json` on exit). The active file is
|
||||
@@ -161,7 +168,7 @@ incus exec transmission-bt -- systemctl start transmission-daemon
|
||||
Usage — the `.torrent` is consumed within a few seconds:
|
||||
|
||||
```sh
|
||||
scp some.torrent root@192.168.0.3:/srv/media/.watchdir/
|
||||
scp some.torrent root@192.168.0.4:/export/media/.watchdir/
|
||||
```
|
||||
|
||||
- `watch-dir-force-generic: true` makes Transmission **poll** the folder
|
||||
@@ -179,7 +186,7 @@ incus exec transmission-bt -- wg show wg0 latest-handshakes # non-zero timesta
|
||||
incus exec transmission-bt -- curl -s https://ifconfig.me # must print 193.70.35.17
|
||||
incus exec transmission-bt -- bash -c "ping -c1 -W2 8.8.8.8 || echo kill-switch OK" # with wg0 down
|
||||
# web UI must be tested from a LAN machine — the macvlan quirk means the
|
||||
# nuc host itself cannot reach 192.168.0.7
|
||||
# nas host itself cannot reach 192.168.0.7 (macvlan, by design)
|
||||
```
|
||||
|
||||
## Notes
|
||||
+32
-18
@@ -1,31 +1,45 @@
|
||||
# Homelab: nuc
|
||||
|
||||
Incus host on the LAN.
|
||||
Incus host on the LAN — **on-demand**: since 2026-08-30 it only needs to
|
||||
run when watching Jellyfin or using the Spotify Connect kiosk. Everything
|
||||
always-on (blocky/DNS, privoxy, transmission-bt) moved to
|
||||
[`nas`](../nas/README.md), which is why nuc can now be powered off.
|
||||
|
||||
⚠️ Powering nuc off has backup consequences — its 03:30 replication to
|
||||
nas only runs while it is up. See [nas-install.md](../nas/nas-install.md) §9.
|
||||
|
||||
- Debian 13, Intel Alder Lake-N (iGPU `i915`, shared by both Jellyfin containers) —
|
||||
bare-metal reinstall: [nuc-install.md](nuc-install.md)
|
||||
- Instances are bridged onto the LAN (192.168.0.0/24)
|
||||
- USB 4 TB WD Red: ZFS pool `usb4t` — `usb4t/backup` → `/backup`
|
||||
(incus exports, `nuc/` + `ks4/` subdatasets, 1 TB quota) and
|
||||
`usb4t/media` → `/srv/media` (media library, shared into containers
|
||||
via `shift=true` disk devices; works because ZFS ≥ 2.2 supports
|
||||
idmapped mounts)
|
||||
- ⚠️ The USB enclosure drops off the bus periodically and suspends the
|
||||
pool, which silently breaks the nightly replication — symptoms,
|
||||
recovery and fixes: [usb4t-dropouts.md](usb4t-dropouts.md)
|
||||
- Backups: all local instances replicated to the USB pool
|
||||
(`/root/scripts/incus-copy.sh -p backup -s nucbackup`; replicas
|
||||
stopped, autostart off) — see
|
||||
[nuc-install.md](nuc-install.md); ks4 replicas pulled into
|
||||
pool `ks4backup` — see [ks4/incus-copy.md](../ks4/incus-copy.md)
|
||||
- Storage: ZFS pool `data` on the SSD (all instance disks). The USB
|
||||
4 TB and its pool **left nuc on 2026-08-30** — the enclosure's JMicron
|
||||
bridge was suspending the pool ([usb4t-dropouts.md](usb4t-dropouts.md));
|
||||
the disk now sits on direct SATA in [`nas`](../nas/nas-install.md) as
|
||||
pool `tank`.
|
||||
- Media library: `/srv/media` is now an **NFSv4 mount from nas**
|
||||
(`192.168.0.4:/media`, read-only). `jellyfin-server` stays here for
|
||||
the iGPU and reads it with `shift=false`
|
||||
([jellyfin-server.md](jellyfin-server.md)).
|
||||
- Backups: nuc and nas **cross-replicate** — nuc pushes all its
|
||||
instances to `nas:nucbackup` at 03:30, nas pushes its own to
|
||||
`nuc:nasbackup` at 04:00, so neither host's instances depend on that
|
||||
host surviving ([nas/nas-install.md](../nas/nas-install.md) §9).
|
||||
The ks4 pull leg moved to nas as well
|
||||
([ks2/nas-seed.md](../ks2/nas-seed.md)), so nuc no longer runs a
|
||||
WireGuard tunnel.
|
||||
|
||||
- ⚠️ **Never reboot nuc with the display powered on** — the i915 probe
|
||||
dies and takes every container with it (including LAN DNS). Symptoms
|
||||
and fix: [jellyfin-client.md](jellyfin-client.md#-never-boot-nuc-with-the-display-active-2026-08-30)
|
||||
|
||||
## Instances
|
||||
|
||||
| Name | IP | Doc | Features |
|
||||
|---|---|---|---|
|
||||
| [jellyfin-server](jellyfin-server.md) | 192.168.0.5 | ✅ | unprivileged, autostart; iGPU render node (`gpu` device, render gid) for QSV/VAAPI transcoding; `/srv/media` disk device (`shift=true`); proxy device → host :8096 |
|
||||
| [jellyfin-server](jellyfin-server.md) | 192.168.0.5 | ✅ | unprivileged, autostart; iGPU render node (`gpu` device, render gid) for QSV/VAAPI transcoding; `/srv/media` disk device (NFS from nas, `shift=false`, read-only); proxy device → host :8096 |
|
||||
| [jellyfin-client](jellyfin-client.md) | 192.168.0.6 | ✅ | **privileged**, autostart; full iGPU (`gpu` device, gid 44) → HDMI kiosk (cage + Jellyfin Media Player); custom `raw.lxc` (bind `/dev/snd`, `/dev/input`, host `/run/udev`); Pioneer USB audio as ALSA default; FR keymap; go-librespot Spotify Connect ("Pioneer A-70") |
|
||||
| [transmission-bt](transmission-bt.md) | 192.168.0.7 | ✅ | unprivileged, autostart; always-on WireGuard full tunnel → ks4 (egress = 193.70.35.17, kill switch: no default route); `/srv/media` disk device (`shift=true`), downloads to `/media/downloads`; web UI :9091 (LAN only) |
|
||||
| blocky | 192.168.0.254 | — | unprivileged, autostart; DNS ad-blocker |
|
||||
| privoxy | 192.168.0.11 | — | unprivileged, autostart; filtering HTTP proxy |
|
||||
| homeassistant | (stopped) | — | **virtual machine**, 50 GiB root disk on pool `data` |
|
||||
|
||||
Moved to [`nas`](../nas/README.md) on 2026-08-30: `transmission-bt`
|
||||
(next to the media dataset it writes to), plus `blocky` and `privoxy`
|
||||
so LAN DNS and the proxy survive nuc being shut down.
|
||||
|
||||
+145
-1
@@ -116,6 +116,15 @@ EOF'
|
||||
# --- Kiosk user + seat management ----------------------------------------------
|
||||
incus exec "$CNAME" -- bash -c 'id kiosk >/dev/null 2>&1 || useradd -m -G video,render,input,audio kiosk'
|
||||
|
||||
# ⚠️ Group names are NOT enough. The host (Debian) and the container
|
||||
# (Ubuntu) allocate dynamic system gids independently, so the container's
|
||||
# `input` group does not necessarily have the same gid as the group that
|
||||
# owns /dev/input/* on the host. Bind the kiosk user to the *numeric*
|
||||
# host gid, whatever it is called inside:
|
||||
HOST_INPUT_GID="$(stat -c %g /dev/input/event0)"
|
||||
incus exec "$CNAME" -- usermod -aG "$HOST_INPUT_GID" kiosk
|
||||
incus exec "$CNAME" -- id kiosk # must list $HOST_INPUT_GID
|
||||
|
||||
# In a container seatd must NOT bind the seat to a VT (there is no usable VT;
|
||||
# it would try to open the host's active tty and hang the compositor forever).
|
||||
incus exec "$CNAME" -- mkdir -p /etc/systemd/system/seatd.service.d
|
||||
@@ -245,10 +254,17 @@ it works regardless of which USB port the dongle lands on.
|
||||
`/etc/udev/rules.d/99-jellyfin-kiosk-recover.rules`:
|
||||
|
||||
```
|
||||
# Logitech Unifying receiver (re)plugged -> recover the jellyfin-client kiosk
|
||||
# A Logitech receiver was (re)plugged -> recover the jellyfin-client kiosk.
|
||||
# c52b = Unifying receiver (K400); c539 = Lightspeed receiver (G603).
|
||||
ACTION=="add", SUBSYSTEM=="usb", ENV{DEVTYPE}=="usb_device", ATTR{idVendor}=="046d", ATTR{idProduct}=="c52b", RUN+="/usr/bin/systemctl --no-block start jellyfin-kiosk-recover.service"
|
||||
ACTION=="add", SUBSYSTEM=="usb", ENV{DEVTYPE}=="usb_device", ATTR{idVendor}=="046d", ATTR{idProduct}=="c539", RUN+="/usr/bin/systemctl --no-block start jellyfin-kiosk-recover.service"
|
||||
```
|
||||
|
||||
⚠️ The match is per product ID, so **a receiver not listed here will not
|
||||
auto-recover** — the kiosk must be restarted by hand after plugging it in
|
||||
(`incus exec jellyfin-client -- systemctl restart jellyfin-kiosk`). Add
|
||||
the new id here when introducing different input hardware.
|
||||
|
||||
`/etc/systemd/system/jellyfin-kiosk-recover.service` — oneshot, so a burst of
|
||||
udev events during one plug merges into a single restart (natural debounce):
|
||||
|
||||
@@ -285,6 +301,134 @@ udevadm control --reload-rules && systemctl daemon-reload
|
||||
journalctl -t jellyfin-kiosk-recover -f
|
||||
```
|
||||
|
||||
## ⚠️ Never boot nuc with the display active (2026-08-30)
|
||||
|
||||
**Symptom:** after a reboot, *no* Incus container starts. `incus list`
|
||||
answers but `incus start <anything>` hangs forever, `systemctl status
|
||||
incus` sits in `activating (start-post)`, and LAN DNS is down because
|
||||
blocky never came up. Nothing in the incus logs explains it.
|
||||
|
||||
**Cause — nothing to do with incus.** If the TV/projector is connected
|
||||
**and powered on** when nuc boots, firmware hands i915 an already-lit
|
||||
pipe. The driver's state readback then trips a series of warnings and
|
||||
the probe never completes:
|
||||
|
||||
```
|
||||
drm_WARN_ON(!pll_active) intel_ddi.c:4019 intel_ddi_get_clock
|
||||
drm_WARN_ON(p0 == 0 || p1 == 0 || p2 == 0) intel_dpll_mgr.c:2878
|
||||
drm_WARN_ON(pixel_rate == 0) skl_watermark.c:1729
|
||||
```
|
||||
(all inside `intel_modeset_setup_hw_state` ← `intel_display_driver_probe_nogem`)
|
||||
|
||||
The cascade:
|
||||
|
||||
1. i915 probe dies → **`/dev/dri` never appears** (no GPU at all)
|
||||
2. `snd_hda_intel` waits forever for i915's audio component →
|
||||
permanent **deferred probe** holding the PCI device lock on `0000:00:1f.3`
|
||||
3. incusd reads that device's `sriov_numvfs` while enumerating
|
||||
resources → blocks in **D state** → the daemon never signals ready,
|
||||
so nothing autostarts and every `incus start` hangs
|
||||
|
||||
**Reproduced identically on 6.12.107 and 6.12.105** — it is the display
|
||||
path, not a kernel regression. Do not waste time pinning kernels.
|
||||
|
||||
**Diagnosis, in order:**
|
||||
|
||||
```sh
|
||||
ls /dev/dri/ # empty = i915 probe failed
|
||||
cat /sys/kernel/debug/devices_deferred # snd_hda_intel entry = the deadlock
|
||||
ps -eLo pid,tid,stat,wchan:26,comm | awk '$3 ~ /D/' # incusd in sriov_numvfs_show
|
||||
dmesg -T | grep -E 'drm_WARN_ON|deferred probe pending'
|
||||
```
|
||||
|
||||
**Fix:** disconnect HDMI (or power the display fully off — not standby),
|
||||
reboot, then **hotplug the cable back in**. Connecting after boot goes
|
||||
through normal connector detection instead of firmware state readback
|
||||
and works fine.
|
||||
|
||||
In-place recovery is *not* possible: `modprobe -r i915` fails (module in
|
||||
use by the wedged probe) and `modprobe i915` times out. A reboot is the
|
||||
only way out.
|
||||
|
||||
**After hotplugging, restart the kiosk** — `cage` started with zero
|
||||
outputs and will not pick the display up on its own:
|
||||
|
||||
```sh
|
||||
incus exec jellyfin-client -- systemctl restart jellyfin-kiosk
|
||||
cat /sys/class/drm/card0-HDMI-A-2/status # expect: connected
|
||||
```
|
||||
|
||||
Note the HDA controller this wedges is only used for **HDMI audio**,
|
||||
which this setup does not use — audio goes to the Pioneer USB DAC via
|
||||
`/etc/asound.conf`. It is pure collateral damage, but it takes the whole
|
||||
host down with it.
|
||||
|
||||
## Input gid mismatch — latent, fix it anyway (2026-08-30)
|
||||
|
||||
> ⚠️ **This was not the cause of the 2026-08-30 outage.** That turned out
|
||||
> to be a flat/switched-off K400 — a G603 on the same port and the same
|
||||
> `event0` worked immediately. The mismatch below is real and worth
|
||||
> correcting, but with `LIBSEAT_BACKEND=seatd` it is **seatd (running as
|
||||
> root) that opens input devices** and passes the fd to cage, so the
|
||||
> kiosk user's group membership is not on the critical path for input.
|
||||
> Fix it for the direct-open fallback path, not as a debugging lead.
|
||||
>
|
||||
> **Before suspecting software, prove the hardware emits anything:**
|
||||
> ```sh
|
||||
> timeout 60 cat /dev/input/eventN | wc -c # press keys; 0 bytes = nothing reached the kernel
|
||||
> ```
|
||||
> That one check would have saved an hour.
|
||||
>
|
||||
> **Dead K400 batteries are invisible from the host.** This K400 exposes
|
||||
> no `hidpp_battery_*` node under `/sys/class/power_supply/`, so charge
|
||||
> cannot be read. Worse, the receiver still lists the keyboard as a paired
|
||||
> peer (`0003:046D:4024.*` under the `C52B` receiver) whether or not it is
|
||||
> awake, and `/dev/input/event0` plus a `Logitech K400` entry in
|
||||
> `/proc/bus/input/devices` are present either way — so every software
|
||||
> check looks perfectly healthy. **Zero bytes from the raw capture is the
|
||||
> only signal.** Swapping in a different receiver on the same port is the
|
||||
> quickest A/B confirmation.
|
||||
|
||||
**Symptom (if it ever does bite):** kiosk renders but input does nothing,
|
||||
with no errors — `WLR_LIBINPUT_NO_DEVICES=1` keeps cage alive rather than
|
||||
failing loudly.
|
||||
|
||||
**Cause:** `/dev/input/*` is `crw-rw---- root:<host input gid>`. The
|
||||
install script adds `kiosk` to the group *named* `input` inside the
|
||||
container, but Debian (host) and Ubuntu (container) allocate dynamic
|
||||
system gids independently:
|
||||
|
||||
```
|
||||
host /dev/input/event0 gid 996 -> "input"
|
||||
container "input" group gid 995 <- kiosk was here
|
||||
container gid 996 -> "systemd-timesync"
|
||||
```
|
||||
|
||||
So the kiosk user was in the wrong group and could not open any input
|
||||
device. The udev side was fine — check it first to rule it out:
|
||||
`cat /run/udev/data/c13:64` should show `E:ID_INPUT=1` etc.
|
||||
|
||||
**Diagnose:**
|
||||
|
||||
```sh
|
||||
stat -c '%n %a %u:%g' /dev/input/event0 # host gid
|
||||
incus exec jellyfin-client -- id kiosk # does it include that gid?
|
||||
# open() test — do NOT use `head`/`cat`, reading an event device blocks
|
||||
# with no pending events and looks like a permission failure:
|
||||
incus exec jellyfin-client -- su -s /bin/bash kiosk -c 'exec 3< /dev/input/event0 && echo OPEN_OK'
|
||||
```
|
||||
|
||||
**Fix** (persists in the container's `/etc/group`):
|
||||
|
||||
```sh
|
||||
HOST_INPUT_GID="$(stat -c %g /dev/input/event0)"
|
||||
incus exec jellyfin-client -- usermod -aG "$HOST_INPUT_GID" kiosk
|
||||
incus exec jellyfin-client -- systemctl restart jellyfin-kiosk
|
||||
```
|
||||
|
||||
Re-check this after any host reinstall — the host's `input` gid is
|
||||
dynamically allocated and can come back different.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
```sh
|
||||
|
||||
+41
-2
@@ -5,8 +5,10 @@ Jellyfin media **server** in an unprivileged Incus container on `nuc`.
|
||||
- Image: `images:ubuntu/24.04`, Jellyfin from the official repo (repo.jellyfin.org)
|
||||
- IP: `192.168.0.5` (LAN bridge) — web UI/API on `http://192.168.0.5:8096`
|
||||
- iGPU render node (`/dev/dri/renderD128`) passed for QSV/VAAPI hardware transcoding
|
||||
- Media library: host `/srv/media` (ZFS dataset `usb4t/media`, USB 4 TB)
|
||||
mounted at `/media` with `shift=true` (needs ZFS ≥ 2.2 for idmapped mounts)
|
||||
- Media library: host `/srv/media` — since 2026-08-30 an **NFSv4 mount
|
||||
from `nas`** (`192.168.0.4:/media`, dataset `tank/media`), mounted at
|
||||
`/media` in the container with **`shift=false`** and `readonly=true`.
|
||||
See [nas/nas-install.md](../nas/nas-install.md) §8.
|
||||
- Port 8096 additionally proxied to the host address (`web` proxy device)
|
||||
|
||||
## Install script
|
||||
@@ -77,6 +79,43 @@ incus restart "$CNAME"
|
||||
echo "Done. Open http://<host-ip>:8096 to run the setup wizard."
|
||||
```
|
||||
|
||||
## Media over NFS (2026-08-30)
|
||||
|
||||
The library moved to `nas` when the 4 TB left nuc's USB enclosure. The
|
||||
container keeps the same path, so everything below still applies — only
|
||||
the mount underneath `/srv/media` changed.
|
||||
|
||||
```sh
|
||||
# nuc host: /etc/fstab
|
||||
192.168.0.4:/media /srv/media nfs4 ro,_netdev,soft,timeo=100,retrans=3 0 0
|
||||
```
|
||||
|
||||
`shift=true` **cannot** be used: idmapped mounts are not supported on
|
||||
NFS (nor CIFS). Per the troubleshooting note below, dropping the shift is
|
||||
enough for a read-only library — the export uses `all_squash` so files
|
||||
carry synthetic world-readable ownership:
|
||||
|
||||
```sh
|
||||
incus stop jellyfin-server # shift cannot be hot-applied
|
||||
incus config device set jellyfin-server media shift=false
|
||||
incus config device set jellyfin-server media readonly=true
|
||||
incus start jellyfin-server
|
||||
```
|
||||
|
||||
⚠️ Two traps:
|
||||
|
||||
- **Boot ordering.** Add `remote-fs.target` to nuc's
|
||||
`/etc/systemd/system/incus.service.d/after-zfs.conf`, or the container
|
||||
starts against an empty mountpoint and Jellyfin shows an empty library
|
||||
(and may prune the library metadata).
|
||||
- **`soft` is deliberate.** A hung nas should fail Jellyfin's reads, not
|
||||
wedge nuc's processes in uninterruptible sleep the way the suspended
|
||||
`usb4t` pool did ([usb4t-dropouts.md](usb4t-dropouts.md)).
|
||||
|
||||
`transmission-bt` is no longer on nuc — it moved to nas and writes to
|
||||
the dataset locally ([nas/transmission-bt.md](../nas/transmission-bt.md)),
|
||||
so nuc's mount is read-only and there is exactly one writer.
|
||||
|
||||
## First-run configuration
|
||||
|
||||
1. Run the setup wizard; add libraries pointing at `/media/...`.
|
||||
|
||||
@@ -265,6 +265,13 @@ Both logs rotate monthly (`/etc/logrotate.d/incus-*`).
|
||||
the install scripts in the per-container docs include it)
|
||||
- [ ] Host boots to `multi-user.target`, nothing grabs the GPU
|
||||
(required by the jellyfin-client kiosk)
|
||||
- [ ] ⚠️ **Reboot with the TV/projector disconnected or powered off.**
|
||||
Booting with the display active kills the i915 probe, which wedges
|
||||
`snd_hda_intel` in a deferred probe, which blocks incusd in
|
||||
`sriov_numvfs_show` — **no container starts at all, including
|
||||
blocky/DNS**. Hotplug the cable back after boot, then
|
||||
`incus exec jellyfin-client -- systemctl restart jellyfin-kiosk`.
|
||||
Full diagnosis: [jellyfin-client.md](jellyfin-client.md)
|
||||
- [ ] Jellyfin web at `http://192.168.0.5:8096`, kiosk UI on HDMI,
|
||||
sound on the Pioneer, "Pioneer A-70" visible in Spotify Connect
|
||||
- [ ] LAN DNS: clients use blocky at `192.168.0.254` (host itself uses
|
||||
|
||||
Reference in New Issue
Block a user