diff --git a/README.md b/README.md index 30668c2..57942a5 100644 --- a/README.md +++ b/README.md @@ -12,7 +12,9 @@ backed up, by which tool, on what schedule, and how to restore. service to ks4 (and getting it backed up automatically). **Tech notes** -- [`nuc/`](nuc/README.md) — home lab on `nuc`: host, storage, instances +- [`nuc/`](nuc/README.md) — home lab on `nuc`: host, iGPU instances +- [`nas/`](nas/README.md) — storage + backup host `nas` (Supermicro + A1SAi-2750F): the 4 TB on direct SATA, media over NFS, backup pools - [`ks4/`](ks4/README.md) — prod server `ks4` at OVH: host, services, network flows - [`ks2/`](ks2/) — legacy backup server being [decommissioned](ks2/plan.md) - [`archer-c7/`](archer-c7/) — home router (TP-Link Archer C7 v5 running on OpenWrt) diff --git a/ks2/nuc-seed.md b/ks2/nas-seed.md similarity index 100% rename from ks2/nuc-seed.md rename to ks2/nas-seed.md diff --git a/nas/README.md b/nas/README.md new file mode 100644 index 0000000..8033d0d --- /dev/null +++ b/nas/README.md @@ -0,0 +1,50 @@ +# Storage: nas + +Storage + backup host on the LAN, added 2026-08. + +- **Always-on host.** nuc is now an on-demand media box (see + [nuc/README.md](../nuc/README.md)), so everything that must stay up — + LAN DNS, the HTTP proxy, torrents — lives here. +- Debian 13, Supermicro **A1SAi-2750F** / Intel Atom C2750 (8 c, 20 W, + ECC DDR3) — build procedure: [nas-install.md](nas-install.md) +- SSH: `ssh -i id_rsa_claude root@192.168.0.4` +- **No iGPU** (Avoton is headless; video is the AST2400 BMC). Anything + needing hardware transcoding stays on nuc. +- Pools: + - `incus` — ZFS mirror across the last partition of both 120 GB SSDs; + holds this host's container roots. OS itself is on mdraid RAID1 + + ext4 across the same disks (rationale in + [nas-install.md](nas-install.md) §2). + - `tank` — the 4 TB WD Red, **on direct SATA**. This is the whole + point of the box: the disk used to hang off a JMicron USB bridge on + nuc that suspended the pool 61 times in 30 days + ([nuc/usb4t-dropouts.md](../nuc/usb4t-dropouts.md)). Single vdev, + accepted — nothing on it is irreplaceable. +- `tank/media` → `/export/media`: the media library. Exported + **read-only over NFSv4 to nuc**, where `jellyfin-server` reads it; + written locally only by `transmission-bt`. +- Backups: nuc ↔ nas **cross-replication** (each host's instances live + on the other), plus the ks4 pull leg — + [nas-install.md](nas-install.md) §9, + [ks2/nas-seed.md](../ks2/nas-seed.md). + +## Instances + +| Name | IP | Doc | Features | +|---|---|---|---| +| blocky | 192.168.0.254 | — | unprivileged, autostart; DNS ad-blocker for the LAN. Moved from nuc 2026-08-30 so it survives nuc being powered off | +| privoxy | 192.168.0.11 | — | unprivileged, autostart; filtering HTTP proxy, listens on **:3128** (not privoxy's default 8118); static config in `/etc/systemd/network/eth0.network`, `DNS=192.168.0.254`. Moved from nuc 2026-08-30 | +| [transmission-bt](transmission-bt.md) | 192.168.0.7 | ✅ | unprivileged, autostart; always-on WireGuard full tunnel → ks4 (egress = 193.70.35.17, kill switch: no default route); `/export/media` disk device (`shift=true`), downloads to `/media/downloads`; web UI :9091 (LAN only). Moved from nuc 2026-08-30 | + +## Backup pools hosted here + +| incus pool | dataset | receives | +|---|---|---| +| `nucbackup` | `tank/backup/nuc` | nuc's instances (pushed nightly, 03:30) | +| `ks4backup` | `tank/backup/ks4` | ks4's instances (pulled over WG, 05:00 — after FTTH) | +| `nasbackup` | `tank/backup/nas` | **nas's own** instances (local copy, 04:00) | + +nas's own instances are replicated **locally** rather than to nuc: nuc is +an on-demand box and usually powered off, so it is not a usable backup +target. `nasbackup` lives on `tank`, a different pool from the `incus` +SSD mirror the instances run on. diff --git a/nas/nas-install.md b/nas/nas-install.md new file mode 100644 index 0000000..77d42ab --- /dev/null +++ b/nas/nas-install.md @@ -0,0 +1,838 @@ +# nas — build procedure + +New storage host on the LAN (`192.168.0.4`), built 2026-08 from a +Supermicro A1SAi-2750F. It exists to solve one specific problem: +[usb4t-dropouts.md](../nuc/usb4t-dropouts.md) ruled out the cable and +USB power management and left the **JMicron 152d:0578 bridge** as the +cause of 61 disconnects in 30 days. The durable fix named there is a +direct SATA connection — this box provides six of them. + +Why it matters beyond the media library: `usb4t` is the intended home +of the ks4 off-site replicas, and *"the pool holding the off-site copy +of ks4 must not be the least reliable device in the setup"*. That makes +this build a **release gate for the ks2 decommission** +([ks2/plan.md](../ks2/plan.md), deadline Sep 30, 2026). + +It also takes over `transmission-bt` from nuc, so nuc keeps only what +needs its iGPU. + +## Hardware + +- Supermicro **A1SAi-2750F** mini-ITX, Intel Atom **C2750** (8 cores, + 2.4 GHz Silvermont, 20 W SoC), AES-NI, no AVX +- RAM: **2× 4 GB DDR3-1600 ECC SO-DIMM fitted = 8 GB** (`Single-bit ECC` + confirmed), in DIMMA1/DIMMB1; **2 slots free**. Board takes 32 GB + officially, 64 GB with 16 GB modules. + ⚠️ 8 GB is modest for a 3.6 TB pool — ARC lands around 4 GB. Fine for + streaming and replication (neither benefits much from cache), but the + first thing to raise if metadata-heavy operations feel slow. +- SATA: **2× SATA3 + 4× SATA2** (6 total) +- BIOS **2.2** (2019-11-22) as shipped by the RMA +- NIC: 4× GbE (Intel i354) + dedicated IPMI LAN +- Video: **ASPEED AST2400 BMC only — there is no iGPU.** Avoton is a + headless server SoC; `/dev/dri` is empty. That is why `jellyfin-server` + stays on nuc (§8). + +### ⚠️ AVR54 — already handled + +The C2750 is on the list of Atom C2000 parts affected by Intel's +**AVR54** erratum: the SoC's `LPC_CLKOUT0/1` signals degrade and stop, +after which the board never boots again — typically after ~18 months of +power-on, i.e. exactly an always-on duty cycle. **This board was RMA'd +by Supermicro for that issue and replaced**, so it carries the fix +(C0 stepping or the LPC pull-up rework). Recorded here so a dead C2000 +board is not re-diagnosed from scratch later. + +### Disk plan + +| Port | Device | Role | +|---|---|---| +| SATA3-0 | Intel `SSDSC2CT120A3` (120 GB) | md mirror + `incus` pool | +| SATA3-1 | Toshiba `Q300.` (120 GB) | md mirror + `incus` pool | +| SATA2-0 | WD Red 4 TB (moved off the USB enclosure) | pool `tank` | +| SATA2-1/2/3 | free | a second 4 TB to mirror `tank`, later | + +The SSDs get the SATA3 ports because they are the only devices that can +use them: both negotiate 6 Gb/s and do ~450–500 MB/s, while the WD Red +tops out near 180 MB/s and cannot saturate SATA2's ~270 MB/s. A future +SLOG would also be fine on SATA2 — it is latency-bound on small sync +writes, not bandwidth-bound. + +⚠️ **Disconnect the 4 TB before partitioning the SSDs.** It carries the +media library and nuc's replicas, nothing in the OS install needs it, +and it keeps the two 120 GB disks unambiguous in the installer's list. +Reconnect it before §5b. + +Device letters shift with enumeration order and mean nothing — mdraid +assembles from superblock UUIDs, ZFS imports by GUID, fstab and GRUB use +UUIDs. **Identify disks by model and serial** +(`lsblk -o NAME,SIZE,MODEL,SERIAL`), never by letter: both SSDs are +120 GB, so size alone does not tell them apart. + +SSD health measured 2026-08-30 (both read over the JMicron bridge on +nuc, `smartctl -d sat`), before deployment: + +| | Intel 330 (25 nm MLC) | Toshiba Q300 (15 nm TLC) | +|---|---|---| +| host writes | 16.0 TiB | 7.5 TiB | +| endurance consumed | **0 %** (`Media_Wearout_Indicator` 100) | **7 %** (`Percentage Used Endurance Indicator`) | +| power-on hours | **unreadable** — attr 9 decodes to 914,563 h on this family | 3,877 | +| power cycles | 98 | 432 | +| defects | 0 reallocated / program-fail / erase-fail | 0 reported uncorrectable | +| SMART error log | **not supported** | supported, clean | +| device statistics log | absent | full ACS-2 set | +| short self-test | passed | passed | +| interface CRC errors | n/a | 16 (baseline — watch for growth) | + +Both are healthy and far from wear-out; at OS-disk write rates endurance +is not the binding constraint for either. They were previously a +**matched pair** — identical layouts, `bpool` / `rpool` / `ubuntu:0` +labels from an Ubuntu ZFS-on-root mirror — which is why they go back +into a mirror here. + +⚠️ **The Intel is effectively unmonitorable**: no error log, no device +statistics, no temperature, no usable hours counter. `smartd` can watch +the Toshiba properly and can only ask the Intel "are you still there". +Expect the Intel to be found dead rather than found degrading. That +asymmetry is the reason for the mirror. + +### `tank` is a single vdev — accepted + +The 4 TB holds the media library, transmission's downloads and the ks4 +replicas, with no redundancy. A single vdev can *detect* corruption but +only self-heal metadata, not data — as seen in the usb4t incident. + +**Decision 2026-08-30: accepted, nothing on `tank` is irreplaceable.** +Media and instances are re-fetchable from their sources; the ks4 +replicas are leg 3 of ks4's 3-2-1 (local `sdb5` + restic/S3 remain — +[backup-strategy.md](../backup-strategy.md)). Losing `tank` costs time, +not data. + +Three SATA ports stay free, so `zpool attach tank ` +turns it into a mirror whenever a spare 4 TB turns up. Not a +prerequisite for anything. Scrub weekly regardless — on a single vdev +the scrub is the only thing that *tells* you a file has rotted. + +## 1. IPMI and BIOS first + +The AST2400 stack is old and has known vulnerabilities. + +- **Decision 2026-08-30: the BMC is never cabled.** The dedicated IPMI + port stays unplugged, so the AST2400's default credentials and its + known vulnerabilities are not reachable from anything. This is the + simplest correct answer for a box that sits on a flat home LAN — no + management VLAN needed, nothing to harden, nothing to patch. +- ⚠️ Consequence: **there is no out-of-band console.** A boot that fails + before sshd needs a physical monitor and keyboard. Worth knowing before + changing anything that affects booting (GRUB, the md arrays, fstab). +- BIOS: enable **restore-on-AC-loss**, enable C-states, and disable the + three unused i354 NICs (each costs about a watt). + +### Switching this board to UEFI + +There is **no "Boot Mode Select" entry** in this BIOS — that option +exists on later Supermicro generations, not here. What works +(verified 2026-08-30): + +- **CSM → Disabled** +- **All OpROM policies → UEFI** (storage, video *and* network) + +Setting the **video** OpROM to UEFI is safe despite the console being +the AST2400 BMC framebuffer — output survives, both on the BMC console +and over IPMI KVM. + +⚠️ **Confirm the mode before you partition anything**, from the +installer (`Ctrl+Alt+F2`): + +```sh +ls /sys/firmware/efi # directory exists = UEFI. Missing = legacy +``` + +This single check is what makes the difference between a working +install and an afternoon lost — see the trap in §3. + +## 2. Prepare the SSDs + +Both report `ATA Security is: Disabled, NOT FROZEN`, so a real secure +erase is available — do that rather than just repartitioning. It +restores the full spare-block pool on ten-year-old NAND. + +```sh +# per disk, from a live system where the disk is NOT the running OS +hdparm --user-master u --security-set-pass Eins /dev/sdX +hdparm --user-master u --security-erase Eins /dev/sdX +hdparm -I /dev/sdX | grep -A2 Security # expect "not enabled" +``` + +Identical GPT layout on both, sized so ~27 GB (22 %) stays +**unallocated** as over-provisioning: + +| Part | Size | Type | Device | Use | +|------|------|------|--------|-----| +| `sdX1` | 1 GB | EFI System Partition | — | `/boot/efi` (two independent ESPs) | +| `sdX2` | 2 GB | Linux RAID | `md0` | `/boot` ext4 | +| `sdX3` | 24 GB | Linux RAID | `md1` | `/` ext4 | +| `sdX4` | 2 GB | Linux RAID | `md2` | swap | +| `sdX5` | 64 GB | Solaris root (bf00) | — | **zpool `incus`** (mirror) | +| — | ~27 GB | **unallocated** | — | over-provisioning | + +- ⚠️ **The ESP is deliberately not a RAID1 array.** ks4 has `md1 → + /boot/efi` because OVH's installer builds it with mdadm metadata + **1.0** (superblock at the *end*, so firmware still sees plain FAT). + `debian-installer` only creates metadata **1.2** arrays, whose + superblock sits at the start and makes the ESP unreadable to + firmware. So: one plain ESP per disk, only one mounted, the second + filled by hand (§3). Deviation from ks4 is intentional. +- A legacy-BIOS variant of this layout was tried first — 1 MB + `bios_grub` instead of the ESP, which makes the mirror simpler + (`grub-install` to both disks, nothing to keep in sync). It was + abandoned because this board's firmware has no way to prefer legacy + targets once CSM is off, and it kept falling through to the UEFI + shell. Recorded so it is not retried: **UEFI is the working path + here.** +- `/` at 24 GB matches the other hosts (nuc 46 GB, ks4 40 GB) — this box + has no desktop and no container roots on `/`. +- Mixing md partitions and a ZFS partition on the same disks is exactly + what ks4 does (`md1/2/3` + ZFS on `sda5`/`sdb5`). + +### Why not full root-on-ZFS + +Decision 2026-08-30. These SSDs previously ran Ubuntu 20.04's +experimental ZFS-root installer (hence the leftover `bpool` / `rpool` +labels), so the option was on the table. Rejected because: + +- **Debian's installer cannot do it.** ZFS is CDDL, shipped only in + `contrib` as `zfs-dkms`; `debian-installer` can neither partition nor + boot from ZFS. Root-on-ZFS means the manual + [OpenZFS Debian Trixie HOWTO](https://openzfs.github.io/openzfs-docs/Getting%20Started/Debian/Debian%20Trixie%20Root%20on%20ZFS.html) + — ~60 steps from a live ISO. Per the repo convention the doc *is* the + rebuild procedure, and that is a bad thing to be executing during an + actual failure. +- **DKMS failure mode.** `zfs-dkms` rebuilds on every kernel upgrade. + If that build fails, root-on-ZFS means the box **does not boot**; + with an ext4 md root it boots normally and only the pools are + missing — recoverable over IPMI with a shell. +- GRUB's ZFS support lags OpenZFS, which is why every root-on-ZFS guide + needs a separate feature-limited `bpool`; `zpool upgrade bpool` is a + known way to make a machine unbootable. +- Ubuntu's version of this is a dead end anyway: the installer option + was nearly dropped in 22.04 and `zsys`, which made boot environments + useful, is abandoned. + +What root-on-ZFS would buy — snapshot and roll back a bad upgrade — is +already covered where the state actually lives: container roots get +incus snapshots plus nightly replication (§9). The host is 24 GB of +packages reproducible from this file. Accepted cost: no pre-upgrade +rollback of the host itself (`etckeeper` covers `/etc` if wanted). + +## 3. Install Debian 13 (trixie) + +Netinst ISO (burned 2026-08-30, sha256 `65273bee…664e7`, verified +against `cdimage.debian.org/debian-cd/13.6.0/amd64/iso-cd/SHA256SUMS`). + +- Manual partitioning per the table above: `sdX1` as ESP, `sdX2..4` as + RAID1 members (three arrays), `sdX5` left untouched. +- Tasks: **SSH server + standard utilities only**. No desktop — there is + no GPU and the console is a BMC framebuffer. +- Sources: `main contrib non-free-firmware` (`contrib` is required by + `zfs-dkms`; the installer does not offer it — add it after first boot). + +Confirm d-i picked the right bootloader once installed: +`dpkg -l | grep grub-efi` — `grub-efi-amd64`, not `grub-pc`. + +### ⚠️ Trap: installer boot mode decides the bootloader (hit 2026-08-30) + +`debian-installer` chooses `grub-pc` or `grub-efi-amd64` from **how the +installer itself booted**, not from what the disks look like. Booting the +USB stick in legacy mode while the firmware prefers UEFI produces: + +1. d-i installs `grub-pc`, targeting the MBR; +2. on a **GPT** disk that needs a 1 MB `bios_grub` partition — absent + here, so `grub-install` fails, easy to click past; +3. the firmware then tries the SSDs as UEFI targets, finds no `.efi` + binary, and drops to the **UEFI shell**. + +Nothing is corrupt; the halves simply disagree. Symptoms and checks: + +```sh +[ -d /sys/firmware/efi ] && echo UEFI || echo legacy # in the installer +lsblk -no PTTYPE,PARTTYPENAME /dev/sdX # gpt + "EFI System"? +dd if=/dev/sdX bs=440 count=1 2>/dev/null | od -c | head -3 # all \0 = no boot code +dpkg -l | grep -E '^ii.*grub-(pc|efi)' # which flavour got installed +``` + +The board's hybrid ISO offers both paths, so the F11 boot menu usually +lists the stick twice — picking the **`UEFI:`** entry avoids the whole +thing. Checking `/sys/firmware/efi` before partitioning is the one step +that prevents it. + +### Second ESP — do this before trusting the mirror + +The installer populates only the ESP it mounted. Until the second one is +written, losing that disk means the box does not boot, mirror or no +mirror. + +⚠️ **Use `/dev/disk/by-id/`, never `/dev/sdX`.** Reconnecting the 4 TB +after the install shifts every letter — observed 2026-08-30: the HDD on +SATA2-0 takes `sda` even with the SSDs on SATA3, because the SATA2 +controller enumerates first on this SoC. A bare `/dev/sdb1` written +during the install then points at a *different disk*, and here that +would mean reformatting the ESP the system actually boots from. + +```sh +ESP2=/dev/disk/by-id/ata-TOSHIBA_Q300._36OB318OK1KU-part1 # the one NOT at /boot/efi +mkfs.vfat -F32 "$ESP2" +mkdir -p /boot/efi2 && mount "$ESP2" /boot/efi2 +grub-install --target=x86_64-efi --efi-directory=/boot/efi2 \ + --bootloader-id=debian-b --recheck +efibootmgr -v # expect: debian, debian-b +echo "UUID=$(blkid -s UUID -o value $ESP2) /boot/efi2 vfat umask=0077 0 1" >> /etc/fstab +mount -a && findmnt /boot/efi2 +``` + +Re-run the `grub-install` after any GRUB or kernel change — the second +ESP is **not** kept in sync automatically. **Verify by pulling one disk +and booting**; the checklist item exists because an untested mirror is a +guess, and this is the component most likely to be silently wrong. + +Do not fix the letter ordering by moving cables: the only arrangement +that makes the SSDs `sda`/`sdb` puts them on SATA2 and the HDD on SATA3, +which caps the only devices that can use 6 Gb/s and gives the bandwidth +to a disk that tops out near 180 MB/s. + +## 4. Base system + +```sh +apt update && apt full-upgrade -y +apt install -y \ + linux-headers-amd64 zfs-dkms zfsutils-linux zfs-zed \ + mdadm smartmontools nfs-kernel-server \ + msmtp msmtp-mta bsd-mailx \ + curl vim htop ripgrep sysstat dmidecode pciutils usbutils \ + stress-ng fio +``` + +Static network — `/etc/network/interfaces` (ifupdown, matching nuc): + +``` +source /etc/network/interfaces.d/* + +auto lo +iface lo inet loopback + +allow-hotplug enp0s20f0 +iface enp0s20f0 inet static + address 192.168.0.4 + netmask 255.255.255.0 + gateway 192.168.0.2 + dns-nameservers 1.1.1.1 9.9.9.9 +``` + +(Gateway is **`192.168.0.2`** — `.1` is gone since 2026-08. The host +uses public resolvers, never blocky, to avoid a bootstrap loop. +Interface name is a guess until the board is up — check `ip -br link`.) + +Restore `/root/.ssh/authorized_keys` (incl. `id_rsa_claude.pub`) and +`timedatectl set-timezone Europe/Paris`. + +## 5. Pools + +### 5a. `incus` — SSD mirror + +```sh +zpool create -o ashift=12 \ + -O compression=zstd -O atime=off -O xattr=sa -O acltype=posixacl \ + incus mirror \ + /dev/disk/by-id/ata-INTEL_SSDSC2CT120A3_CVMP250400ES120BGN-part5 \ + /dev/disk/by-id/ata-TOSHIBA_Q300._36OB318OK1KU-part5 +zpool set autotrim=on incus +``` + +Weekly scrubs come from the packaged systemd timers rather than cron: + +```sh +systemctl enable --now zfs-scrub-weekly@tank.timer zfs-scrub-weekly@incus.timer +systemctl list-timers 'zfs-scrub*' +``` + +`autotrim` matters on ten-year-old NAND — it is what keeps the +unallocated 22 % actually available to the controller as spare. + +### 5b. `tank` — move the 4 TB off USB onto SATA + +The risky step. The pool holds `usb4t/backup/nuc` (nuc's replicas), +`usb4t/backup/ks4` (empty, awaiting the FTTH seed) and `usb4t/media`. + +⚠️ **Between export and import, nuc has no replica target and Jellyfin +has no media.** Plan a maintenance window and disable nuc's 03:30 cron +first, so it fails loudly rather than half-running. + +```sh +# --- on nuc, first --- +zpool scrub usb4t # start clean; wait for it +zpool status usb4t +zpool export usb4t +``` + +Move the disk to SATA2-0 (reconnect it now if you unplugged it for the +install), then: + +```sh +# --- on nas --- +zpool import # confirm it is seen +zpool import usb4t tank # rename: it is not USB any more +zpool set cachefile=/etc/zfs/zpool.cache tank +zpool status tank +``` + +Properties survive from creation (`ashift=12`, `compression=zstd`, +`atime=off`, `xattr=sa`, `acltype=posixacl`). Re-point the mountpoints +and add the two new backup datasets: + +```sh +zfs set mountpoint=/export/media tank/media # NFSv4 export root (§8) +zfs set mountpoint=none tank/backup +zfs list -o name,used,avail,mountpoint +``` + +Resulting layout: + +``` +incus mirror, 2× SSD — nas's own container roots +tank 4 TB, single vdev +├── tank/media → /export/media NFS ro → nuc; local device → transmission-bt +└── tank/backup + ├── tank/backup/nuc → incus pool `nucbackup` (nuc pushes here) + └── tank/backup/ks4 → incus pool `ks4backup` (nas pulls from ks4) + +nas's own replicas live on **nuc** (`data/backup/nas` → pool `nasbackup`), +not here — see §9. +``` + +Backup pools are named after the **source** host, matching +[ks4/incus-copy.md](../ks4/incus-copy.md). nas's *own* instances are not +backed up here — they cross-replicate to nuc (§9b), so neither host's +instances depend on that host surviving. + +Once `tank` has run a week on SATA with **zero** pool suspensions and no +CRC errors, the `usb4t-dropouts` gate is cleared — record that in +[ks2/plan.md](../ks2/plan.md). + +## 6. Incus + +Same Zabbly stable repo as nuc and ks4: + +```sh +mkdir -p /etc/apt/keyrings +curl -fsSL https://pkgs.zabbly.com/key.asc -o /etc/apt/keyrings/zabbly.asc +cat > /etc/apt/sources.list.d/zabbly-incus-stable.sources <> /etc/exports <<'EOF' +/export 192.168.0.3(ro,fsid=0,crossmnt,no_subtree_check) +/export/media 192.168.0.3(ro,no_subtree_check,all_squash,anonuid=65534,anongid=65534) +EOF +exportfs -ra && exportfs -v +``` + +```sh +# --- nuc: mount at the SAME path, so jellyfin-server.md still applies --- +mkdir -p /srv/media +echo '192.168.0.4:/media /srv/media nfs4 ro,_netdev,soft,timeo=100,retrans=3 0 0' >> /etc/fstab +mount /srv/media && ls /srv/media +``` + +The container device changes only in losing the shift — per +[nuc/jellyfin-server.md](../nuc/jellyfin-server.md)'s own troubleshooting +note, *"on CIFS files are world-readable synthetic ownership, enough for +a read-only library"*; the same holds for NFS with `all_squash`: + +```sh +incus stop jellyfin-server # shift cannot be hot-applied +incus config device set jellyfin-server media shift=false +incus config device set jellyfin-server media readonly=true +incus start jellyfin-server +incus exec jellyfin-server -- ls /media # must list the library +``` + +Requirements for that to work: media files must be **world-readable** +(`find /export/media -type f ! -perm -o=r`), and transmission must keep +creating them that way (it does — `umask`/`0775` per its doc). + +⚠️ Boot ordering on nuc: incus is already ordered after +`zfs-mount.service`; add `remote-fs.target` to that drop-in, or +`jellyfin-server` starts against an empty mountpoint and shows an empty +library. + +⚠️ `soft` mount is deliberate: a hung NAS should fail Jellyfin's reads, +not wedge nuc's processes in uninterruptible sleep the way the suspended +`usb4t` pool did. + +## 9. Backup legs + +The driver is unchanged — +[`incus-copy.sh`](https://git.lutran.fr/julien/scripts/src/branch/main/incus-copy.sh), +deployed to `/root/scripts` as everywhere else. + +### 9a. nuc's instances -> nas (systemd timer, not cron) + +⚠️ **nuc is an on-demand media box** (see [nuc/README.md](../nuc/README.md)): +since 2026-08-30 it only runs when watching Jellyfin or using the Spotify +kiosk, so it is usually powered off at 03:30. Cron cannot catch up a +missed window; a **systemd timer with `Persistent=true`** runs the job +shortly after the next boot. That is the whole reason this is a timer. + +`/etc/systemd/system/incus-copy.service` on nuc: + +```ini +[Unit] +Description=Replicate all nuc instances to nas (project backup, pool nucbackup) +After=network-online.target incus.service +Wants=network-online.target +Requires=incus.service + +[Service] +Type=oneshot +ExecStart=/bin/sh -c "/root/scripts/incus-copy.sh -d nas -m push -s nucbackup -p backup >> /var/log/incus-copy.log 2>&1" +``` + +`/etc/systemd/system/incus-copy.timer`: + +```ini +[Timer] +OnCalendar=03:30 +Persistent=true +RandomizedDelaySec=300 + +[Install] +WantedBy=timers.target +``` + +```sh +systemctl enable --now incus-copy.timer && systemctl list-timers incus-copy +``` + +⚠️ **`-p backup` is not optional.** Without it the replicas land in +`default` on nas and collide with nas's *live* instances — both hosts are +on the same macvlan LAN and the replicas carry the same static IPs. + +### 9b. nas's own instances -> local pool `nasbackup` + +**Decision 2026-08-30: local, not cross-replicated.** nuc is powered off +most of the time, so it is not a usable backup target — a nightly push to +it would fail noisily and, once mail works, alarm every morning. The +replicas instead go to `tank/backup/nas`, which is a **different pool** +from the instances themselves (`incus`, the SSD mirror), so it survives +losing that mirror. All three instances here are rebuildable from their +docs, so same-host is proportionate. + +```sh +zfs create tank/backup/nas +zfs set mountpoint=legacy tank/backup/nas # see the trap below +incus storage create nasbackup zfs source=tank/backup/nas +``` + +```cron +# nas, /etc/cron.d/incus-copy +0 4 * * * root /root/scripts/incus-copy.sh -p backup -s nasbackup >> /var/log/incus-copy.log 2>&1 +``` + +⚠️ **Trap: `incus storage create` hangs forever on a `mountpoint=none` +dataset.** `tank/backup` is set to `mountpoint=none`, so any child +created afterwards inherits it, and the pool create then blocks with no +error and no entry in `incus operation list` — it looks exactly like I/O +contention (a scrub was running, which sent me down that path for 20 +minutes). Set the child to `legacy` to match its siblings first, and it +completes instantly. + +### 9c. Cleanup on nuc after the pool move + + +Exporting `usb4t` leaves nuc with an incus storage pool whose backing +dataset is gone, plus replica records in the `backup` project pointing +at it. Remove them once §6's `incus admin recover` has re-registered the +same volumes on nas — **verify there first, then delete here**: + +```sh +# on nas: confirm the replicas are registered +incus list --project backup -c ns -f csv + +# on nuc: only then +incus delete --project backup --force +incus storage delete nucbackup +zpool status # only `data` should remain +``` + +### 9d. ks4 pull leg moves from nuc to nas + +The leg [ks2/nas-seed.md](../ks2/nas-seed.md) prepared. The target pool +moves, so the **host** WireGuard tunnel moves too — nuc drops `wg-ks4` +entirely once this works (its only other tunnel user, `transmission-bt`, +is now here and carries its own). + +- **ks4**: add nas as a peer on the `wireguard` container — + `wg set wg0 peer allowed-ips 10.8.0.22/32 && wg-quick save wg0`. + The existing ufw rule (`allow in on incusbr0 from 192.168.1.18 to any + port 8443 proto tcp`) already covers it: traffic arrives masqueraded as + the WG container whichever peer sent it. +- **nas**: `/etc/wireguard/wg-ks4.conf` modelled on nuc's — + `Address = 10.8.0.22/32`, peer pubkey `TVs6d7…`, + `Endpoint = 193.70.35.17:51845`, + `AllowedIPs = 10.8.0.0/24, 192.168.1.1/32`, keepalive 25 — then + `systemctl enable --now wg-quick@wg-ks4`. +- `incus remote add ks4 https://192.168.1.1:8443 --accept-certificate --token '…'` + (cross-check the fingerprint against the token). +- **nuc, after verification**: `systemctl disable --now wg-quick@wg-ks4`, + remove `/etc/wireguard/wg-ks4.conf`, and drop the `10.8.0.20/32` peer + on ks4. + +Seed manually in tmux — the first pass is **~1.75 TiB** over the home +downlink and only makes sense once FTTH is up: + +```sh +/root/scripts/incus-copy.sh -r ks4 -s ks4backup 2>&1 | tee -a /var/log/incus-copy-ks4.log +``` + +Then test-restore one instance before ticking the gate in +[ks2/plan.md](../ks2/plan.md). + +### Resulting schedule + +| When | Host | What | +|---|---|---| +| 03:00 | nuc, nas | instance snapshots (profile) | +| 03:30 | nuc | push all instances → `nas:nucbackup` (**systemd timer**, `Persistent=true` — catches up after boot) | +| 04:00 | nas | local copy of nas instances → `nasbackup` (`tank/backup/nas`) | +| 05:00 | nas | pull `ks4:*` → `ks4backup` (after FTTH) | +| 05:30 | nuc | apt upgrade all containers | +| Mon ~00:12 | nas | `zfs-scrub-weekly@tank.timer` / `@incus.timer` (systemd, not cron) | + +Staggered around ks4's own 01:00 / 05:00 jobs. + +⚠️ Keep `snapshots.schedule` on every source. A refresh with no common +snapshot degrades to a **full re-send** — the failure mode that cost +933 G on ks4 ([ks4/local-backup-cron.md](../ks4/local-backup-cron.md)). +The homeassistant **VM** on nuc re-sends its whole volume without them. +Note `incus-copy.sh` takes a **global** `flock`: an overrunning ks4 pull +aborts that night's other run loudly rather than racing it. + +## 10. Monitoring + +Do not repeat nuc's 11-day blind spot +([usb4t-dropouts.md](../nuc/usb4t-dropouts.md) — zed was running, but the +host had no MTA). + +**Status: live since 2026-08-30**, verified end to end (`smtpstatus=250`). + +- **msmtp**, same shape as nuc: this box shares the dynamic home IP with + no PTR and no SPF alignment, so it must use **submission (587) with + auth**, not port 25 — rspamd rejects the port-25 path as spam. + `/etc/msmtprc` mode 600, host `mail.lutran.fr`, STARTTLS. Give SMTP + tests ≥ 30 s: the missing PTR delays the greeting. + The `zed@lutran.fr` account is the **same credential as nuc** — it is + SMTP-AUTH, not IP-bound, so `/etc/msmtprc` can simply be copied between + hosts (mode 600, root:root). The password lives in the password + manager; no host-specific setup is needed. +- **zed**: `/etc/zfs/zed.d/zed.rc` mode 600 with + `ZED_EMAIL_ADDR="julien@lutran.fr"`, `ZED_EMAIL_PROG="mail"`, + `ZED_NOTIFY_VERBOSE=1`, **`ZED_NOTIFY_DATA=1`**, + `ZED_NOTIFY_INTERVAL_SECS=3600`. +- **[`zpool-health.sh`](https://git.lutran.fr/julien/scripts/src/branch/main/zpool-health.sh)** + every 15 min via `/etc/cron.d/zpool-health` — zed does *not* report a + suspended pool (the vdev stays `ONLINE`, so `statechange-notify.sh` + never fires). That watchdog is the only thing that catches the exact + failure this box was built to prevent. +- **smartd** — `/etc/smartd.conf`. The Toshiba carries the useful + attributes; the Intel is a liveness check only: + +``` +# Toshiba Q300 — endurance + temperature are real here +/dev/disk/by-id/ata-TOSHIBA_Q300._36OB318OK1KU -a -o on -S on -s (S/../.././02|L/../../6/03) -W 4,50,55 -m julien@lutran.fr +# Intel 330 — no error log, no devstat, no temperature: liveness only +/dev/disk/by-id/ata-INTEL_SSDSC2CT120A3_CVMP250400ES120BGN -H -s (S/../.././03|L/../../6/04) -m julien@lutran.fr +# WD Red +/dev/disk/by-id/ata-WDC_WD40EFRX-68WT0N0_WD-WCC4E6NLPJJE -a -o on -S on -s (S/../.././04|L/../../6/05) -W 4,45,50 -m julien@lutran.fr +``` + +Verify the whole chain the day you build it, not the day you need it: + +```sh +/root/scripts/zpool-health.sh -m julien@lutran.fr -t +tail -2 /var/log/msmtp.log # expect smtpstatus=250 +smartctl -d sat -l devstat /dev/sdX | grep -i endurance +``` + +## 11. Power baseline + +Take the measurement **before** the box goes into service, so later +readings mean something. Expect roughly 25–35 W idle with three disks: +the 20 W SoC plus a BMC drawing several watts continuously, even at +soft-off. + +```sh +# baseline: 10 min idle, everything settled +# CPU in steps — the interesting curve for an always-on box +for l in 25 50 75 100; do echo "=== ${l}% $(date +%s)"; stress-ng --cpu 0 --cpu-load $l --timeout 120s; done +# disk: random I/O is what moves an HDD's power, not throughput +fio --name=rr --directory=/export/media --size=20G --rw=randread --bs=4k \ + --iodepth=32 --numjobs=4 --ioengine=libaio --direct=1 --runtime=300 --time_based +``` + +Measure at the wall (PDU or smart plug) — RAPL is unreliable on Avoton +and sees neither disks nor fans. Log epoch timestamps per step so the +trace can be recut against the meter's series afterwards. + +## 12. Post-install checklist + +- [x] IPMI **left unplugged by decision** (2026-08-30) — no BMC on the + LAN, therefore no out-of-band console either +- [ ] `cat /proc/mdstat` — all three arrays `[UU]`; GRUB written to **both** + ESPs (`efibootmgr -v` shows two entries), + and boot tested with one disk unplugged +- [ ] `zpool status` healthy for `incus` and `tank`; weekly scrubs + scheduled; both import cleanly after a reboot (`cachefile` set) +- [ ] **7 days with zero pool suspensions and zero CRC errors** — the + gate that closes [usb4t-dropouts.md](../nuc/usb4t-dropouts.md) +- [x] `smartd` monitoring all 3 disks; `zpool-health.sh -t` mail + delivered (`smtpstatus=250` in `/var/log/msmtp.log`) — done + 2026-08-30 +- [ ] `transmission-bt` on nas: egress is `193.70.35.17`, **no default + route**, downloads land in `/export/media/downloads`, watch folder + works from the new path +- [ ] `jellyfin-server` on nuc lists the library over NFS after a **cold + reboot of both hosts** (the boot-ordering trap) +- [ ] `incus admin recover` ran on nas **before** the first push, so the + inherited `tank/backup/nuc` replicas refresh incrementally instead + of re-sending (check the first run's duration, not just `rc=0`) +- [ ] nuc's 03:30 leg → `nas:nucbackup`, `rc=0`, all nuc instances + present (`incus list --project backup -c ns -f csv` on nas) +- [ ] nas's 04:00 local leg → `nasbackup`, `rc=0`, and **blocky, + privoxy and transmission-bt all appear by name** in + `incus list --project backup` +- [ ] stale `nucbackup` pool removed from nuc (§9c), `zpool status` + shows only `data` +- [ ] replicas are **stopped** with `boot.autostart=false` — they hold + the live containers' LAN addresses +- [ ] ks4 pull leg seeded + one instance test-restored → tick the gate + in [ks2/plan.md](../ks2/plan.md); then retire nuc's `wg-ks4` +- [ ] power baseline recorded above, with the meter reading +- [ ] this file updated with what was built (RAM fitted, NIC name, WD Red + serial) diff --git a/nuc/transmission-bt.md b/nas/transmission-bt.md similarity index 88% rename from nuc/transmission-bt.md rename to nas/transmission-bt.md index 45d19be..e3fe580 100644 --- a/nuc/transmission-bt.md +++ b/nas/transmission-bt.md @@ -1,6 +1,13 @@ # transmission-bt -BitTorrent client in an unprivileged Incus container on `nuc`, with an + +> Moved from nuc to `nas` on 2026-08-30, together with the media +> dataset ([nas-install.md](nas-install.md) §7). Its WireGuard tunnel is +> entirely in-container, so ks4 needed no change — the peer is still +> `10.8.0.21`. The watch folder moved with it: +> `/export/media/.watchdir` on nas, not `/srv/media/.watchdir` on nuc. + +BitTorrent client in an unprivileged Incus container on `nas`, with an **always-on VPN**: all peer traffic exits via ks4's public IP through a WireGuard tunnel to the `wireguard` container on ks4. Kill switch by construction — `eth0` has **no default route**, so with the tunnel down @@ -12,10 +19,10 @@ the container simply has no path to the internet. whitelist (`192.168.0.*` only) - Egress: WG peer `10.8.0.21` → `193.70.35.17:51845`, `AllowedIPs 0.0.0.0/0` (verified: `curl ifconfig.me` from the container returns ks4's IP) -- Downloads: `/media/downloads` (= `usb4t/media`, same dataset Jellyfin +- Downloads: `/media/downloads` (= `tank/media`, same dataset Jellyfin reads); in-progress files in `/media/.incomplete` so Jellyfin never scans partials -- Watch folder: `scp` a `.torrent` into `/srv/media/.watchdir` on the host +- Watch folder: `scp` a `.torrent` into `/export/media/.watchdir` on the host and it auto-downloads (see "Watch folder" below) - `transmission-daemon` is `BindsTo=wg-quick@wg0.service` and binds peer traffic to `10.8.0.21` — three independent layers against leaks @@ -90,7 +97,7 @@ EOF' incus exec "$CNAME" -- systemctl enable --now wg-quick@wg0 # media share (same dataset as jellyfin-server) -incus config device add "$CNAME" media disk source=/srv/media path=/media shift=true +incus config device add "$CNAME" media disk source=/export/media path=/media shift=true incus exec "$CNAME" -- mkdir -p /media/downloads /media/.incomplete incus exec "$CNAME" -- chown debian-transmission:debian-transmission \ /media/downloads /media/.incomplete @@ -129,9 +136,9 @@ incus exec wireguard -- wg-quick save wg0 ## Watch folder (auto-add torrents) -Drop a `.torrent` into `/srv/media/.watchdir` on the host and Transmission +Drop a `.torrent` into `/export/media/.watchdir` on the host and Transmission auto-adds it and starts downloading — no web UI needed. The folder lives on -the shared `usb4t/media` dataset (`/media/.watchdir` inside the container). +the shared `tank/media` dataset (`/media/.watchdir` inside the container). Edit the **active** config only while the daemon is stopped (it rewrites `settings.json` on exit). The active file is @@ -161,7 +168,7 @@ incus exec transmission-bt -- systemctl start transmission-daemon Usage — the `.torrent` is consumed within a few seconds: ```sh -scp some.torrent root@192.168.0.3:/srv/media/.watchdir/ +scp some.torrent root@192.168.0.4:/export/media/.watchdir/ ``` - `watch-dir-force-generic: true` makes Transmission **poll** the folder @@ -179,7 +186,7 @@ incus exec transmission-bt -- wg show wg0 latest-handshakes # non-zero timesta incus exec transmission-bt -- curl -s https://ifconfig.me # must print 193.70.35.17 incus exec transmission-bt -- bash -c "ping -c1 -W2 8.8.8.8 || echo kill-switch OK" # with wg0 down # web UI must be tested from a LAN machine — the macvlan quirk means the -# nuc host itself cannot reach 192.168.0.7 +# nas host itself cannot reach 192.168.0.7 (macvlan, by design) ``` ## Notes