# nas — build procedure New storage host on the LAN (`192.168.0.4`), built 2026-08 from a Supermicro A1SAi-2750F. It exists to solve one specific problem: [usb4t-dropouts.md](../nuc/usb4t-dropouts.md) ruled out the cable and USB power management and left the **JMicron 152d:0578 bridge** as the cause of 61 disconnects in 30 days. The durable fix named there is a direct SATA connection — this box provides six of them. Why it matters beyond the media library: `usb4t` is the intended home of the ks4 off-site replicas, and *"the pool holding the off-site copy of ks4 must not be the least reliable device in the setup"*. That makes this build a **release gate for the ks2 decommission** ([ks2/plan.md](../ks2/plan.md), deadline Sep 30, 2026). It also takes over `transmission-bt` from nuc, so nuc keeps only what needs its iGPU. ## Hardware - Supermicro **A1SAi-2750F** mini-ITX, Intel Atom **C2750** (8 cores, 2.4 GHz Silvermont, 20 W SoC), AES-NI, no AVX - RAM: **2× 4 GB DDR3-1600 ECC SO-DIMM fitted = 8 GB** (`Single-bit ECC` confirmed), in DIMMA1/DIMMB1; **2 slots free**. Board takes 32 GB officially, 64 GB with 16 GB modules. ⚠️ 8 GB is modest for a 3.6 TB pool — ARC lands around 4 GB. Fine for streaming and replication (neither benefits much from cache), but the first thing to raise if metadata-heavy operations feel slow. - SATA: **2× SATA3 + 4× SATA2** (6 total) - BIOS **2.2** (2019-11-22) as shipped by the RMA - NIC: 4× GbE (Intel i354) + dedicated IPMI LAN - Video: **ASPEED AST2400 BMC only — there is no iGPU.** Avoton is a headless server SoC; `/dev/dri` is empty. That is why `jellyfin-server` stays on nuc (§8). ### ⚠️ AVR54 — already handled The C2750 is on the list of Atom C2000 parts affected by Intel's **AVR54** erratum: the SoC's `LPC_CLKOUT0/1` signals degrade and stop, after which the board never boots again — typically after ~18 months of power-on, i.e. exactly an always-on duty cycle. **This board was RMA'd by Supermicro for that issue and replaced**, so it carries the fix (C0 stepping or the LPC pull-up rework). Recorded here so a dead C2000 board is not re-diagnosed from scratch later. ### Disk plan | Port | Device | Role | |---|---|---| | SATA3-0 | Intel `SSDSC2CT120A3` (120 GB) | md mirror + `incus` pool | | SATA3-1 | Toshiba `Q300.` (120 GB) | md mirror + `incus` pool | | SATA2-0 | WD Red 4 TB (moved off the USB enclosure) | pool `tank` | | SATA2-1/2/3 | free | a second 4 TB to mirror `tank`, later | The SSDs get the SATA3 ports because they are the only devices that can use them: both negotiate 6 Gb/s and do ~450–500 MB/s, while the WD Red tops out near 180 MB/s and cannot saturate SATA2's ~270 MB/s. A future SLOG would also be fine on SATA2 — it is latency-bound on small sync writes, not bandwidth-bound. ⚠️ **Disconnect the 4 TB before partitioning the SSDs.** It carries the media library and nuc's replicas, nothing in the OS install needs it, and it keeps the two 120 GB disks unambiguous in the installer's list. Reconnect it before §5b. Device letters shift with enumeration order and mean nothing — mdraid assembles from superblock UUIDs, ZFS imports by GUID, fstab and GRUB use UUIDs. **Identify disks by model and serial** (`lsblk -o NAME,SIZE,MODEL,SERIAL`), never by letter: both SSDs are 120 GB, so size alone does not tell them apart. SSD health measured 2026-08-30 (both read over the JMicron bridge on nuc, `smartctl -d sat`), before deployment: | | Intel 330 (25 nm MLC) | Toshiba Q300 (15 nm TLC) | |---|---|---| | host writes | 16.0 TiB | 7.5 TiB | | endurance consumed | **0 %** (`Media_Wearout_Indicator` 100) | **7 %** (`Percentage Used Endurance Indicator`) | | power-on hours | **unreadable** — attr 9 decodes to 914,563 h on this family | 3,877 | | power cycles | 98 | 432 | | defects | 0 reallocated / program-fail / erase-fail | 0 reported uncorrectable | | SMART error log | **not supported** | supported, clean | | device statistics log | absent | full ACS-2 set | | short self-test | passed | passed | | interface CRC errors | n/a | 16 (baseline — watch for growth) | Both are healthy and far from wear-out; at OS-disk write rates endurance is not the binding constraint for either. They were previously a **matched pair** — identical layouts, `bpool` / `rpool` / `ubuntu:0` labels from an Ubuntu ZFS-on-root mirror — which is why they go back into a mirror here. ⚠️ **The Intel is effectively unmonitorable**: no error log, no device statistics, no temperature, no usable hours counter. `smartd` can watch the Toshiba properly and can only ask the Intel "are you still there". Expect the Intel to be found dead rather than found degrading. That asymmetry is the reason for the mirror. ### `tank` is a single vdev — accepted The 4 TB holds the media library, transmission's downloads and the ks4 replicas, with no redundancy. A single vdev can *detect* corruption but only self-heal metadata, not data — as seen in the usb4t incident. **Decision 2026-08-30: accepted, nothing on `tank` is irreplaceable.** Media and instances are re-fetchable from their sources; the ks4 replicas are leg 3 of ks4's 3-2-1 (local `sdb5` + restic/S3 remain — [backup-strategy.md](../backup-strategy.md)). Losing `tank` costs time, not data. Three SATA ports stay free, so `zpool attach tank ` turns it into a mirror whenever a spare 4 TB turns up. Not a prerequisite for anything. Scrub weekly regardless — on a single vdev the scrub is the only thing that *tells* you a file has rotted. ## 1. IPMI and BIOS first The AST2400 stack is old and has known vulnerabilities. - **Decision 2026-08-30: the BMC is never cabled.** The dedicated IPMI port stays unplugged, so the AST2400's default credentials and its known vulnerabilities are not reachable from anything. This is the simplest correct answer for a box that sits on a flat home LAN — no management VLAN needed, nothing to harden, nothing to patch. - ⚠️ Consequence: **there is no out-of-band console.** A boot that fails before sshd needs a physical monitor and keyboard. Worth knowing before changing anything that affects booting (GRUB, the md arrays, fstab). - BIOS: enable **restore-on-AC-loss**, enable C-states, and disable the three unused i354 NICs (each costs about a watt). ### Switching this board to UEFI There is **no "Boot Mode Select" entry** in this BIOS — that option exists on later Supermicro generations, not here. What works (verified 2026-08-30): - **CSM → Disabled** - **All OpROM policies → UEFI** (storage, video *and* network) Setting the **video** OpROM to UEFI is safe despite the console being the AST2400 BMC framebuffer — output survives, both on the BMC console and over IPMI KVM. ⚠️ **Confirm the mode before you partition anything**, from the installer (`Ctrl+Alt+F2`): ```sh ls /sys/firmware/efi # directory exists = UEFI. Missing = legacy ``` This single check is what makes the difference between a working install and an afternoon lost — see the trap in §3. ## 2. Prepare the SSDs Both report `ATA Security is: Disabled, NOT FROZEN`, so a real secure erase is available — do that rather than just repartitioning. It restores the full spare-block pool on ten-year-old NAND. ```sh # per disk, from a live system where the disk is NOT the running OS hdparm --user-master u --security-set-pass Eins /dev/sdX hdparm --user-master u --security-erase Eins /dev/sdX hdparm -I /dev/sdX | grep -A2 Security # expect "not enabled" ``` Identical GPT layout on both, sized so ~27 GB (22 %) stays **unallocated** as over-provisioning: | Part | Size | Type | Device | Use | |------|------|------|--------|-----| | `sdX1` | 1 GB | EFI System Partition | — | `/boot/efi` (two independent ESPs) | | `sdX2` | 2 GB | Linux RAID | `md0` | `/boot` ext4 | | `sdX3` | 24 GB | Linux RAID | `md1` | `/` ext4 | | `sdX4` | 2 GB | Linux RAID | `md2` | swap | | `sdX5` | 64 GB | Solaris root (bf00) | — | **zpool `incus`** (mirror) | | — | ~27 GB | **unallocated** | — | over-provisioning | - ⚠️ **The ESP is deliberately not a RAID1 array.** ks4 has `md1 → /boot/efi` because OVH's installer builds it with mdadm metadata **1.0** (superblock at the *end*, so firmware still sees plain FAT). `debian-installer` only creates metadata **1.2** arrays, whose superblock sits at the start and makes the ESP unreadable to firmware. So: one plain ESP per disk, only one mounted, the second filled by hand (§3). Deviation from ks4 is intentional. - A legacy-BIOS variant of this layout was tried first — 1 MB `bios_grub` instead of the ESP, which makes the mirror simpler (`grub-install` to both disks, nothing to keep in sync). It was abandoned because this board's firmware has no way to prefer legacy targets once CSM is off, and it kept falling through to the UEFI shell. Recorded so it is not retried: **UEFI is the working path here.** - `/` at 24 GB matches the other hosts (nuc 46 GB, ks4 40 GB) — this box has no desktop and no container roots on `/`. - Mixing md partitions and a ZFS partition on the same disks is exactly what ks4 does (`md1/2/3` + ZFS on `sda5`/`sdb5`). ### Why not full root-on-ZFS Decision 2026-08-30. These SSDs previously ran Ubuntu 20.04's experimental ZFS-root installer (hence the leftover `bpool` / `rpool` labels), so the option was on the table. Rejected because: - **Debian's installer cannot do it.** ZFS is CDDL, shipped only in `contrib` as `zfs-dkms`; `debian-installer` can neither partition nor boot from ZFS. Root-on-ZFS means the manual [OpenZFS Debian Trixie HOWTO](https://openzfs.github.io/openzfs-docs/Getting%20Started/Debian/Debian%20Trixie%20Root%20on%20ZFS.html) — ~60 steps from a live ISO. Per the repo convention the doc *is* the rebuild procedure, and that is a bad thing to be executing during an actual failure. - **DKMS failure mode.** `zfs-dkms` rebuilds on every kernel upgrade. If that build fails, root-on-ZFS means the box **does not boot**; with an ext4 md root it boots normally and only the pools are missing — recoverable over IPMI with a shell. - GRUB's ZFS support lags OpenZFS, which is why every root-on-ZFS guide needs a separate feature-limited `bpool`; `zpool upgrade bpool` is a known way to make a machine unbootable. - Ubuntu's version of this is a dead end anyway: the installer option was nearly dropped in 22.04 and `zsys`, which made boot environments useful, is abandoned. What root-on-ZFS would buy — snapshot and roll back a bad upgrade — is already covered where the state actually lives: container roots get incus snapshots plus nightly replication (§9). The host is 24 GB of packages reproducible from this file. Accepted cost: no pre-upgrade rollback of the host itself (`etckeeper` covers `/etc` if wanted). ## 3. Install Debian 13 (trixie) Netinst ISO (burned 2026-08-30, sha256 `65273bee…664e7`, verified against `cdimage.debian.org/debian-cd/13.6.0/amd64/iso-cd/SHA256SUMS`). - Manual partitioning per the table above: `sdX1` as ESP, `sdX2..4` as RAID1 members (three arrays), `sdX5` left untouched. - Tasks: **SSH server + standard utilities only**. No desktop — there is no GPU and the console is a BMC framebuffer. - Sources: `main contrib non-free-firmware` (`contrib` is required by `zfs-dkms`; the installer does not offer it — add it after first boot). Confirm d-i picked the right bootloader once installed: `dpkg -l | grep grub-efi` — `grub-efi-amd64`, not `grub-pc`. ### ⚠️ Trap: installer boot mode decides the bootloader (hit 2026-08-30) `debian-installer` chooses `grub-pc` or `grub-efi-amd64` from **how the installer itself booted**, not from what the disks look like. Booting the USB stick in legacy mode while the firmware prefers UEFI produces: 1. d-i installs `grub-pc`, targeting the MBR; 2. on a **GPT** disk that needs a 1 MB `bios_grub` partition — absent here, so `grub-install` fails, easy to click past; 3. the firmware then tries the SSDs as UEFI targets, finds no `.efi` binary, and drops to the **UEFI shell**. Nothing is corrupt; the halves simply disagree. Symptoms and checks: ```sh [ -d /sys/firmware/efi ] && echo UEFI || echo legacy # in the installer lsblk -no PTTYPE,PARTTYPENAME /dev/sdX # gpt + "EFI System"? dd if=/dev/sdX bs=440 count=1 2>/dev/null | od -c | head -3 # all \0 = no boot code dpkg -l | grep -E '^ii.*grub-(pc|efi)' # which flavour got installed ``` The board's hybrid ISO offers both paths, so the F11 boot menu usually lists the stick twice — picking the **`UEFI:`** entry avoids the whole thing. Checking `/sys/firmware/efi` before partitioning is the one step that prevents it. ### Second ESP — do this before trusting the mirror The installer populates only the ESP it mounted. Until the second one is written, losing that disk means the box does not boot, mirror or no mirror. ⚠️ **Use `/dev/disk/by-id/`, never `/dev/sdX`.** Reconnecting the 4 TB after the install shifts every letter — observed 2026-08-30: the HDD on SATA2-0 takes `sda` even with the SSDs on SATA3, because the SATA2 controller enumerates first on this SoC. A bare `/dev/sdb1` written during the install then points at a *different disk*, and here that would mean reformatting the ESP the system actually boots from. ```sh ESP2=/dev/disk/by-id/ata-TOSHIBA_Q300._36OB318OK1KU-part1 # the one NOT at /boot/efi mkfs.vfat -F32 "$ESP2" mkdir -p /boot/efi2 && mount "$ESP2" /boot/efi2 grub-install --target=x86_64-efi --efi-directory=/boot/efi2 \ --bootloader-id=debian-b --recheck efibootmgr -v # expect: debian, debian-b echo "UUID=$(blkid -s UUID -o value $ESP2) /boot/efi2 vfat umask=0077 0 1" >> /etc/fstab mount -a && findmnt /boot/efi2 ``` Re-run the `grub-install` after any GRUB or kernel change — the second ESP is **not** kept in sync automatically. **Verify by pulling one disk and booting**; the checklist item exists because an untested mirror is a guess, and this is the component most likely to be silently wrong. Do not fix the letter ordering by moving cables: the only arrangement that makes the SSDs `sda`/`sdb` puts them on SATA2 and the HDD on SATA3, which caps the only devices that can use 6 Gb/s and gives the bandwidth to a disk that tops out near 180 MB/s. ## 4. Base system ```sh apt update && apt full-upgrade -y apt install -y \ linux-headers-amd64 zfs-dkms zfsutils-linux zfs-zed \ mdadm smartmontools nfs-kernel-server \ msmtp msmtp-mta bsd-mailx \ curl vim htop ripgrep sysstat dmidecode pciutils usbutils \ stress-ng fio ``` Static network — `/etc/network/interfaces` (ifupdown, matching nuc): ``` source /etc/network/interfaces.d/* auto lo iface lo inet loopback allow-hotplug enp0s20f0 iface enp0s20f0 inet static address 192.168.0.4 netmask 255.255.255.0 gateway 192.168.0.2 dns-nameservers 1.1.1.1 9.9.9.9 ``` (Gateway is **`192.168.0.2`** — `.1` is gone since 2026-08. The host uses public resolvers, never blocky, to avoid a bootstrap loop. Interface name is a guess until the board is up — check `ip -br link`.) Restore `/root/.ssh/authorized_keys` (incl. `id_rsa_claude.pub`) and `timedatectl set-timezone Europe/Paris`. ## 5. Pools ### 5a. `incus` — SSD mirror ```sh zpool create -o ashift=12 \ -O compression=zstd -O atime=off -O xattr=sa -O acltype=posixacl \ incus mirror \ /dev/disk/by-id/ata-INTEL_SSDSC2CT120A3_CVMP250400ES120BGN-part5 \ /dev/disk/by-id/ata-TOSHIBA_Q300._36OB318OK1KU-part5 zpool set autotrim=on incus ``` Weekly scrubs come from the packaged systemd timers rather than cron: ```sh systemctl enable --now zfs-scrub-weekly@tank.timer zfs-scrub-weekly@incus.timer systemctl list-timers 'zfs-scrub*' ``` `autotrim` matters on ten-year-old NAND — it is what keeps the unallocated 22 % actually available to the controller as spare. ### 5b. `tank` — move the 4 TB off USB onto SATA The risky step. The pool holds `usb4t/backup/nuc` (nuc's replicas), `usb4t/backup/ks4` (empty, awaiting the FTTH seed) and `usb4t/media`. ⚠️ **Between export and import, nuc has no replica target and Jellyfin has no media.** Plan a maintenance window and disable nuc's 03:30 cron first, so it fails loudly rather than half-running. ```sh # --- on nuc, first --- zpool scrub usb4t # start clean; wait for it zpool status usb4t zpool export usb4t ``` Move the disk to SATA2-0 (reconnect it now if you unplugged it for the install), then: ```sh # --- on nas --- zpool import # confirm it is seen zpool import usb4t tank # rename: it is not USB any more zpool set cachefile=/etc/zfs/zpool.cache tank zpool status tank ``` Properties survive from creation (`ashift=12`, `compression=zstd`, `atime=off`, `xattr=sa`, `acltype=posixacl`). Re-point the mountpoints and add the two new backup datasets: ```sh zfs set mountpoint=/export/media tank/media # NFSv4 export root (§8) zfs set mountpoint=none tank/backup zfs list -o name,used,avail,mountpoint ``` Resulting layout: ``` incus mirror, 2× SSD — nas's own container roots tank 4 TB, single vdev ├── tank/media → /export/media NFS ro → nuc; local device → transmission-bt └── tank/backup ├── tank/backup/nuc → incus pool `nucbackup` (nuc pushes here) └── tank/backup/ks4 → incus pool `ks4backup` (nas pulls from ks4) nas's own replicas live on **nuc** (`data/backup/nas` → pool `nasbackup`), not here — see §9. ``` Backup pools are named after the **source** host, matching [ks4/incus-copy.md](../ks4/incus-copy.md). nas's *own* instances are not backed up here — they cross-replicate to nuc (§9b), so neither host's instances depend on that host surviving. Once `tank` has run a week on SATA with **zero** pool suspensions and no CRC errors, the `usb4t-dropouts` gate is cleared — record that in [ks2/plan.md](../ks2/plan.md). Day 1 was clean (2026-08-31). ### Clearing the inherited `` errors — order matters The pool imported carrying `:<0x0>` and `:<0x3d>` from the 2026-08-29 USB dropout. A scrub found **0 errors and repaired 0B**, yet the entries stayed, and a plain `zpool clear` afterwards did not drop them either. ZFS flushes its persistent error log on a scrub that runs **after** the clear — so the working order is: ```sh zpool clear tank zpool scrub tank # this is the run that flushes the log ``` Result 2026-08-31: `scrub repaired 0B in 02:34:24 with 0 errors`, `errors: No known data errors`, `all pools are healthy`. They were artefacts of interrupted writes, not corruption — matching the [2026-08-28 incident](../nuc/usb4t-dropouts.md). ⚠️ This matters for monitoring, not just tidiness: while those entries stand, `zpool status -x` reports the pool unhealthy permanently, so `zpool-health.sh` sits in the alarm state and **cannot signal a new problem**. Clear them before trusting the watchdog. ## 6. Incus Same Zabbly stable repo as nuc and ks4: ```sh mkdir -p /etc/apt/keyrings curl -fsSL https://pkgs.zabbly.com/key.asc -o /etc/apt/keyrings/zabbly.asc cat > /etc/apt/sources.list.d/zabbly-incus-stable.sources <> /etc/exports <<'EOF' /export 192.168.0.3(ro,fsid=0,crossmnt,no_subtree_check) /export/media 192.168.0.3(ro,no_subtree_check,all_squash,anonuid=65534,anongid=65534) EOF exportfs -ra && exportfs -v ``` ```sh # --- nuc: mount at the SAME path, so jellyfin-server.md still applies --- mkdir -p /srv/media echo '192.168.0.4:/media /srv/media nfs4 ro,_netdev,soft,timeo=100,retrans=3 0 0' >> /etc/fstab mount /srv/media && ls /srv/media ``` The container device changes only in losing the shift — per [nuc/jellyfin-server.md](../nuc/jellyfin-server.md)'s own troubleshooting note, *"on CIFS files are world-readable synthetic ownership, enough for a read-only library"*; the same holds for NFS with `all_squash`: ```sh incus stop jellyfin-server # shift cannot be hot-applied incus config device set jellyfin-server media shift=false incus config device set jellyfin-server media readonly=true incus start jellyfin-server incus exec jellyfin-server -- ls /media # must list the library ``` Requirements for that to work: media files must be **world-readable** (`find /export/media -type f ! -perm -o=r`), and transmission must keep creating them that way (it does — `umask`/`0775` per its doc). ⚠️ Boot ordering on nuc: incus is already ordered after `zfs-mount.service`; add `remote-fs.target` to that drop-in, or `jellyfin-server` starts against an empty mountpoint and shows an empty library. ⚠️ `soft` mount is deliberate: a hung NAS should fail Jellyfin's reads, not wedge nuc's processes in uninterruptible sleep the way the suspended `usb4t` pool did. ## 9. Backup legs The driver is unchanged — [`incus-copy.sh`](https://git.lutran.fr/julien/scripts/src/branch/main/incus-copy.sh), deployed to `/root/scripts` as everywhere else. ### 9a. nuc's instances -> nas Root crontab on nuc: ```cron 30 3 * * * /root/scripts/incus-copy.sh -d nas -m push -s nucbackup -p backup >> /var/log/incus-copy.log 2>&1 ``` ⚠️ **nuc is an on-demand media box** (see [nuc/README.md](../nuc/README.md)): since 2026-08-30 it only runs when watching Jellyfin or using the Spotify kiosk, so it is often powered off at 03:30 and **that night's push is simply skipped** — cron does not catch up missed windows. Accepted deliberately (2026-08-31): nuc's instances change rarely and the next time it is up the refresh is incremental anyway. This ran briefly as a `systemd` timer with `Persistent=true` (which *does* catch up after boot); the units are still on disk, disabled, at `/etc/systemd/system/incus-copy.{service,timer}` if that behaviour is ever wanted back: `systemctl enable --now incus-copy.timer` (and remove the cron line). ⚠️ **`-p backup` is not optional.** Without it the replicas land in `default` on nas and collide with nas's *live* instances — both hosts are on the same macvlan LAN and the replicas carry the same static IPs. ### 9b. nas's own instances -> local pool `nasbackup` **Decision 2026-08-30: local, not cross-replicated.** nuc is powered off most of the time, so it is not a usable backup target — a nightly push to it would fail noisily and, once mail works, alarm every morning. The replicas instead go to `tank/backup/nas`, which is a **different pool** from the instances themselves (`incus`, the SSD mirror), so it survives losing that mirror. All three instances here are rebuildable from their docs, so same-host is proportionate. ```sh zfs create tank/backup/nas zfs set mountpoint=legacy tank/backup/nas # see the trap below incus storage create nasbackup zfs source=tank/backup/nas ``` ```cron # nas, root crontab 0 4 * * * root /root/scripts/incus-copy.sh -p backup -s nasbackup >> /var/log/incus-copy.log 2>&1 ``` ⚠️ **Trap: `incus storage create` hangs forever on a `mountpoint=none` dataset.** `tank/backup` is set to `mountpoint=none`, so any child created afterwards inherits it, and the pool create then blocks with no error and no entry in `incus operation list` — it looks exactly like I/O contention (a scrub was running, which sent me down that path for 20 minutes). Set the child to `legacy` to match its siblings first, and it completes instantly. ### 9c. Cleanup on nuc after the pool move Exporting `usb4t` leaves nuc with an incus storage pool whose backing dataset is gone, plus replica records in the `backup` project pointing at it. Remove them once §6's `incus admin recover` has re-registered the same volumes on nas — **verify there first, then delete here**: ```sh # on nas: confirm the replicas are registered incus list --project backup -c ns -f csv # on nuc: only then incus delete --project backup --force incus storage delete nucbackup zpool status # only `data` should remain ``` ### 9d. ks4 pull leg moves from nuc to nas The leg [ks2/nas-seed.md](../ks2/nas-seed.md) prepared. The target pool moves, so the **host** WireGuard tunnel moves too — nuc drops `wg-ks4` entirely once this works (its only other tunnel user, `transmission-bt`, is now here and carries its own). - **ks4**: add nas as a peer on the `wireguard` container — `wg set wg0 peer allowed-ips 10.8.0.22/32 && wg-quick save wg0`. The existing ufw rule (`allow in on incusbr0 from 192.168.1.18 to any port 8443 proto tcp`) already covers it: traffic arrives masqueraded as the WG container whichever peer sent it. - **nas**: `/etc/wireguard/wg-ks4.conf` modelled on nuc's — `Address = 10.8.0.22/32`, peer pubkey `TVs6d7…`, `Endpoint = 193.70.35.17:51845`, `AllowedIPs = 10.8.0.0/24, 192.168.1.1/32`, keepalive 25 — then `systemctl enable --now wg-quick@wg-ks4`. - `incus remote add ks4 https://192.168.1.1:8443 --accept-certificate --token '…'` (cross-check the fingerprint against the token). - **nuc, after verification**: `systemctl disable --now wg-quick@wg-ks4`, remove `/etc/wireguard/wg-ks4.conf`, and drop the `10.8.0.20/32` peer on ks4. Seed manually in tmux — the first pass is **~1.75 TiB** over the home downlink and only makes sense once FTTH is up: ```sh /root/scripts/incus-copy.sh -r ks4 -s ks4backup 2>&1 | tee -a /var/log/incus-copy-ks4.log ``` Then test-restore one instance before ticking the gate in [ks2/plan.md](../ks2/plan.md). ### Resulting schedule | When | Host | What | |---|---|---| | 03:00 | nuc, nas | instance snapshots (profile) | | 03:30 | nuc | push all instances → `nas:nucbackup` (root crontab; skipped when nuc is off) | | 04:00 | nas | local copy of nas instances → `nasbackup` (`tank/backup/nas`) | | 05:00 | nas | pull `ks4:*` → `ks4backup` (after FTTH) | | 05:30 | nuc | apt upgrade all containers | | Mon ~00:12 | nas | `zfs-scrub-weekly@tank.timer` / `@incus.timer` (systemd, not cron) | Staggered around ks4's own 01:00 / 05:00 jobs. ⚠️ Keep `snapshots.schedule` on every source. A refresh with no common snapshot degrades to a **full re-send** — the failure mode that cost 933 G on ks4 ([ks4/local-backup-cron.md](../ks4/local-backup-cron.md)). The homeassistant **VM** on nuc re-sends its whole volume without them. Note `incus-copy.sh` takes a **global** `flock`: an overrunning ks4 pull aborts that night's other run loudly rather than racing it. ## 10. Monitoring Do not repeat nuc's 11-day blind spot ([usb4t-dropouts.md](../nuc/usb4t-dropouts.md) — zed was running, but the host had no MTA). **Status: live since 2026-08-30**, verified end to end (`smtpstatus=250`). - **msmtp**, same shape as nuc: this box shares the dynamic home IP with no PTR and no SPF alignment, so it must use **submission (587) with auth**, not port 25 — rspamd rejects the port-25 path as spam. `/etc/msmtprc` mode 600, host `mail.lutran.fr`, STARTTLS. Give SMTP tests ≥ 30 s: the missing PTR delays the greeting. The `zed@lutran.fr` account is the **same credential as nuc** — it is SMTP-AUTH, not IP-bound, so `/etc/msmtprc` can simply be copied between hosts (mode 600, root:root). The password lives in the password manager; no host-specific setup is needed. - **zed**: `/etc/zfs/zed.d/zed.rc` mode 600 with `ZED_EMAIL_ADDR="julien@lutran.fr"`, `ZED_EMAIL_PROG="mail"`, `ZED_NOTIFY_VERBOSE=1`, **`ZED_NOTIFY_DATA=1`**, `ZED_NOTIFY_INTERVAL_SECS=3600`. - **[`zpool-health.sh`](https://git.lutran.fr/julien/scripts/src/branch/main/zpool-health.sh)** every 15 min from root's crontab — zed does *not* report a suspended pool (the vdev stays `ONLINE`, so `statechange-notify.sh` never fires). That watchdog is the only thing that catches the exact failure this box was built to prevent. - **smartd** — `/etc/smartd.conf`. The Toshiba carries the useful attributes; the Intel is a liveness check only: ``` # Toshiba Q300 — endurance + temperature are real here /dev/disk/by-id/ata-TOSHIBA_Q300._36OB318OK1KU -a -o on -S on -s (S/../.././02|L/../../6/03) -W 4,50,55 -m julien@lutran.fr # Intel 330 — no error log, no devstat, no temperature: liveness only /dev/disk/by-id/ata-INTEL_SSDSC2CT120A3_CVMP250400ES120BGN -H -s (S/../.././03|L/../../6/04) -m julien@lutran.fr # WD Red /dev/disk/by-id/ata-WDC_WD40EFRX-68WT0N0_WD-WCC4E6NLPJJE -a -o on -S on -s (S/../.././04|L/../../6/05) -W 4,45,50 -m julien@lutran.fr ``` Verify the whole chain the day you build it, not the day you need it: ```sh /root/scripts/zpool-health.sh -m julien@lutran.fr -t tail -2 /var/log/msmtp.log # expect smtpstatus=250 smartctl -d sat -l devstat /dev/sdX | grep -i endurance ``` ## 11. Power baseline Take the measurement **before** the box goes into service, so later readings mean something. Expect roughly 25–35 W idle with three disks: the 20 W SoC plus a BMC drawing several watts continuously, even at soft-off. ```sh # baseline: 10 min idle, everything settled # CPU in steps — the interesting curve for an always-on box for l in 25 50 75 100; do echo "=== ${l}% $(date +%s)"; stress-ng --cpu 0 --cpu-load $l --timeout 120s; done # disk: random I/O is what moves an HDD's power, not throughput fio --name=rr --directory=/export/media --size=20G --rw=randread --bs=4k \ --iodepth=32 --numjobs=4 --ioengine=libaio --direct=1 --runtime=300 --time_based ``` Measure at the wall (PDU or smart plug) — RAPL is unreliable on Avoton and sees neither disks nor fans. Log epoch timestamps per step so the trace can be recut against the meter's series afterwards. ## 12. Post-install checklist - [x] IPMI **left unplugged by decision** (2026-08-30) — no BMC on the LAN, therefore no out-of-band console either - [x] `cat /proc/mdstat` — all three arrays `[UU]`; GRUB written to **both** ESPs, both mounted, `debian` + `debian-b` boot entries present (verified across a cold boot 2026-08-31) - [ ] boot still untested with **one disk physically unplugged** — the mirror is a guess until that is done - [x] `zpool status` healthy for `incus` and `tank`; weekly scrubs scheduled; **cold boot verified 2026-08-31** — both pools imported from `/etc/zfs/zpool.cache`, all instances autostarted, NFS exports republished, 0 failed units - [ ] **7 days with zero pool suspensions and zero CRC errors** — the gate that closes [usb4t-dropouts.md](../nuc/usb4t-dropouts.md) - [x] `smartd` monitoring all 3 disks; `zpool-health.sh -t` mail delivered (`smtpstatus=250` in `/var/log/msmtp.log`) — done 2026-08-30 - [ ] `transmission-bt` on nas: egress is `193.70.35.17`, **no default route**, downloads land in `/export/media/downloads`, watch folder works from the new path - [ ] `jellyfin-server` on nuc lists the library over NFS after a **cold reboot of both hosts** (the boot-ordering trap) - [ ] `incus admin recover` ran on nas **before** the first push, so the inherited `tank/backup/nuc` replicas refresh incrementally instead of re-sending (check the first run's duration, not just `rc=0`) - [ ] nuc's 03:30 leg → `nas:nucbackup`, `rc=0`, all nuc instances present (`incus list --project backup -c ns -f csv` on nas) - [ ] nas's 04:00 local leg → `nasbackup`, `rc=0`, and **blocky, privoxy and transmission-bt all appear by name** in `incus list --project backup` - [ ] stale `nucbackup` pool removed from nuc (§9c), `zpool status` shows only `data` - [ ] replicas are **stopped** with `boot.autostart=false` — they hold the live containers' LAN addresses - [ ] ks4 pull leg seeded + one instance test-restored → tick the gate in [ks2/plan.md](../ks2/plan.md); then retire nuc's `wg-ks4` - [ ] power baseline recorded above, with the meter reading - [ ] this file updated with what was built (RAM fitted, NIC name, WD Red serial)