doc/nas/nas-install.md

# nas — build procedure

New storage host on the LAN (`192.168.0.4`), built 2026-08 from a
Supermicro A1SAi-2750F. It exists to solve one specific problem:
[usb4t-dropouts.md](../nuc/usb4t-dropouts.md) ruled out the cable and
USB power management and left the **JMicron 152d:0578 bridge** as the
cause of 61 disconnects in 30 days. The durable fix named there is a
direct SATA connection — this box provides six of them.

Why it matters beyond the media library: `usb4t` is the intended home
of the ks4 off-site replicas, and *"the pool holding the off-site copy
of ks4 must not be the least reliable device in the setup"*. That makes
this build a **release gate for the ks2 decommission**
([ks2/plan.md](../ks2/plan.md), deadline Sep 30, 2026).

It also takes over `transmission-bt` from nuc, so nuc keeps only what
needs its iGPU.

## Hardware

- Supermicro **A1SAi-2750F** mini-ITX, Intel Atom **C2750** (8 cores,
  2.4 GHz Silvermont, 20 W SoC), AES-NI, no AVX
- RAM: **2× 4 GB DDR3-1600 ECC SO-DIMM fitted = 8 GB** (`Single-bit ECC`
  confirmed), in DIMMA1/DIMMB1; **2 slots free**. Board takes 32 GB
  officially, 64 GB with 16 GB modules.
  ⚠️ 8 GB is modest for a 3.6 TB pool — ARC lands around 4 GB. Fine for
  streaming and replication (neither benefits much from cache), but the
  first thing to raise if metadata-heavy operations feel slow.
- SATA: **2× SATA3 + 4× SATA2** (6 total)
- BIOS **2.2** (2019-11-22) as shipped by the RMA
- NIC: 4× GbE (Intel i354) + dedicated IPMI LAN
- Video: **ASPEED AST2400 BMC only — there is no iGPU.** Avoton is a
  headless server SoC; `/dev/dri` is empty. That is why `jellyfin-server`
  stays on nuc (§8).

### ⚠️ AVR54 — already handled

The C2750 is on the list of Atom C2000 parts affected by Intel's
**AVR54** erratum: the SoC's `LPC_CLKOUT0/1` signals degrade and stop,
after which the board never boots again — typically after ~18 months of
power-on, i.e. exactly an always-on duty cycle. **This board was RMA'd
by Supermicro for that issue and replaced**, so it carries the fix
(C0 stepping or the LPC pull-up rework). Recorded here so a dead C2000
board is not re-diagnosed from scratch later.

### Disk plan

| Port | Device | Role |
|---|---|---|
| SATA3-0 | Intel `SSDSC2CT120A3` (120 GB) | md mirror + `incus` pool |
| SATA3-1 | Toshiba `Q300.` (120 GB) | md mirror + `incus` pool |
| SATA2-0 | WD Red 4 TB (moved off the USB enclosure) | pool `tank` |
| SATA2-1/2/3 | free | a second 4 TB to mirror `tank`, later |

The SSDs get the SATA3 ports because they are the only devices that can
use them: both negotiate 6 Gb/s and do ~450–500 MB/s, while the WD Red
tops out near 180 MB/s and cannot saturate SATA2's ~270 MB/s. A future
SLOG would also be fine on SATA2 — it is latency-bound on small sync
writes, not bandwidth-bound.

⚠️ **Disconnect the 4 TB before partitioning the SSDs.** It carries the
media library and nuc's replicas, nothing in the OS install needs it,
and it keeps the two 120 GB disks unambiguous in the installer's list.
Reconnect it before §5b.

Device letters shift with enumeration order and mean nothing — mdraid
assembles from superblock UUIDs, ZFS imports by GUID, fstab and GRUB use
UUIDs. **Identify disks by model and serial**
(`lsblk -o NAME,SIZE,MODEL,SERIAL`), never by letter: both SSDs are
120 GB, so size alone does not tell them apart.

SSD health measured 2026-08-30 (both read over the JMicron bridge on
nuc, `smartctl -d sat`), before deployment:

| | Intel 330 (25 nm MLC) | Toshiba Q300 (15 nm TLC) |
|---|---|---|
| host writes | 16.0 TiB | 7.5 TiB |
| endurance consumed | **0 %** (`Media_Wearout_Indicator` 100) | **7 %** (`Percentage Used Endurance Indicator`) |
| power-on hours | **unreadable** — attr 9 decodes to 914,563 h on this family | 3,877 |
| power cycles | 98 | 432 |
| defects | 0 reallocated / program-fail / erase-fail | 0 reported uncorrectable |
| SMART error log | **not supported** | supported, clean |
| device statistics log | absent | full ACS-2 set |
| short self-test | passed | passed |
| interface CRC errors | n/a | 16 (baseline — watch for growth) |

Both are healthy and far from wear-out; at OS-disk write rates endurance
is not the binding constraint for either. They were previously a
**matched pair** — identical layouts, `bpool` / `rpool` / `ubuntu:0`
labels from an Ubuntu ZFS-on-root mirror — which is why they go back
into a mirror here.

⚠️ **The Intel is effectively unmonitorable**: no error log, no device
statistics, no temperature, no usable hours counter. `smartd` can watch
the Toshiba properly and can only ask the Intel "are you still there".
Expect the Intel to be found dead rather than found degrading. That
asymmetry is the reason for the mirror.

### `tank` is a single vdev — accepted

The 4 TB holds the media library, transmission's downloads and the ks4
replicas, with no redundancy. A single vdev can *detect* corruption but
only self-heal metadata, not data — as seen in the usb4t incident.

**Decision 2026-08-30: accepted, nothing on `tank` is irreplaceable.**
Media and instances are re-fetchable from their sources; the ks4
replicas are leg 3 of ks4's 3-2-1 (local `sdb5` + restic/S3 remain —
[backup-strategy.md](../backup-strategy.md)). Losing `tank` costs time,
not data.

Three SATA ports stay free, so `zpool attach tank <existing> <new>`
turns it into a mirror whenever a spare 4 TB turns up. Not a
prerequisite for anything. Scrub weekly regardless — on a single vdev
the scrub is the only thing that *tells* you a file has rotted.

## 1. IPMI and BIOS first

The AST2400 stack is old and has known vulnerabilities.

- **Decision 2026-08-30: the BMC is never cabled.** The dedicated IPMI
  port stays unplugged, so the AST2400's default credentials and its
  known vulnerabilities are not reachable from anything. This is the
  simplest correct answer for a box that sits on a flat home LAN — no
  management VLAN needed, nothing to harden, nothing to patch.
- ⚠️ Consequence: **there is no out-of-band console.** A boot that fails
  before sshd needs a physical monitor and keyboard. Worth knowing before
  changing anything that affects booting (GRUB, the md arrays, fstab).
- BIOS: enable **restore-on-AC-loss**, enable C-states, and disable the
  three unused i354 NICs (each costs about a watt).

### Switching this board to UEFI

There is **no "Boot Mode Select" entry** in this BIOS — that option
exists on later Supermicro generations, not here. What works
(verified 2026-08-30):

- **CSM → Disabled**
- **All OpROM policies → UEFI** (storage, video *and* network)

Setting the **video** OpROM to UEFI is safe despite the console being
the AST2400 BMC framebuffer — output survives, both on the BMC console
and over IPMI KVM.

⚠️ **Confirm the mode before you partition anything**, from the
installer (`Ctrl+Alt+F2`):

```sh
ls /sys/firmware/efi     # directory exists = UEFI. Missing = legacy
```

This single check is what makes the difference between a working
install and an afternoon lost — see the trap in §3.

## 2. Prepare the SSDs

Both report `ATA Security is: Disabled, NOT FROZEN`, so a real secure
erase is available — do that rather than just repartitioning. It
restores the full spare-block pool on ten-year-old NAND.

```sh
# per disk, from a live system where the disk is NOT the running OS
hdparm --user-master u --security-set-pass Eins /dev/sdX
hdparm --user-master u --security-erase   Eins /dev/sdX
hdparm -I /dev/sdX | grep -A2 Security      # expect "not enabled"
```

Identical GPT layout on both, sized so ~27 GB (22 %) stays
**unallocated** as over-provisioning:

| Part | Size | Type | Device | Use |
|------|------|------|--------|-----|
| `sdX1` | 1 GB | EFI System Partition | — | `/boot/efi` (two independent ESPs) |
| `sdX2` | 2 GB | Linux RAID | `md0` | `/boot` ext4 |
| `sdX3` | 24 GB | Linux RAID | `md1` | `/` ext4 |
| `sdX4` | 2 GB | Linux RAID | `md2` | swap |
| `sdX5` | 64 GB | Solaris root (bf00) | — | **zpool `incus`** (mirror) |
| — | ~27 GB | **unallocated** | — | over-provisioning |

- ⚠️ **The ESP is deliberately not a RAID1 array.** ks4 has `md1 →
  /boot/efi` because OVH's installer builds it with mdadm metadata
  **1.0** (superblock at the *end*, so firmware still sees plain FAT).
  `debian-installer` only creates metadata **1.2** arrays, whose
  superblock sits at the start and makes the ESP unreadable to
  firmware. So: one plain ESP per disk, only one mounted, the second
  filled by hand (§3). Deviation from ks4 is intentional.
- A legacy-BIOS variant of this layout was tried first — 1 MB
  `bios_grub` instead of the ESP, which makes the mirror simpler
  (`grub-install` to both disks, nothing to keep in sync). It was
  abandoned because this board's firmware has no way to prefer legacy
  targets once CSM is off, and it kept falling through to the UEFI
  shell. Recorded so it is not retried: **UEFI is the working path
  here.**
- `/` at 24 GB matches the other hosts (nuc 46 GB, ks4 40 GB) — this box
  has no desktop and no container roots on `/`.
- Mixing md partitions and a ZFS partition on the same disks is exactly
  what ks4 does (`md1/2/3` + ZFS on `sda5`/`sdb5`).

### Why not full root-on-ZFS

Decision 2026-08-30. These SSDs previously ran Ubuntu 20.04's
experimental ZFS-root installer (hence the leftover `bpool` / `rpool`
labels), so the option was on the table. Rejected because:

- **Debian's installer cannot do it.** ZFS is CDDL, shipped only in
  `contrib` as `zfs-dkms`; `debian-installer` can neither partition nor
  boot from ZFS. Root-on-ZFS means the manual
  [OpenZFS Debian Trixie HOWTO](https://openzfs.github.io/openzfs-docs/Getting%20Started/Debian/Debian%20Trixie%20Root%20on%20ZFS.html)
  — ~60 steps from a live ISO. Per the repo convention the doc *is* the
  rebuild procedure, and that is a bad thing to be executing during an
  actual failure.
- **DKMS failure mode.** `zfs-dkms` rebuilds on every kernel upgrade.
  If that build fails, root-on-ZFS means the box **does not boot**;
  with an ext4 md root it boots normally and only the pools are
  missing — recoverable over IPMI with a shell.
- GRUB's ZFS support lags OpenZFS, which is why every root-on-ZFS guide
  needs a separate feature-limited `bpool`; `zpool upgrade bpool` is a
  known way to make a machine unbootable.
- Ubuntu's version of this is a dead end anyway: the installer option
  was nearly dropped in 22.04 and `zsys`, which made boot environments
  useful, is abandoned.

What root-on-ZFS would buy — snapshot and roll back a bad upgrade — is
already covered where the state actually lives: container roots get
incus snapshots plus nightly replication (§9). The host is 24 GB of
packages reproducible from this file. Accepted cost: no pre-upgrade
rollback of the host itself (`etckeeper` covers `/etc` if wanted).

## 3. Install Debian 13 (trixie)

Netinst ISO (burned 2026-08-30, sha256 `65273bee…664e7`, verified
against `cdimage.debian.org/debian-cd/13.6.0/amd64/iso-cd/SHA256SUMS`).

- Manual partitioning per the table above: `sdX1` as ESP, `sdX2..4` as
  RAID1 members (three arrays), `sdX5` left untouched.
- Tasks: **SSH server + standard utilities only**. No desktop — there is
  no GPU and the console is a BMC framebuffer.
- Sources: `main contrib non-free-firmware` (`contrib` is required by
  `zfs-dkms`; the installer does not offer it — add it after first boot).

Confirm d-i picked the right bootloader once installed:
`dpkg -l | grep grub-efi` — `grub-efi-amd64`, not `grub-pc`.

### ⚠️ Trap: installer boot mode decides the bootloader (hit 2026-08-30)

`debian-installer` chooses `grub-pc` or `grub-efi-amd64` from **how the
installer itself booted**, not from what the disks look like. Booting the
USB stick in legacy mode while the firmware prefers UEFI produces:

1. d-i installs `grub-pc`, targeting the MBR;
2. on a **GPT** disk that needs a 1 MB `bios_grub` partition — absent
   here, so `grub-install` fails, easy to click past;
3. the firmware then tries the SSDs as UEFI targets, finds no `.efi`
   binary, and drops to the **UEFI shell**.

Nothing is corrupt; the halves simply disagree. Symptoms and checks:

```sh
[ -d /sys/firmware/efi ] && echo UEFI || echo legacy   # in the installer
lsblk -no PTTYPE,PARTTYPENAME /dev/sdX                 # gpt + "EFI System"?
dd if=/dev/sdX bs=440 count=1 2>/dev/null | od -c | head -3   # all \0 = no boot code
dpkg -l | grep -E '^ii.*grub-(pc|efi)'                 # which flavour got installed
```

The board's hybrid ISO offers both paths, so the F11 boot menu usually
lists the stick twice — picking the **`UEFI:`** entry avoids the whole
thing. Checking `/sys/firmware/efi` before partitioning is the one step
that prevents it.

### Second ESP — do this before trusting the mirror

The installer populates only the ESP it mounted. Until the second one is
written, losing that disk means the box does not boot, mirror or no
mirror.

⚠️ **Use `/dev/disk/by-id/`, never `/dev/sdX`.** Reconnecting the 4 TB
after the install shifts every letter — observed 2026-08-30: the HDD on
SATA2-0 takes `sda` even with the SSDs on SATA3, because the SATA2
controller enumerates first on this SoC. A bare `/dev/sdb1` written
during the install then points at a *different disk*, and here that
would mean reformatting the ESP the system actually boots from.

```sh
ESP2=/dev/disk/by-id/ata-TOSHIBA_Q300._36OB318OK1KU-part1   # the one NOT at /boot/efi
mkfs.vfat -F32 "$ESP2"
mkdir -p /boot/efi2 && mount "$ESP2" /boot/efi2
grub-install --target=x86_64-efi --efi-directory=/boot/efi2 \
             --bootloader-id=debian-b --recheck
efibootmgr -v                       # expect: debian, debian-b
echo "UUID=$(blkid -s UUID -o value $ESP2) /boot/efi2 vfat umask=0077 0 1" >> /etc/fstab
mount -a && findmnt /boot/efi2
```

Re-run the `grub-install` after any GRUB or kernel change — the second
ESP is **not** kept in sync automatically. **Verify by pulling one disk
and booting**; the checklist item exists because an untested mirror is a
guess, and this is the component most likely to be silently wrong.

Do not fix the letter ordering by moving cables: the only arrangement
that makes the SSDs `sda`/`sdb` puts them on SATA2 and the HDD on SATA3,
which caps the only devices that can use 6 Gb/s and gives the bandwidth
to a disk that tops out near 180 MB/s.

## 4. Base system

```sh
apt update && apt full-upgrade -y
apt install -y \
  linux-headers-amd64 zfs-dkms zfsutils-linux zfs-zed \
  mdadm smartmontools nfs-kernel-server \
  msmtp msmtp-mta bsd-mailx \
  curl vim htop ripgrep sysstat dmidecode pciutils usbutils \
  stress-ng fio
```

Static network — `/etc/network/interfaces` (ifupdown, matching nuc):

```
source /etc/network/interfaces.d/*

auto lo
iface lo inet loopback

allow-hotplug enp0s20f0
iface enp0s20f0 inet static
    address 192.168.0.4
    netmask 255.255.255.0
    gateway 192.168.0.1
    dns-nameservers 1.1.1.1 9.9.9.9
```

(Gateway is **`192.168.0.1`** — the FTTH box, since 2026-09. The host
uses public resolvers, never blocky, to avoid a bootstrap loop.
Interface name is a guess until the board is up — check `ip -br link`.)

Restore `/root/.ssh/authorized_keys` (incl. `id_rsa_claude.pub`) and
`timedatectl set-timezone Europe/Paris`.

## 5. Pools

### 5a. `incus` — SSD mirror

```sh
zpool create -o ashift=12 \
  -O compression=zstd -O atime=off -O xattr=sa -O acltype=posixacl \
  incus mirror \
  /dev/disk/by-id/ata-INTEL_SSDSC2CT120A3_CVMP250400ES120BGN-part5 \
  /dev/disk/by-id/ata-TOSHIBA_Q300._36OB318OK1KU-part5
zpool set autotrim=on incus
```

Weekly scrubs come from the packaged systemd timers rather than cron:

```sh
systemctl enable --now zfs-scrub-weekly@tank.timer zfs-scrub-weekly@incus.timer
systemctl list-timers 'zfs-scrub*'
```

`autotrim` matters on ten-year-old NAND — it is what keeps the
unallocated 22 % actually available to the controller as spare.

### 5b. `tank` — move the 4 TB off USB onto SATA

The risky step. The pool holds `usb4t/backup/nuc` (nuc's replicas),
`usb4t/backup/ks4` (empty, awaiting the FTTH seed) and `usb4t/media`.

⚠️ **Between export and import, nuc has no replica target and Jellyfin
has no media.** Plan a maintenance window and disable nuc's 03:30 cron
first, so it fails loudly rather than half-running.

```sh
# --- on nuc, first ---
zpool scrub usb4t                            # start clean; wait for it
zpool status usb4t
zpool export usb4t
```

Move the disk to SATA2-0 (reconnect it now if you unplugged it for the
install), then:

```sh
# --- on nas ---
zpool import                                 # confirm it is seen
zpool import usb4t tank                      # rename: it is not USB any more
zpool set cachefile=/etc/zfs/zpool.cache tank
zpool status tank
```

Properties survive from creation (`ashift=12`, `compression=zstd`,
`atime=off`, `xattr=sa`, `acltype=posixacl`). Re-point the mountpoints
and add the two new backup datasets:

```sh
zfs set mountpoint=/export/media tank/media   # NFSv4 export root (§8)
zfs set mountpoint=none          tank/backup
zfs list -o name,used,avail,mountpoint
```

Resulting layout:

```
incus                       mirror, 2× SSD — nas's own container roots
tank                        4 TB, single vdev
├── tank/media          → /export/media   NFS ro → nuc; local device → transmission-bt
└── tank/backup
    ├── tank/backup/nuc  → incus pool `nucbackup`   (nuc pushes here)
    └── tank/backup/ks4  → incus pool `ks4backup`   (nas pulls from ks4)

nas's own replicas live on **nuc** (`data/backup/nas` → pool `nasbackup`),
not here — see §9.
```

Backup pools are named after the **source** host, matching
[ks4/incus-copy.md](../ks4/incus-copy.md). nas's *own* instances are not
backed up here — they cross-replicate to nuc (§9b), so neither host's
instances depend on that host surviving.

Once `tank` has run a week on SATA with **zero** pool suspensions and no
CRC errors, the `usb4t-dropouts` gate is cleared — record that in
[ks2/plan.md](../ks2/plan.md). Day 1 was clean (2026-08-31).

### Clearing the inherited `<metadata>` errors — order matters

The pool imported carrying `<metadata>:<0x0>` and `<metadata>:<0x3d>` from
the 2026-08-29 USB dropout. A scrub found **0 errors and repaired 0B**,
yet the entries stayed, and a plain `zpool clear` afterwards did not drop
them either. ZFS flushes its persistent error log on a scrub that runs
**after** the clear — so the working order is:

```sh
zpool clear tank
zpool scrub tank        # this is the run that flushes the log
```

Result 2026-08-31: `scrub repaired 0B in 02:34:24 with 0 errors`,
`errors: No known data errors`, `all pools are healthy`. They were
artefacts of interrupted writes, not corruption — matching the
[2026-08-28 incident](../nuc/usb4t-dropouts.md).

⚠️ This matters for monitoring, not just tidiness: while those entries
stand, `zpool status -x` reports the pool unhealthy permanently, so
`zpool-health.sh` sits in the alarm state and **cannot signal a new
problem**. Clear them before trusting the watchdog.

## 6. Incus

Same Zabbly stable repo as nuc and ks4:

```sh
mkdir -p /etc/apt/keyrings
curl -fsSL https://pkgs.zabbly.com/key.asc -o /etc/apt/keyrings/zabbly.asc
cat > /etc/apt/sources.list.d/zabbly-incus-stable.sources <<EOF
Enabled: yes
Types: deb
URIs: https://pkgs.zabbly.com/incus/stable
Suites: trixie
Components: main
Architectures: amd64
Signed-By: /etc/apt/keyrings/zabbly.asc
EOF
apt update && apt install -y incus
```

macvlan like nuc, so instances get real LAN addresses — `transmission-bt`
keeps `192.168.0.7` when it moves:

```sh
cat <<EOF | incus admin init --preseed
config:
  core.https_address: :8443
storage_pools:
- name: incus
  driver: zfs
  config:
    source: incus
networks:
- name: macvlan
  type: macvlan
  config:
    parent: enp0s20f0
profiles:
- name: default
  devices:
    eth0: {name: eth0, network: macvlan, type: nic}
    root: {path: /, pool: incus, type: disk}
EOF

incus profile set default snapshots.schedule="0 3 * * *" snapshots.expiry=7d
```

Same macvlan quirk as nuc: **the host cannot talk to its own instances**,
and vice versa. Test container services from another LAN host or from
inside the container, never from `nas`.

Backup pools and the replica project:

```sh
incus storage create nucbackup zfs source=tank/backup/nuc
incus storage create ks4backup zfs source=tank/backup/ks4
incus project create backup -c features.images=false -c features.profiles=false
```

⚠️ **`tank/backup/nuc` already contains nuc's replicas** — they came
across with the pool. Re-register them so refreshes stay
ZFS-incremental instead of re-sending everything (the homeassistant VM
alone is a 50 GiB volume):

```sh
incus admin recover        # point it at pool nucbackup; project backup
incus list --project backup
```

Same call nuc-install.md §5 uses after a rebuild. If `recover` is
skipped, the first push in §9a silently becomes a full re-send of every
nuc instance.

Order incus after the ZFS mounts, as on nuc —
`/etc/systemd/system/incus.service.d/after-zfs.conf`:

```ini
[Unit]
After=zfs-mount.service zfs.target
```

## 7. Move `transmission-bt` from nuc

Its WireGuard tunnel is **entirely inside the container** (`wg0`,
`10.8.0.21`, `wg-quick@wg0`, `BindsTo=` on the daemon), so the container
carries its own keys and **ks4 needs no change at all** — the peer stays
`10.8.0.21/32`. The kill-switch `/32` route points at the gateway
`192.168.0.1`, which is the same from here.

```sh
# on nuc — remote already added in §9a
incus stop transmission-bt
incus move transmission-bt nas: --storage incus
```

Then on nas, re-point the media device at the local dataset — this is a
plain `shift=true` device again, because the data is local ZFS:

```sh
incus config device remove transmission-bt media
incus config device add transmission-bt media disk \
    source=/export/media path=/media shift=true
incus config set transmission-bt boot.autostart=true
incus start transmission-bt
```

Verify the tunnel and the kill switch before trusting it:

```sh
incus exec transmission-bt -- wg show
incus exec transmission-bt -- curl -s ifconfig.me    # must print 193.70.35.17
incus exec transmission-bt -- ip route               # must have NO default route
```

⚠️ **The watch-folder workflow moves with it** —
[transmission-bt.md](transmission-bt.md) says
`scp some.torrent root@192.168.0.3:/srv/media/.watchdir/`; it is now
`root@192.168.0.4:/export/media/.watchdir/`.

## 8. Media over NFS — `jellyfin-server` stays on nuc

`jellyfin-server` needs the Alder Lake-N iGPU for QSV/VAAPI; the C2750
has no render device at all, and software transcoding on Silvermont
manages 1–2 concurrent 1080p H.264 streams at best. So the container
stays on nuc and reaches the library over NFS, **read-only** —
`transmission-bt` is the only writer and it now lives here.

```sh
# --- nas: NFSv4 export, read-only, nuc only ---
cat >> /etc/exports <<'EOF'
/export           192.168.0.3(ro,fsid=0,crossmnt,no_subtree_check)
/export/media     192.168.0.3(ro,no_subtree_check,all_squash,anonuid=65534,anongid=65534)
EOF
exportfs -ra && exportfs -v
```

```sh
# --- nuc: mount at the SAME path, so jellyfin-server.md still applies ---
mkdir -p /srv/media
echo '192.168.0.4:/media /srv/media nfs4 ro,_netdev,soft,timeo=100,retrans=3 0 0' >> /etc/fstab
mount /srv/media && ls /srv/media
```

The container device changes only in losing the shift — per
[nuc/jellyfin-server.md](../nuc/jellyfin-server.md)'s own troubleshooting
note, *"on CIFS files are world-readable synthetic ownership, enough for
a read-only library"*; the same holds for NFS with `all_squash`:

```sh
incus stop jellyfin-server                    # shift cannot be hot-applied
incus config device set jellyfin-server media shift=false
incus config device set jellyfin-server media readonly=true
incus start jellyfin-server
incus exec jellyfin-server -- ls /media       # must list the library
```

Requirements for that to work: media files must be **world-readable**
(`find /export/media -type f ! -perm -o=r`), and transmission must keep
creating them that way (it does — `umask`/`0775` per its doc).

⚠️ Boot ordering on nuc: incus is already ordered after
`zfs-mount.service`; add `remote-fs.target` to that drop-in, or
`jellyfin-server` starts against an empty mountpoint and shows an empty
library.

⚠️ `soft` mount is deliberate: a hung NAS should fail Jellyfin's reads,
not wedge nuc's processes in uninterruptible sleep the way the suspended
`usb4t` pool did.

## 9. Backup legs

The driver is unchanged —
[`incus-copy.sh`](https://git.lutran.fr/julien/scripts/src/branch/main/incus-copy.sh),
deployed to `/root/scripts` as everywhere else.

### 9a. nuc's instances -> nas

Root crontab on nuc:

```cron
30 3 * * * /root/scripts/incus-copy.sh -d nas -m push -s nucbackup -p backup >> /var/log/incus-copy.log 2>&1
```

⚠️ **nuc is an on-demand media box** (see
[nuc/README.md](../nuc/README.md)): since 2026-08-30 it only runs when
watching Jellyfin or using the Spotify kiosk, so it is often powered
off at 03:30 and **that night's push is simply skipped** — cron does
not catch up missed windows. Accepted deliberately (2026-08-31):
nuc's instances change rarely and the next time it is up the refresh
is incremental anyway.
This ran briefly as a `systemd` timer with `Persistent=true` (which
*does* catch up after boot); the units are still on disk, disabled, at
`/etc/systemd/system/incus-copy.{service,timer}` if that behaviour is
ever wanted back:
`systemctl enable --now incus-copy.timer` (and remove the cron line).

⚠️ **`-p backup` is not optional.** Without it the replicas land in
`default` on nas and collide with nas's *live* instances — both hosts are
on the same macvlan LAN and the replicas carry the same static IPs.

### 9b. nas's own instances -> local pool `nasbackup`

**Decision 2026-08-30: local, not cross-replicated.** nuc is powered off
most of the time, so it is not a usable backup target — a nightly push to
it would fail noisily and, once mail works, alarm every morning. The
replicas instead go to `tank/backup/nas`, which is a **different pool**
from the instances themselves (`incus`, the SSD mirror), so it survives
losing that mirror. All three instances here are rebuildable from their
docs, so same-host is proportionate.

```sh
zfs create tank/backup/nas
zfs set mountpoint=legacy tank/backup/nas    # see the trap below
incus storage create nasbackup zfs source=tank/backup/nas
```

```cron
# nas, root crontab
0 4 * * * root /root/scripts/incus-copy.sh -p backup -s nasbackup >> /var/log/incus-copy.log 2>&1
```

⚠️ **Trap: `incus storage create` hangs forever on a `mountpoint=none`
dataset.** `tank/backup` is set to `mountpoint=none`, so any child
created afterwards inherits it, and the pool create then blocks with no
error and no entry in `incus operation list` — it looks exactly like I/O
contention (a scrub was running, which sent me down that path for 20
minutes). Set the child to `legacy` to match its siblings first, and it
completes instantly.

### 9c. Cleanup on nuc after the pool move


Exporting `usb4t` leaves nuc with an incus storage pool whose backing
dataset is gone, plus replica records in the `backup` project pointing
at it. Remove them once §6's `incus admin recover` has re-registered the
same volumes on nas — **verify there first, then delete here**:

```sh
# on nas: confirm the replicas are registered
incus list --project backup -c ns -f csv

# on nuc: only then
incus delete --project backup --force <each-replica>
incus storage delete nucbackup
zpool status              # only `data` should remain
```

### 9d. ks4 pull leg moves from nuc to nas

The leg [ks2/nas-seed.md](../ks2/nas-seed.md) prepared. The target pool
moves, so the **host** WireGuard tunnel moves too — nuc drops `wg-ks4`
entirely once this works (its only other tunnel user, `transmission-bt`,
is now here and carries its own).

- **ks4**: add nas as a peer on the `wireguard` container —
  `wg set wg0 peer <nas-pubkey> allowed-ips 10.8.0.22/32 && wg-quick save wg0`.
  The existing ufw rule (`allow in on incusbr0 from 192.168.1.18 to any
  port 8443 proto tcp`) already covers it: traffic arrives masqueraded as
  the WG container whichever peer sent it.
- **nas**: `/etc/wireguard/wg-ks4.conf` modelled on nuc's —
  `Address = 10.8.0.22/32`, peer pubkey `TVs6d7…`,
  `Endpoint = 193.70.35.17:51845`,
  `AllowedIPs = 10.8.0.0/24, 192.168.1.1/32`, keepalive 25 — then
  `systemctl enable --now wg-quick@wg-ks4`.
- `incus remote add ks4 https://192.168.1.1:8443 --accept-certificate --token '…'`
  (cross-check the fingerprint against the token).
- **nuc, after verification**: `systemctl disable --now wg-quick@wg-ks4`,
  remove `/etc/wireguard/wg-ks4.conf`, and drop the `10.8.0.20/32` peer
  on ks4.

Seed manually in tmux — the first pass is **~1.75 TiB** over the home
downlink and only makes sense once FTTH is up:

```sh
/root/scripts/incus-copy.sh -r ks4 -s ks4backup 2>&1 | tee -a /var/log/incus-copy-ks4.log
```

Then test-restore one instance before ticking the gate in
[ks2/plan.md](../ks2/plan.md).

### Resulting schedule

| When | Host | What |
|---|---|---|
| 03:00 | nuc, nas | instance snapshots (profile) |
| 03:30 | nuc | push all instances → `nas:nucbackup` (root crontab; skipped when nuc is off) |
| 04:00 | nas | local copy of nas instances → `nasbackup` (`tank/backup/nas`) |
| 05:00 | nas | pull `ks4:*` → `ks4backup` (after FTTH) |
| 05:30 | nuc | apt upgrade all containers |
| 06:00 | nas | apt upgrade all containers (`incus-container-upgrade.sh`, added 2026-09-01; also refreshes `user.os`) |
| Mon ~00:12 | nas | `zfs-scrub-weekly@tank.timer` / `@incus.timer` (systemd, not cron) |

Staggered around ks4's own 01:00 / 05:00 jobs.

⚠️ Keep `snapshots.schedule` on every source. A refresh with no common
snapshot degrades to a **full re-send** — the failure mode that cost
933 G on ks4 ([ks4/local-backup-cron.md](../ks4/local-backup-cron.md)).
The homeassistant **VM** on nuc re-sends its whole volume without them.
Note `incus-copy.sh` takes a **global** `flock`: an overrunning ks4 pull
aborts that night's other run loudly rather than racing it.

## 10. Monitoring

Do not repeat nuc's 11-day blind spot
([usb4t-dropouts.md](../nuc/usb4t-dropouts.md) — zed was running, but the
host had no MTA).

**Status: live since 2026-08-30**, verified end to end (`smtpstatus=250`).

- **msmtp**, same shape as nuc: this box shares the dynamic home IP with
  no PTR and no SPF alignment, so it must use **submission (587) with
  auth**, not port 25 — rspamd rejects the port-25 path as spam.
  `/etc/msmtprc` mode 600, host `mail.lutran.fr`, STARTTLS. Give SMTP
  tests ≥ 30 s: the missing PTR delays the greeting.
  The `zed@lutran.fr` account is the **same credential as nuc** — it is
  SMTP-AUTH, not IP-bound, so `/etc/msmtprc` can simply be copied between
  hosts (mode 600, root:root). The password lives in the password
  manager; no host-specific setup is needed.
- **zed**: `/etc/zfs/zed.d/zed.rc` mode 600 with
  `ZED_EMAIL_ADDR="julien@lutran.fr"`, `ZED_EMAIL_PROG="mail"`,
  `ZED_NOTIFY_VERBOSE=1`, **`ZED_NOTIFY_DATA=1`**,
  `ZED_NOTIFY_INTERVAL_SECS=3600`.
- **[`zpool-health.sh`](https://git.lutran.fr/julien/scripts/src/branch/main/zpool-health.sh)**
  every 15 min from root's crontab — zed does *not* report a
  suspended pool (the vdev stays `ONLINE`, so `statechange-notify.sh`
  never fires). That watchdog is the only thing that catches the exact
  failure this box was built to prevent.
- **smartd** — `/etc/smartd.conf`. The Toshiba carries the useful
  attributes; the Intel is a liveness check only:

```
# Toshiba Q300 — endurance + temperature are real here
/dev/disk/by-id/ata-TOSHIBA_Q300._36OB318OK1KU -a -o on -S on -s (S/../.././02|L/../../6/03) -W 4,50,55 -m julien@lutran.fr
# Intel 330 — no error log, no devstat, no temperature: liveness only
/dev/disk/by-id/ata-INTEL_SSDSC2CT120A3_CVMP250400ES120BGN -H -s (S/../.././03|L/../../6/04) -m julien@lutran.fr
# WD Red
/dev/disk/by-id/ata-WDC_WD40EFRX-68WT0N0_WD-WCC4E6NLPJJE -a -o on -S on -s (S/../.././04|L/../../6/05) -W 4,45,50 -m julien@lutran.fr
```

Verify the whole chain the day you build it, not the day you need it:

```sh
/root/scripts/zpool-health.sh -m julien@lutran.fr -t
tail -2 /var/log/msmtp.log        # expect smtpstatus=250
smartctl -d sat -l devstat /dev/sdX | grep -i endurance
```

## 11. Power baseline

Take the measurement **before** the box goes into service, so later
readings mean something. Expect roughly 25–35 W idle with three disks:
the 20 W SoC plus a BMC drawing several watts continuously, even at
soft-off.

```sh
# baseline: 10 min idle, everything settled
# CPU in steps — the interesting curve for an always-on box
for l in 25 50 75 100; do echo "=== ${l}% $(date +%s)"; stress-ng --cpu 0 --cpu-load $l --timeout 120s; done
# disk: random I/O is what moves an HDD's power, not throughput
fio --name=rr --directory=/export/media --size=20G --rw=randread --bs=4k \
    --iodepth=32 --numjobs=4 --ioengine=libaio --direct=1 --runtime=300 --time_based
```

Measure at the wall (PDU or smart plug) — RAPL is unreliable on Avoton
and sees neither disks nor fans. Log epoch timestamps per step so the
trace can be recut against the meter's series afterwards.

## 12. Post-install checklist

- [x] IPMI **left unplugged by decision** (2026-08-30) — no BMC on the
      LAN, therefore no out-of-band console either
- [x] `cat /proc/mdstat` — all three arrays `[UU]`; GRUB written to
      **both** ESPs, both mounted, `debian` + `debian-b` boot entries
      present (verified across a cold boot 2026-08-31)
- [ ] boot still untested with **one disk physically unplugged** — the
      mirror is a guess until that is done
- [x] `zpool status` healthy for `incus` and `tank`; weekly scrubs
      scheduled; **cold boot verified 2026-08-31** — both pools imported
      from `/etc/zfs/zpool.cache`, all instances autostarted, NFS exports
      republished, 0 failed units
- [ ] **7 days with zero pool suspensions and zero CRC errors** — the
      gate that closes [usb4t-dropouts.md](../nuc/usb4t-dropouts.md)
- [x] `smartd` monitoring all 3 disks; `zpool-health.sh -t` mail
      delivered (`smtpstatus=250` in `/var/log/msmtp.log`) — done
      2026-08-30
- [ ] `transmission-bt` on nas: egress is `193.70.35.17`, **no default
      route**, downloads land in `/export/media/downloads`, watch folder
      works from the new path
- [ ] `jellyfin-server` on nuc lists the library over NFS after a **cold
      reboot of both hosts** (the boot-ordering trap)
- [ ] `incus admin recover` ran on nas **before** the first push, so the
      inherited `tank/backup/nuc` replicas refresh incrementally instead
      of re-sending (check the first run's duration, not just `rc=0`)
- [ ] nuc's 03:30 leg → `nas:nucbackup`, `rc=0`, all nuc instances
      present (`incus list --project backup -c ns -f csv` on nas)
- [ ] nas's 04:00 local leg → `nasbackup`, `rc=0`, and **blocky,
      privoxy and transmission-bt all appear by name** in
      `incus list --project backup`
- [ ] stale `nucbackup` pool removed from nuc (§9c), `zpool status`
      shows only `data`
- [ ] replicas are **stopped** with `boot.autostart=false` — they hold
      the live containers' LAN addresses
- [ ] ks4 pull leg seeded + one instance test-restored → tick the gate
      in [ks2/plan.md](../ks2/plan.md); then retire nuc's `wg-ks4`
- [ ] power baseline recorded above, with the meter reading
- [ ] this file updated with what was built (RAM fitted, NIC name, WD Red
      serial)