nas: new storage host on A1SAi-2750F — 4 TB off USB onto SATA
Builds the box that ends the usb4t dropouts: the JMicron bridge was the least reliable device in the setup and it held the intended off-site copy of ks4. The 4 TB now sits on direct SATA as pool `tank`. - OS on mdraid RAID1 across two 120 GB SSDs (Intel 330 + Toshiba Q300), both ESPs bootable; `incus` ZFS mirror on their tails, ~22% left unallocated as over-provisioning - media at /export/media, exported read-only over NFSv4 to nuc - transmission-bt moves here (its WireGuard tunnel is in-container, so ks4 needed no change) and writes to the dataset locally - backup pools nucbackup / ks4backup / nasbackup - monitoring live: msmtp (submission+auth, verified 250), zed with NOTIFY_DATA, zpool-health.sh every 15 min, smartd on all three disks Traps recorded because none of them point at their own cause: booting with the display active kills the i915 probe and wedges incus; d-i picks grub-pc vs grub-efi from how the installer booted; `incus storage create` hangs forever on a mountpoint=none dataset; the BMC is deliberately never cabled, so there is no out-of-band console. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Julien Lutran
co-authored by
Claude Opus 5
Claude Opus 5
parent
a2003a11e8
commit
58dcaa5d41
@@ -12,7 +12,9 @@ backed up, by which tool, on what schedule, and how to restore.
|
||||
service to ks4 (and getting it backed up automatically).
|
||||
|
||||
**Tech notes**
|
||||
- [`nuc/`](nuc/README.md) — home lab on `nuc`: host, storage, instances
|
||||
- [`nuc/`](nuc/README.md) — home lab on `nuc`: host, iGPU instances
|
||||
- [`nas/`](nas/README.md) — storage + backup host `nas` (Supermicro
|
||||
A1SAi-2750F): the 4 TB on direct SATA, media over NFS, backup pools
|
||||
- [`ks4/`](ks4/README.md) — prod server `ks4` at OVH: host, services, network flows
|
||||
- [`ks2/`](ks2/) — legacy backup server being [decommissioned](ks2/plan.md)
|
||||
- [`archer-c7/`](archer-c7/) — home router (TP-Link Archer C7 v5 running on OpenWrt)
|
||||
|
||||
@@ -0,0 +1,50 @@
|
||||
# Storage: nas
|
||||
|
||||
Storage + backup host on the LAN, added 2026-08.
|
||||
|
||||
- **Always-on host.** nuc is now an on-demand media box (see
|
||||
[nuc/README.md](../nuc/README.md)), so everything that must stay up —
|
||||
LAN DNS, the HTTP proxy, torrents — lives here.
|
||||
- Debian 13, Supermicro **A1SAi-2750F** / Intel Atom C2750 (8 c, 20 W,
|
||||
ECC DDR3) — build procedure: [nas-install.md](nas-install.md)
|
||||
- SSH: `ssh -i id_rsa_claude root@192.168.0.4`
|
||||
- **No iGPU** (Avoton is headless; video is the AST2400 BMC). Anything
|
||||
needing hardware transcoding stays on nuc.
|
||||
- Pools:
|
||||
- `incus` — ZFS mirror across the last partition of both 120 GB SSDs;
|
||||
holds this host's container roots. OS itself is on mdraid RAID1 +
|
||||
ext4 across the same disks (rationale in
|
||||
[nas-install.md](nas-install.md) §2).
|
||||
- `tank` — the 4 TB WD Red, **on direct SATA**. This is the whole
|
||||
point of the box: the disk used to hang off a JMicron USB bridge on
|
||||
nuc that suspended the pool 61 times in 30 days
|
||||
([nuc/usb4t-dropouts.md](../nuc/usb4t-dropouts.md)). Single vdev,
|
||||
accepted — nothing on it is irreplaceable.
|
||||
- `tank/media` → `/export/media`: the media library. Exported
|
||||
**read-only over NFSv4 to nuc**, where `jellyfin-server` reads it;
|
||||
written locally only by `transmission-bt`.
|
||||
- Backups: nuc ↔ nas **cross-replication** (each host's instances live
|
||||
on the other), plus the ks4 pull leg —
|
||||
[nas-install.md](nas-install.md) §9,
|
||||
[ks2/nas-seed.md](../ks2/nas-seed.md).
|
||||
|
||||
## Instances
|
||||
|
||||
| Name | IP | Doc | Features |
|
||||
|---|---|---|---|
|
||||
| blocky | 192.168.0.254 | — | unprivileged, autostart; DNS ad-blocker for the LAN. Moved from nuc 2026-08-30 so it survives nuc being powered off |
|
||||
| privoxy | 192.168.0.11 | — | unprivileged, autostart; filtering HTTP proxy, listens on **:3128** (not privoxy's default 8118); static config in `/etc/systemd/network/eth0.network`, `DNS=192.168.0.254`. Moved from nuc 2026-08-30 |
|
||||
| [transmission-bt](transmission-bt.md) | 192.168.0.7 | ✅ | unprivileged, autostart; always-on WireGuard full tunnel → ks4 (egress = 193.70.35.17, kill switch: no default route); `/export/media` disk device (`shift=true`), downloads to `/media/downloads`; web UI :9091 (LAN only). Moved from nuc 2026-08-30 |
|
||||
|
||||
## Backup pools hosted here
|
||||
|
||||
| incus pool | dataset | receives |
|
||||
|---|---|---|
|
||||
| `nucbackup` | `tank/backup/nuc` | nuc's instances (pushed nightly, 03:30) |
|
||||
| `ks4backup` | `tank/backup/ks4` | ks4's instances (pulled over WG, 05:00 — after FTTH) |
|
||||
| `nasbackup` | `tank/backup/nas` | **nas's own** instances (local copy, 04:00) |
|
||||
|
||||
nas's own instances are replicated **locally** rather than to nuc: nuc is
|
||||
an on-demand box and usually powered off, so it is not a usable backup
|
||||
target. `nasbackup` lives on `tank`, a different pool from the `incus`
|
||||
SSD mirror the instances run on.
|
||||
@@ -0,0 +1,838 @@
|
||||
# nas — build procedure
|
||||
|
||||
New storage host on the LAN (`192.168.0.4`), built 2026-08 from a
|
||||
Supermicro A1SAi-2750F. It exists to solve one specific problem:
|
||||
[usb4t-dropouts.md](../nuc/usb4t-dropouts.md) ruled out the cable and
|
||||
USB power management and left the **JMicron 152d:0578 bridge** as the
|
||||
cause of 61 disconnects in 30 days. The durable fix named there is a
|
||||
direct SATA connection — this box provides six of them.
|
||||
|
||||
Why it matters beyond the media library: `usb4t` is the intended home
|
||||
of the ks4 off-site replicas, and *"the pool holding the off-site copy
|
||||
of ks4 must not be the least reliable device in the setup"*. That makes
|
||||
this build a **release gate for the ks2 decommission**
|
||||
([ks2/plan.md](../ks2/plan.md), deadline Sep 30, 2026).
|
||||
|
||||
It also takes over `transmission-bt` from nuc, so nuc keeps only what
|
||||
needs its iGPU.
|
||||
|
||||
## Hardware
|
||||
|
||||
- Supermicro **A1SAi-2750F** mini-ITX, Intel Atom **C2750** (8 cores,
|
||||
2.4 GHz Silvermont, 20 W SoC), AES-NI, no AVX
|
||||
- RAM: **2× 4 GB DDR3-1600 ECC SO-DIMM fitted = 8 GB** (`Single-bit ECC`
|
||||
confirmed), in DIMMA1/DIMMB1; **2 slots free**. Board takes 32 GB
|
||||
officially, 64 GB with 16 GB modules.
|
||||
⚠️ 8 GB is modest for a 3.6 TB pool — ARC lands around 4 GB. Fine for
|
||||
streaming and replication (neither benefits much from cache), but the
|
||||
first thing to raise if metadata-heavy operations feel slow.
|
||||
- SATA: **2× SATA3 + 4× SATA2** (6 total)
|
||||
- BIOS **2.2** (2019-11-22) as shipped by the RMA
|
||||
- NIC: 4× GbE (Intel i354) + dedicated IPMI LAN
|
||||
- Video: **ASPEED AST2400 BMC only — there is no iGPU.** Avoton is a
|
||||
headless server SoC; `/dev/dri` is empty. That is why `jellyfin-server`
|
||||
stays on nuc (§8).
|
||||
|
||||
### ⚠️ AVR54 — already handled
|
||||
|
||||
The C2750 is on the list of Atom C2000 parts affected by Intel's
|
||||
**AVR54** erratum: the SoC's `LPC_CLKOUT0/1` signals degrade and stop,
|
||||
after which the board never boots again — typically after ~18 months of
|
||||
power-on, i.e. exactly an always-on duty cycle. **This board was RMA'd
|
||||
by Supermicro for that issue and replaced**, so it carries the fix
|
||||
(C0 stepping or the LPC pull-up rework). Recorded here so a dead C2000
|
||||
board is not re-diagnosed from scratch later.
|
||||
|
||||
### Disk plan
|
||||
|
||||
| Port | Device | Role |
|
||||
|---|---|---|
|
||||
| SATA3-0 | Intel `SSDSC2CT120A3` (120 GB) | md mirror + `incus` pool |
|
||||
| SATA3-1 | Toshiba `Q300.` (120 GB) | md mirror + `incus` pool |
|
||||
| SATA2-0 | WD Red 4 TB (moved off the USB enclosure) | pool `tank` |
|
||||
| SATA2-1/2/3 | free | a second 4 TB to mirror `tank`, later |
|
||||
|
||||
The SSDs get the SATA3 ports because they are the only devices that can
|
||||
use them: both negotiate 6 Gb/s and do ~450–500 MB/s, while the WD Red
|
||||
tops out near 180 MB/s and cannot saturate SATA2's ~270 MB/s. A future
|
||||
SLOG would also be fine on SATA2 — it is latency-bound on small sync
|
||||
writes, not bandwidth-bound.
|
||||
|
||||
⚠️ **Disconnect the 4 TB before partitioning the SSDs.** It carries the
|
||||
media library and nuc's replicas, nothing in the OS install needs it,
|
||||
and it keeps the two 120 GB disks unambiguous in the installer's list.
|
||||
Reconnect it before §5b.
|
||||
|
||||
Device letters shift with enumeration order and mean nothing — mdraid
|
||||
assembles from superblock UUIDs, ZFS imports by GUID, fstab and GRUB use
|
||||
UUIDs. **Identify disks by model and serial**
|
||||
(`lsblk -o NAME,SIZE,MODEL,SERIAL`), never by letter: both SSDs are
|
||||
120 GB, so size alone does not tell them apart.
|
||||
|
||||
SSD health measured 2026-08-30 (both read over the JMicron bridge on
|
||||
nuc, `smartctl -d sat`), before deployment:
|
||||
|
||||
| | Intel 330 (25 nm MLC) | Toshiba Q300 (15 nm TLC) |
|
||||
|---|---|---|
|
||||
| host writes | 16.0 TiB | 7.5 TiB |
|
||||
| endurance consumed | **0 %** (`Media_Wearout_Indicator` 100) | **7 %** (`Percentage Used Endurance Indicator`) |
|
||||
| power-on hours | **unreadable** — attr 9 decodes to 914,563 h on this family | 3,877 |
|
||||
| power cycles | 98 | 432 |
|
||||
| defects | 0 reallocated / program-fail / erase-fail | 0 reported uncorrectable |
|
||||
| SMART error log | **not supported** | supported, clean |
|
||||
| device statistics log | absent | full ACS-2 set |
|
||||
| short self-test | passed | passed |
|
||||
| interface CRC errors | n/a | 16 (baseline — watch for growth) |
|
||||
|
||||
Both are healthy and far from wear-out; at OS-disk write rates endurance
|
||||
is not the binding constraint for either. They were previously a
|
||||
**matched pair** — identical layouts, `bpool` / `rpool` / `ubuntu:0`
|
||||
labels from an Ubuntu ZFS-on-root mirror — which is why they go back
|
||||
into a mirror here.
|
||||
|
||||
⚠️ **The Intel is effectively unmonitorable**: no error log, no device
|
||||
statistics, no temperature, no usable hours counter. `smartd` can watch
|
||||
the Toshiba properly and can only ask the Intel "are you still there".
|
||||
Expect the Intel to be found dead rather than found degrading. That
|
||||
asymmetry is the reason for the mirror.
|
||||
|
||||
### `tank` is a single vdev — accepted
|
||||
|
||||
The 4 TB holds the media library, transmission's downloads and the ks4
|
||||
replicas, with no redundancy. A single vdev can *detect* corruption but
|
||||
only self-heal metadata, not data — as seen in the usb4t incident.
|
||||
|
||||
**Decision 2026-08-30: accepted, nothing on `tank` is irreplaceable.**
|
||||
Media and instances are re-fetchable from their sources; the ks4
|
||||
replicas are leg 3 of ks4's 3-2-1 (local `sdb5` + restic/S3 remain —
|
||||
[backup-strategy.md](../backup-strategy.md)). Losing `tank` costs time,
|
||||
not data.
|
||||
|
||||
Three SATA ports stay free, so `zpool attach tank <existing> <new>`
|
||||
turns it into a mirror whenever a spare 4 TB turns up. Not a
|
||||
prerequisite for anything. Scrub weekly regardless — on a single vdev
|
||||
the scrub is the only thing that *tells* you a file has rotted.
|
||||
|
||||
## 1. IPMI and BIOS first
|
||||
|
||||
The AST2400 stack is old and has known vulnerabilities.
|
||||
|
||||
- **Decision 2026-08-30: the BMC is never cabled.** The dedicated IPMI
|
||||
port stays unplugged, so the AST2400's default credentials and its
|
||||
known vulnerabilities are not reachable from anything. This is the
|
||||
simplest correct answer for a box that sits on a flat home LAN — no
|
||||
management VLAN needed, nothing to harden, nothing to patch.
|
||||
- ⚠️ Consequence: **there is no out-of-band console.** A boot that fails
|
||||
before sshd needs a physical monitor and keyboard. Worth knowing before
|
||||
changing anything that affects booting (GRUB, the md arrays, fstab).
|
||||
- BIOS: enable **restore-on-AC-loss**, enable C-states, and disable the
|
||||
three unused i354 NICs (each costs about a watt).
|
||||
|
||||
### Switching this board to UEFI
|
||||
|
||||
There is **no "Boot Mode Select" entry** in this BIOS — that option
|
||||
exists on later Supermicro generations, not here. What works
|
||||
(verified 2026-08-30):
|
||||
|
||||
- **CSM → Disabled**
|
||||
- **All OpROM policies → UEFI** (storage, video *and* network)
|
||||
|
||||
Setting the **video** OpROM to UEFI is safe despite the console being
|
||||
the AST2400 BMC framebuffer — output survives, both on the BMC console
|
||||
and over IPMI KVM.
|
||||
|
||||
⚠️ **Confirm the mode before you partition anything**, from the
|
||||
installer (`Ctrl+Alt+F2`):
|
||||
|
||||
```sh
|
||||
ls /sys/firmware/efi # directory exists = UEFI. Missing = legacy
|
||||
```
|
||||
|
||||
This single check is what makes the difference between a working
|
||||
install and an afternoon lost — see the trap in §3.
|
||||
|
||||
## 2. Prepare the SSDs
|
||||
|
||||
Both report `ATA Security is: Disabled, NOT FROZEN`, so a real secure
|
||||
erase is available — do that rather than just repartitioning. It
|
||||
restores the full spare-block pool on ten-year-old NAND.
|
||||
|
||||
```sh
|
||||
# per disk, from a live system where the disk is NOT the running OS
|
||||
hdparm --user-master u --security-set-pass Eins /dev/sdX
|
||||
hdparm --user-master u --security-erase Eins /dev/sdX
|
||||
hdparm -I /dev/sdX | grep -A2 Security # expect "not enabled"
|
||||
```
|
||||
|
||||
Identical GPT layout on both, sized so ~27 GB (22 %) stays
|
||||
**unallocated** as over-provisioning:
|
||||
|
||||
| Part | Size | Type | Device | Use |
|
||||
|------|------|------|--------|-----|
|
||||
| `sdX1` | 1 GB | EFI System Partition | — | `/boot/efi` (two independent ESPs) |
|
||||
| `sdX2` | 2 GB | Linux RAID | `md0` | `/boot` ext4 |
|
||||
| `sdX3` | 24 GB | Linux RAID | `md1` | `/` ext4 |
|
||||
| `sdX4` | 2 GB | Linux RAID | `md2` | swap |
|
||||
| `sdX5` | 64 GB | Solaris root (bf00) | — | **zpool `incus`** (mirror) |
|
||||
| — | ~27 GB | **unallocated** | — | over-provisioning |
|
||||
|
||||
- ⚠️ **The ESP is deliberately not a RAID1 array.** ks4 has `md1 →
|
||||
/boot/efi` because OVH's installer builds it with mdadm metadata
|
||||
**1.0** (superblock at the *end*, so firmware still sees plain FAT).
|
||||
`debian-installer` only creates metadata **1.2** arrays, whose
|
||||
superblock sits at the start and makes the ESP unreadable to
|
||||
firmware. So: one plain ESP per disk, only one mounted, the second
|
||||
filled by hand (§3). Deviation from ks4 is intentional.
|
||||
- A legacy-BIOS variant of this layout was tried first — 1 MB
|
||||
`bios_grub` instead of the ESP, which makes the mirror simpler
|
||||
(`grub-install` to both disks, nothing to keep in sync). It was
|
||||
abandoned because this board's firmware has no way to prefer legacy
|
||||
targets once CSM is off, and it kept falling through to the UEFI
|
||||
shell. Recorded so it is not retried: **UEFI is the working path
|
||||
here.**
|
||||
- `/` at 24 GB matches the other hosts (nuc 46 GB, ks4 40 GB) — this box
|
||||
has no desktop and no container roots on `/`.
|
||||
- Mixing md partitions and a ZFS partition on the same disks is exactly
|
||||
what ks4 does (`md1/2/3` + ZFS on `sda5`/`sdb5`).
|
||||
|
||||
### Why not full root-on-ZFS
|
||||
|
||||
Decision 2026-08-30. These SSDs previously ran Ubuntu 20.04's
|
||||
experimental ZFS-root installer (hence the leftover `bpool` / `rpool`
|
||||
labels), so the option was on the table. Rejected because:
|
||||
|
||||
- **Debian's installer cannot do it.** ZFS is CDDL, shipped only in
|
||||
`contrib` as `zfs-dkms`; `debian-installer` can neither partition nor
|
||||
boot from ZFS. Root-on-ZFS means the manual
|
||||
[OpenZFS Debian Trixie HOWTO](https://openzfs.github.io/openzfs-docs/Getting%20Started/Debian/Debian%20Trixie%20Root%20on%20ZFS.html)
|
||||
— ~60 steps from a live ISO. Per the repo convention the doc *is* the
|
||||
rebuild procedure, and that is a bad thing to be executing during an
|
||||
actual failure.
|
||||
- **DKMS failure mode.** `zfs-dkms` rebuilds on every kernel upgrade.
|
||||
If that build fails, root-on-ZFS means the box **does not boot**;
|
||||
with an ext4 md root it boots normally and only the pools are
|
||||
missing — recoverable over IPMI with a shell.
|
||||
- GRUB's ZFS support lags OpenZFS, which is why every root-on-ZFS guide
|
||||
needs a separate feature-limited `bpool`; `zpool upgrade bpool` is a
|
||||
known way to make a machine unbootable.
|
||||
- Ubuntu's version of this is a dead end anyway: the installer option
|
||||
was nearly dropped in 22.04 and `zsys`, which made boot environments
|
||||
useful, is abandoned.
|
||||
|
||||
What root-on-ZFS would buy — snapshot and roll back a bad upgrade — is
|
||||
already covered where the state actually lives: container roots get
|
||||
incus snapshots plus nightly replication (§9). The host is 24 GB of
|
||||
packages reproducible from this file. Accepted cost: no pre-upgrade
|
||||
rollback of the host itself (`etckeeper` covers `/etc` if wanted).
|
||||
|
||||
## 3. Install Debian 13 (trixie)
|
||||
|
||||
Netinst ISO (burned 2026-08-30, sha256 `65273bee…664e7`, verified
|
||||
against `cdimage.debian.org/debian-cd/13.6.0/amd64/iso-cd/SHA256SUMS`).
|
||||
|
||||
- Manual partitioning per the table above: `sdX1` as ESP, `sdX2..4` as
|
||||
RAID1 members (three arrays), `sdX5` left untouched.
|
||||
- Tasks: **SSH server + standard utilities only**. No desktop — there is
|
||||
no GPU and the console is a BMC framebuffer.
|
||||
- Sources: `main contrib non-free-firmware` (`contrib` is required by
|
||||
`zfs-dkms`; the installer does not offer it — add it after first boot).
|
||||
|
||||
Confirm d-i picked the right bootloader once installed:
|
||||
`dpkg -l | grep grub-efi` — `grub-efi-amd64`, not `grub-pc`.
|
||||
|
||||
### ⚠️ Trap: installer boot mode decides the bootloader (hit 2026-08-30)
|
||||
|
||||
`debian-installer` chooses `grub-pc` or `grub-efi-amd64` from **how the
|
||||
installer itself booted**, not from what the disks look like. Booting the
|
||||
USB stick in legacy mode while the firmware prefers UEFI produces:
|
||||
|
||||
1. d-i installs `grub-pc`, targeting the MBR;
|
||||
2. on a **GPT** disk that needs a 1 MB `bios_grub` partition — absent
|
||||
here, so `grub-install` fails, easy to click past;
|
||||
3. the firmware then tries the SSDs as UEFI targets, finds no `.efi`
|
||||
binary, and drops to the **UEFI shell**.
|
||||
|
||||
Nothing is corrupt; the halves simply disagree. Symptoms and checks:
|
||||
|
||||
```sh
|
||||
[ -d /sys/firmware/efi ] && echo UEFI || echo legacy # in the installer
|
||||
lsblk -no PTTYPE,PARTTYPENAME /dev/sdX # gpt + "EFI System"?
|
||||
dd if=/dev/sdX bs=440 count=1 2>/dev/null | od -c | head -3 # all \0 = no boot code
|
||||
dpkg -l | grep -E '^ii.*grub-(pc|efi)' # which flavour got installed
|
||||
```
|
||||
|
||||
The board's hybrid ISO offers both paths, so the F11 boot menu usually
|
||||
lists the stick twice — picking the **`UEFI:`** entry avoids the whole
|
||||
thing. Checking `/sys/firmware/efi` before partitioning is the one step
|
||||
that prevents it.
|
||||
|
||||
### Second ESP — do this before trusting the mirror
|
||||
|
||||
The installer populates only the ESP it mounted. Until the second one is
|
||||
written, losing that disk means the box does not boot, mirror or no
|
||||
mirror.
|
||||
|
||||
⚠️ **Use `/dev/disk/by-id/`, never `/dev/sdX`.** Reconnecting the 4 TB
|
||||
after the install shifts every letter — observed 2026-08-30: the HDD on
|
||||
SATA2-0 takes `sda` even with the SSDs on SATA3, because the SATA2
|
||||
controller enumerates first on this SoC. A bare `/dev/sdb1` written
|
||||
during the install then points at a *different disk*, and here that
|
||||
would mean reformatting the ESP the system actually boots from.
|
||||
|
||||
```sh
|
||||
ESP2=/dev/disk/by-id/ata-TOSHIBA_Q300._36OB318OK1KU-part1 # the one NOT at /boot/efi
|
||||
mkfs.vfat -F32 "$ESP2"
|
||||
mkdir -p /boot/efi2 && mount "$ESP2" /boot/efi2
|
||||
grub-install --target=x86_64-efi --efi-directory=/boot/efi2 \
|
||||
--bootloader-id=debian-b --recheck
|
||||
efibootmgr -v # expect: debian, debian-b
|
||||
echo "UUID=$(blkid -s UUID -o value $ESP2) /boot/efi2 vfat umask=0077 0 1" >> /etc/fstab
|
||||
mount -a && findmnt /boot/efi2
|
||||
```
|
||||
|
||||
Re-run the `grub-install` after any GRUB or kernel change — the second
|
||||
ESP is **not** kept in sync automatically. **Verify by pulling one disk
|
||||
and booting**; the checklist item exists because an untested mirror is a
|
||||
guess, and this is the component most likely to be silently wrong.
|
||||
|
||||
Do not fix the letter ordering by moving cables: the only arrangement
|
||||
that makes the SSDs `sda`/`sdb` puts them on SATA2 and the HDD on SATA3,
|
||||
which caps the only devices that can use 6 Gb/s and gives the bandwidth
|
||||
to a disk that tops out near 180 MB/s.
|
||||
|
||||
## 4. Base system
|
||||
|
||||
```sh
|
||||
apt update && apt full-upgrade -y
|
||||
apt install -y \
|
||||
linux-headers-amd64 zfs-dkms zfsutils-linux zfs-zed \
|
||||
mdadm smartmontools nfs-kernel-server \
|
||||
msmtp msmtp-mta bsd-mailx \
|
||||
curl vim htop ripgrep sysstat dmidecode pciutils usbutils \
|
||||
stress-ng fio
|
||||
```
|
||||
|
||||
Static network — `/etc/network/interfaces` (ifupdown, matching nuc):
|
||||
|
||||
```
|
||||
source /etc/network/interfaces.d/*
|
||||
|
||||
auto lo
|
||||
iface lo inet loopback
|
||||
|
||||
allow-hotplug enp0s20f0
|
||||
iface enp0s20f0 inet static
|
||||
address 192.168.0.4
|
||||
netmask 255.255.255.0
|
||||
gateway 192.168.0.2
|
||||
dns-nameservers 1.1.1.1 9.9.9.9
|
||||
```
|
||||
|
||||
(Gateway is **`192.168.0.2`** — `.1` is gone since 2026-08. The host
|
||||
uses public resolvers, never blocky, to avoid a bootstrap loop.
|
||||
Interface name is a guess until the board is up — check `ip -br link`.)
|
||||
|
||||
Restore `/root/.ssh/authorized_keys` (incl. `id_rsa_claude.pub`) and
|
||||
`timedatectl set-timezone Europe/Paris`.
|
||||
|
||||
## 5. Pools
|
||||
|
||||
### 5a. `incus` — SSD mirror
|
||||
|
||||
```sh
|
||||
zpool create -o ashift=12 \
|
||||
-O compression=zstd -O atime=off -O xattr=sa -O acltype=posixacl \
|
||||
incus mirror \
|
||||
/dev/disk/by-id/ata-INTEL_SSDSC2CT120A3_CVMP250400ES120BGN-part5 \
|
||||
/dev/disk/by-id/ata-TOSHIBA_Q300._36OB318OK1KU-part5
|
||||
zpool set autotrim=on incus
|
||||
```
|
||||
|
||||
Weekly scrubs come from the packaged systemd timers rather than cron:
|
||||
|
||||
```sh
|
||||
systemctl enable --now zfs-scrub-weekly@tank.timer zfs-scrub-weekly@incus.timer
|
||||
systemctl list-timers 'zfs-scrub*'
|
||||
```
|
||||
|
||||
`autotrim` matters on ten-year-old NAND — it is what keeps the
|
||||
unallocated 22 % actually available to the controller as spare.
|
||||
|
||||
### 5b. `tank` — move the 4 TB off USB onto SATA
|
||||
|
||||
The risky step. The pool holds `usb4t/backup/nuc` (nuc's replicas),
|
||||
`usb4t/backup/ks4` (empty, awaiting the FTTH seed) and `usb4t/media`.
|
||||
|
||||
⚠️ **Between export and import, nuc has no replica target and Jellyfin
|
||||
has no media.** Plan a maintenance window and disable nuc's 03:30 cron
|
||||
first, so it fails loudly rather than half-running.
|
||||
|
||||
```sh
|
||||
# --- on nuc, first ---
|
||||
zpool scrub usb4t # start clean; wait for it
|
||||
zpool status usb4t
|
||||
zpool export usb4t
|
||||
```
|
||||
|
||||
Move the disk to SATA2-0 (reconnect it now if you unplugged it for the
|
||||
install), then:
|
||||
|
||||
```sh
|
||||
# --- on nas ---
|
||||
zpool import # confirm it is seen
|
||||
zpool import usb4t tank # rename: it is not USB any more
|
||||
zpool set cachefile=/etc/zfs/zpool.cache tank
|
||||
zpool status tank
|
||||
```
|
||||
|
||||
Properties survive from creation (`ashift=12`, `compression=zstd`,
|
||||
`atime=off`, `xattr=sa`, `acltype=posixacl`). Re-point the mountpoints
|
||||
and add the two new backup datasets:
|
||||
|
||||
```sh
|
||||
zfs set mountpoint=/export/media tank/media # NFSv4 export root (§8)
|
||||
zfs set mountpoint=none tank/backup
|
||||
zfs list -o name,used,avail,mountpoint
|
||||
```
|
||||
|
||||
Resulting layout:
|
||||
|
||||
```
|
||||
incus mirror, 2× SSD — nas's own container roots
|
||||
tank 4 TB, single vdev
|
||||
├── tank/media → /export/media NFS ro → nuc; local device → transmission-bt
|
||||
└── tank/backup
|
||||
├── tank/backup/nuc → incus pool `nucbackup` (nuc pushes here)
|
||||
└── tank/backup/ks4 → incus pool `ks4backup` (nas pulls from ks4)
|
||||
|
||||
nas's own replicas live on **nuc** (`data/backup/nas` → pool `nasbackup`),
|
||||
not here — see §9.
|
||||
```
|
||||
|
||||
Backup pools are named after the **source** host, matching
|
||||
[ks4/incus-copy.md](../ks4/incus-copy.md). nas's *own* instances are not
|
||||
backed up here — they cross-replicate to nuc (§9b), so neither host's
|
||||
instances depend on that host surviving.
|
||||
|
||||
Once `tank` has run a week on SATA with **zero** pool suspensions and no
|
||||
CRC errors, the `usb4t-dropouts` gate is cleared — record that in
|
||||
[ks2/plan.md](../ks2/plan.md).
|
||||
|
||||
## 6. Incus
|
||||
|
||||
Same Zabbly stable repo as nuc and ks4:
|
||||
|
||||
```sh
|
||||
mkdir -p /etc/apt/keyrings
|
||||
curl -fsSL https://pkgs.zabbly.com/key.asc -o /etc/apt/keyrings/zabbly.asc
|
||||
cat > /etc/apt/sources.list.d/zabbly-incus-stable.sources <<EOF
|
||||
Enabled: yes
|
||||
Types: deb
|
||||
URIs: https://pkgs.zabbly.com/incus/stable
|
||||
Suites: trixie
|
||||
Components: main
|
||||
Architectures: amd64
|
||||
Signed-By: /etc/apt/keyrings/zabbly.asc
|
||||
EOF
|
||||
apt update && apt install -y incus
|
||||
```
|
||||
|
||||
macvlan like nuc, so instances get real LAN addresses — `transmission-bt`
|
||||
keeps `192.168.0.7` when it moves:
|
||||
|
||||
```sh
|
||||
cat <<EOF | incus admin init --preseed
|
||||
config:
|
||||
core.https_address: :8443
|
||||
storage_pools:
|
||||
- name: incus
|
||||
driver: zfs
|
||||
config:
|
||||
source: incus
|
||||
networks:
|
||||
- name: macvlan
|
||||
type: macvlan
|
||||
config:
|
||||
parent: enp0s20f0
|
||||
profiles:
|
||||
- name: default
|
||||
devices:
|
||||
eth0: {name: eth0, network: macvlan, type: nic}
|
||||
root: {path: /, pool: incus, type: disk}
|
||||
EOF
|
||||
|
||||
incus profile set default snapshots.schedule="0 3 * * *" snapshots.expiry=7d
|
||||
```
|
||||
|
||||
Same macvlan quirk as nuc: **the host cannot talk to its own instances**,
|
||||
and vice versa. Test container services from another LAN host or from
|
||||
inside the container, never from `nas`.
|
||||
|
||||
Backup pools and the replica project:
|
||||
|
||||
```sh
|
||||
incus storage create nucbackup zfs source=tank/backup/nuc
|
||||
incus storage create ks4backup zfs source=tank/backup/ks4
|
||||
incus project create backup -c features.images=false -c features.profiles=false
|
||||
```
|
||||
|
||||
⚠️ **`tank/backup/nuc` already contains nuc's replicas** — they came
|
||||
across with the pool. Re-register them so refreshes stay
|
||||
ZFS-incremental instead of re-sending everything (the homeassistant VM
|
||||
alone is a 50 GiB volume):
|
||||
|
||||
```sh
|
||||
incus admin recover # point it at pool nucbackup; project backup
|
||||
incus list --project backup
|
||||
```
|
||||
|
||||
Same call nuc-install.md §5 uses after a rebuild. If `recover` is
|
||||
skipped, the first push in §9a silently becomes a full re-send of every
|
||||
nuc instance.
|
||||
|
||||
Order incus after the ZFS mounts, as on nuc —
|
||||
`/etc/systemd/system/incus.service.d/after-zfs.conf`:
|
||||
|
||||
```ini
|
||||
[Unit]
|
||||
After=zfs-mount.service zfs.target
|
||||
```
|
||||
|
||||
## 7. Move `transmission-bt` from nuc
|
||||
|
||||
Its WireGuard tunnel is **entirely inside the container** (`wg0`,
|
||||
`10.8.0.21`, `wg-quick@wg0`, `BindsTo=` on the daemon), so the container
|
||||
carries its own keys and **ks4 needs no change at all** — the peer stays
|
||||
`10.8.0.21/32`. The kill-switch `/32` route points at the gateway
|
||||
`192.168.0.2`, which is the same from here.
|
||||
|
||||
```sh
|
||||
# on nuc — remote already added in §9a
|
||||
incus stop transmission-bt
|
||||
incus move transmission-bt nas: --storage incus
|
||||
```
|
||||
|
||||
Then on nas, re-point the media device at the local dataset — this is a
|
||||
plain `shift=true` device again, because the data is local ZFS:
|
||||
|
||||
```sh
|
||||
incus config device remove transmission-bt media
|
||||
incus config device add transmission-bt media disk \
|
||||
source=/export/media path=/media shift=true
|
||||
incus config set transmission-bt boot.autostart=true
|
||||
incus start transmission-bt
|
||||
```
|
||||
|
||||
Verify the tunnel and the kill switch before trusting it:
|
||||
|
||||
```sh
|
||||
incus exec transmission-bt -- wg show
|
||||
incus exec transmission-bt -- curl -s ifconfig.me # must print 193.70.35.17
|
||||
incus exec transmission-bt -- ip route # must have NO default route
|
||||
```
|
||||
|
||||
⚠️ **The watch-folder workflow moves with it** —
|
||||
[transmission-bt.md](transmission-bt.md) says
|
||||
`scp some.torrent root@192.168.0.3:/srv/media/.watchdir/`; it is now
|
||||
`root@192.168.0.4:/export/media/.watchdir/`.
|
||||
|
||||
## 8. Media over NFS — `jellyfin-server` stays on nuc
|
||||
|
||||
`jellyfin-server` needs the Alder Lake-N iGPU for QSV/VAAPI; the C2750
|
||||
has no render device at all, and software transcoding on Silvermont
|
||||
manages 1–2 concurrent 1080p H.264 streams at best. So the container
|
||||
stays on nuc and reaches the library over NFS, **read-only** —
|
||||
`transmission-bt` is the only writer and it now lives here.
|
||||
|
||||
```sh
|
||||
# --- nas: NFSv4 export, read-only, nuc only ---
|
||||
cat >> /etc/exports <<'EOF'
|
||||
/export 192.168.0.3(ro,fsid=0,crossmnt,no_subtree_check)
|
||||
/export/media 192.168.0.3(ro,no_subtree_check,all_squash,anonuid=65534,anongid=65534)
|
||||
EOF
|
||||
exportfs -ra && exportfs -v
|
||||
```
|
||||
|
||||
```sh
|
||||
# --- nuc: mount at the SAME path, so jellyfin-server.md still applies ---
|
||||
mkdir -p /srv/media
|
||||
echo '192.168.0.4:/media /srv/media nfs4 ro,_netdev,soft,timeo=100,retrans=3 0 0' >> /etc/fstab
|
||||
mount /srv/media && ls /srv/media
|
||||
```
|
||||
|
||||
The container device changes only in losing the shift — per
|
||||
[nuc/jellyfin-server.md](../nuc/jellyfin-server.md)'s own troubleshooting
|
||||
note, *"on CIFS files are world-readable synthetic ownership, enough for
|
||||
a read-only library"*; the same holds for NFS with `all_squash`:
|
||||
|
||||
```sh
|
||||
incus stop jellyfin-server # shift cannot be hot-applied
|
||||
incus config device set jellyfin-server media shift=false
|
||||
incus config device set jellyfin-server media readonly=true
|
||||
incus start jellyfin-server
|
||||
incus exec jellyfin-server -- ls /media # must list the library
|
||||
```
|
||||
|
||||
Requirements for that to work: media files must be **world-readable**
|
||||
(`find /export/media -type f ! -perm -o=r`), and transmission must keep
|
||||
creating them that way (it does — `umask`/`0775` per its doc).
|
||||
|
||||
⚠️ Boot ordering on nuc: incus is already ordered after
|
||||
`zfs-mount.service`; add `remote-fs.target` to that drop-in, or
|
||||
`jellyfin-server` starts against an empty mountpoint and shows an empty
|
||||
library.
|
||||
|
||||
⚠️ `soft` mount is deliberate: a hung NAS should fail Jellyfin's reads,
|
||||
not wedge nuc's processes in uninterruptible sleep the way the suspended
|
||||
`usb4t` pool did.
|
||||
|
||||
## 9. Backup legs
|
||||
|
||||
The driver is unchanged —
|
||||
[`incus-copy.sh`](https://git.lutran.fr/julien/scripts/src/branch/main/incus-copy.sh),
|
||||
deployed to `/root/scripts` as everywhere else.
|
||||
|
||||
### 9a. nuc's instances -> nas (systemd timer, not cron)
|
||||
|
||||
⚠️ **nuc is an on-demand media box** (see [nuc/README.md](../nuc/README.md)):
|
||||
since 2026-08-30 it only runs when watching Jellyfin or using the Spotify
|
||||
kiosk, so it is usually powered off at 03:30. Cron cannot catch up a
|
||||
missed window; a **systemd timer with `Persistent=true`** runs the job
|
||||
shortly after the next boot. That is the whole reason this is a timer.
|
||||
|
||||
`/etc/systemd/system/incus-copy.service` on nuc:
|
||||
|
||||
```ini
|
||||
[Unit]
|
||||
Description=Replicate all nuc instances to nas (project backup, pool nucbackup)
|
||||
After=network-online.target incus.service
|
||||
Wants=network-online.target
|
||||
Requires=incus.service
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
ExecStart=/bin/sh -c "/root/scripts/incus-copy.sh -d nas -m push -s nucbackup -p backup >> /var/log/incus-copy.log 2>&1"
|
||||
```
|
||||
|
||||
`/etc/systemd/system/incus-copy.timer`:
|
||||
|
||||
```ini
|
||||
[Timer]
|
||||
OnCalendar=03:30
|
||||
Persistent=true
|
||||
RandomizedDelaySec=300
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
```
|
||||
|
||||
```sh
|
||||
systemctl enable --now incus-copy.timer && systemctl list-timers incus-copy
|
||||
```
|
||||
|
||||
⚠️ **`-p backup` is not optional.** Without it the replicas land in
|
||||
`default` on nas and collide with nas's *live* instances — both hosts are
|
||||
on the same macvlan LAN and the replicas carry the same static IPs.
|
||||
|
||||
### 9b. nas's own instances -> local pool `nasbackup`
|
||||
|
||||
**Decision 2026-08-30: local, not cross-replicated.** nuc is powered off
|
||||
most of the time, so it is not a usable backup target — a nightly push to
|
||||
it would fail noisily and, once mail works, alarm every morning. The
|
||||
replicas instead go to `tank/backup/nas`, which is a **different pool**
|
||||
from the instances themselves (`incus`, the SSD mirror), so it survives
|
||||
losing that mirror. All three instances here are rebuildable from their
|
||||
docs, so same-host is proportionate.
|
||||
|
||||
```sh
|
||||
zfs create tank/backup/nas
|
||||
zfs set mountpoint=legacy tank/backup/nas # see the trap below
|
||||
incus storage create nasbackup zfs source=tank/backup/nas
|
||||
```
|
||||
|
||||
```cron
|
||||
# nas, /etc/cron.d/incus-copy
|
||||
0 4 * * * root /root/scripts/incus-copy.sh -p backup -s nasbackup >> /var/log/incus-copy.log 2>&1
|
||||
```
|
||||
|
||||
⚠️ **Trap: `incus storage create` hangs forever on a `mountpoint=none`
|
||||
dataset.** `tank/backup` is set to `mountpoint=none`, so any child
|
||||
created afterwards inherits it, and the pool create then blocks with no
|
||||
error and no entry in `incus operation list` — it looks exactly like I/O
|
||||
contention (a scrub was running, which sent me down that path for 20
|
||||
minutes). Set the child to `legacy` to match its siblings first, and it
|
||||
completes instantly.
|
||||
|
||||
### 9c. Cleanup on nuc after the pool move
|
||||
|
||||
|
||||
Exporting `usb4t` leaves nuc with an incus storage pool whose backing
|
||||
dataset is gone, plus replica records in the `backup` project pointing
|
||||
at it. Remove them once §6's `incus admin recover` has re-registered the
|
||||
same volumes on nas — **verify there first, then delete here**:
|
||||
|
||||
```sh
|
||||
# on nas: confirm the replicas are registered
|
||||
incus list --project backup -c ns -f csv
|
||||
|
||||
# on nuc: only then
|
||||
incus delete --project backup --force <each-replica>
|
||||
incus storage delete nucbackup
|
||||
zpool status # only `data` should remain
|
||||
```
|
||||
|
||||
### 9d. ks4 pull leg moves from nuc to nas
|
||||
|
||||
The leg [ks2/nas-seed.md](../ks2/nas-seed.md) prepared. The target pool
|
||||
moves, so the **host** WireGuard tunnel moves too — nuc drops `wg-ks4`
|
||||
entirely once this works (its only other tunnel user, `transmission-bt`,
|
||||
is now here and carries its own).
|
||||
|
||||
- **ks4**: add nas as a peer on the `wireguard` container —
|
||||
`wg set wg0 peer <nas-pubkey> allowed-ips 10.8.0.22/32 && wg-quick save wg0`.
|
||||
The existing ufw rule (`allow in on incusbr0 from 192.168.1.18 to any
|
||||
port 8443 proto tcp`) already covers it: traffic arrives masqueraded as
|
||||
the WG container whichever peer sent it.
|
||||
- **nas**: `/etc/wireguard/wg-ks4.conf` modelled on nuc's —
|
||||
`Address = 10.8.0.22/32`, peer pubkey `TVs6d7…`,
|
||||
`Endpoint = 193.70.35.17:51845`,
|
||||
`AllowedIPs = 10.8.0.0/24, 192.168.1.1/32`, keepalive 25 — then
|
||||
`systemctl enable --now wg-quick@wg-ks4`.
|
||||
- `incus remote add ks4 https://192.168.1.1:8443 --accept-certificate --token '…'`
|
||||
(cross-check the fingerprint against the token).
|
||||
- **nuc, after verification**: `systemctl disable --now wg-quick@wg-ks4`,
|
||||
remove `/etc/wireguard/wg-ks4.conf`, and drop the `10.8.0.20/32` peer
|
||||
on ks4.
|
||||
|
||||
Seed manually in tmux — the first pass is **~1.75 TiB** over the home
|
||||
downlink and only makes sense once FTTH is up:
|
||||
|
||||
```sh
|
||||
/root/scripts/incus-copy.sh -r ks4 -s ks4backup 2>&1 | tee -a /var/log/incus-copy-ks4.log
|
||||
```
|
||||
|
||||
Then test-restore one instance before ticking the gate in
|
||||
[ks2/plan.md](../ks2/plan.md).
|
||||
|
||||
### Resulting schedule
|
||||
|
||||
| When | Host | What |
|
||||
|---|---|---|
|
||||
| 03:00 | nuc, nas | instance snapshots (profile) |
|
||||
| 03:30 | nuc | push all instances → `nas:nucbackup` (**systemd timer**, `Persistent=true` — catches up after boot) |
|
||||
| 04:00 | nas | local copy of nas instances → `nasbackup` (`tank/backup/nas`) |
|
||||
| 05:00 | nas | pull `ks4:*` → `ks4backup` (after FTTH) |
|
||||
| 05:30 | nuc | apt upgrade all containers |
|
||||
| Mon ~00:12 | nas | `zfs-scrub-weekly@tank.timer` / `@incus.timer` (systemd, not cron) |
|
||||
|
||||
Staggered around ks4's own 01:00 / 05:00 jobs.
|
||||
|
||||
⚠️ Keep `snapshots.schedule` on every source. A refresh with no common
|
||||
snapshot degrades to a **full re-send** — the failure mode that cost
|
||||
933 G on ks4 ([ks4/local-backup-cron.md](../ks4/local-backup-cron.md)).
|
||||
The homeassistant **VM** on nuc re-sends its whole volume without them.
|
||||
Note `incus-copy.sh` takes a **global** `flock`: an overrunning ks4 pull
|
||||
aborts that night's other run loudly rather than racing it.
|
||||
|
||||
## 10. Monitoring
|
||||
|
||||
Do not repeat nuc's 11-day blind spot
|
||||
([usb4t-dropouts.md](../nuc/usb4t-dropouts.md) — zed was running, but the
|
||||
host had no MTA).
|
||||
|
||||
**Status: live since 2026-08-30**, verified end to end (`smtpstatus=250`).
|
||||
|
||||
- **msmtp**, same shape as nuc: this box shares the dynamic home IP with
|
||||
no PTR and no SPF alignment, so it must use **submission (587) with
|
||||
auth**, not port 25 — rspamd rejects the port-25 path as spam.
|
||||
`/etc/msmtprc` mode 600, host `mail.lutran.fr`, STARTTLS. Give SMTP
|
||||
tests ≥ 30 s: the missing PTR delays the greeting.
|
||||
The `zed@lutran.fr` account is the **same credential as nuc** — it is
|
||||
SMTP-AUTH, not IP-bound, so `/etc/msmtprc` can simply be copied between
|
||||
hosts (mode 600, root:root). The password lives in the password
|
||||
manager; no host-specific setup is needed.
|
||||
- **zed**: `/etc/zfs/zed.d/zed.rc` mode 600 with
|
||||
`ZED_EMAIL_ADDR="julien@lutran.fr"`, `ZED_EMAIL_PROG="mail"`,
|
||||
`ZED_NOTIFY_VERBOSE=1`, **`ZED_NOTIFY_DATA=1`**,
|
||||
`ZED_NOTIFY_INTERVAL_SECS=3600`.
|
||||
- **[`zpool-health.sh`](https://git.lutran.fr/julien/scripts/src/branch/main/zpool-health.sh)**
|
||||
every 15 min via `/etc/cron.d/zpool-health` — zed does *not* report a
|
||||
suspended pool (the vdev stays `ONLINE`, so `statechange-notify.sh`
|
||||
never fires). That watchdog is the only thing that catches the exact
|
||||
failure this box was built to prevent.
|
||||
- **smartd** — `/etc/smartd.conf`. The Toshiba carries the useful
|
||||
attributes; the Intel is a liveness check only:
|
||||
|
||||
```
|
||||
# Toshiba Q300 — endurance + temperature are real here
|
||||
/dev/disk/by-id/ata-TOSHIBA_Q300._36OB318OK1KU -a -o on -S on -s (S/../.././02|L/../../6/03) -W 4,50,55 -m julien@lutran.fr
|
||||
# Intel 330 — no error log, no devstat, no temperature: liveness only
|
||||
/dev/disk/by-id/ata-INTEL_SSDSC2CT120A3_CVMP250400ES120BGN -H -s (S/../.././03|L/../../6/04) -m julien@lutran.fr
|
||||
# WD Red
|
||||
/dev/disk/by-id/ata-WDC_WD40EFRX-68WT0N0_WD-WCC4E6NLPJJE -a -o on -S on -s (S/../.././04|L/../../6/05) -W 4,45,50 -m julien@lutran.fr
|
||||
```
|
||||
|
||||
Verify the whole chain the day you build it, not the day you need it:
|
||||
|
||||
```sh
|
||||
/root/scripts/zpool-health.sh -m julien@lutran.fr -t
|
||||
tail -2 /var/log/msmtp.log # expect smtpstatus=250
|
||||
smartctl -d sat -l devstat /dev/sdX | grep -i endurance
|
||||
```
|
||||
|
||||
## 11. Power baseline
|
||||
|
||||
Take the measurement **before** the box goes into service, so later
|
||||
readings mean something. Expect roughly 25–35 W idle with three disks:
|
||||
the 20 W SoC plus a BMC drawing several watts continuously, even at
|
||||
soft-off.
|
||||
|
||||
```sh
|
||||
# baseline: 10 min idle, everything settled
|
||||
# CPU in steps — the interesting curve for an always-on box
|
||||
for l in 25 50 75 100; do echo "=== ${l}% $(date +%s)"; stress-ng --cpu 0 --cpu-load $l --timeout 120s; done
|
||||
# disk: random I/O is what moves an HDD's power, not throughput
|
||||
fio --name=rr --directory=/export/media --size=20G --rw=randread --bs=4k \
|
||||
--iodepth=32 --numjobs=4 --ioengine=libaio --direct=1 --runtime=300 --time_based
|
||||
```
|
||||
|
||||
Measure at the wall (PDU or smart plug) — RAPL is unreliable on Avoton
|
||||
and sees neither disks nor fans. Log epoch timestamps per step so the
|
||||
trace can be recut against the meter's series afterwards.
|
||||
|
||||
## 12. Post-install checklist
|
||||
|
||||
- [x] IPMI **left unplugged by decision** (2026-08-30) — no BMC on the
|
||||
LAN, therefore no out-of-band console either
|
||||
- [ ] `cat /proc/mdstat` — all three arrays `[UU]`; GRUB written to **both**
|
||||
ESPs (`efibootmgr -v` shows two entries),
|
||||
and boot tested with one disk unplugged
|
||||
- [ ] `zpool status` healthy for `incus` and `tank`; weekly scrubs
|
||||
scheduled; both import cleanly after a reboot (`cachefile` set)
|
||||
- [ ] **7 days with zero pool suspensions and zero CRC errors** — the
|
||||
gate that closes [usb4t-dropouts.md](../nuc/usb4t-dropouts.md)
|
||||
- [x] `smartd` monitoring all 3 disks; `zpool-health.sh -t` mail
|
||||
delivered (`smtpstatus=250` in `/var/log/msmtp.log`) — done
|
||||
2026-08-30
|
||||
- [ ] `transmission-bt` on nas: egress is `193.70.35.17`, **no default
|
||||
route**, downloads land in `/export/media/downloads`, watch folder
|
||||
works from the new path
|
||||
- [ ] `jellyfin-server` on nuc lists the library over NFS after a **cold
|
||||
reboot of both hosts** (the boot-ordering trap)
|
||||
- [ ] `incus admin recover` ran on nas **before** the first push, so the
|
||||
inherited `tank/backup/nuc` replicas refresh incrementally instead
|
||||
of re-sending (check the first run's duration, not just `rc=0`)
|
||||
- [ ] nuc's 03:30 leg → `nas:nucbackup`, `rc=0`, all nuc instances
|
||||
present (`incus list --project backup -c ns -f csv` on nas)
|
||||
- [ ] nas's 04:00 local leg → `nasbackup`, `rc=0`, and **blocky,
|
||||
privoxy and transmission-bt all appear by name** in
|
||||
`incus list --project backup`
|
||||
- [ ] stale `nucbackup` pool removed from nuc (§9c), `zpool status`
|
||||
shows only `data`
|
||||
- [ ] replicas are **stopped** with `boot.autostart=false` — they hold
|
||||
the live containers' LAN addresses
|
||||
- [ ] ks4 pull leg seeded + one instance test-restored → tick the gate
|
||||
in [ks2/plan.md](../ks2/plan.md); then retire nuc's `wg-ks4`
|
||||
- [ ] power baseline recorded above, with the meter reading
|
||||
- [ ] this file updated with what was built (RAM fitted, NIC name, WD Red
|
||||
serial)
|
||||
@@ -1,6 +1,13 @@
|
||||
# transmission-bt
|
||||
|
||||
BitTorrent client in an unprivileged Incus container on `nuc`, with an
|
||||
|
||||
> Moved from nuc to `nas` on 2026-08-30, together with the media
|
||||
> dataset ([nas-install.md](nas-install.md) §7). Its WireGuard tunnel is
|
||||
> entirely in-container, so ks4 needed no change — the peer is still
|
||||
> `10.8.0.21`. The watch folder moved with it:
|
||||
> `/export/media/.watchdir` on nas, not `/srv/media/.watchdir` on nuc.
|
||||
|
||||
BitTorrent client in an unprivileged Incus container on `nas`, with an
|
||||
**always-on VPN**: all peer traffic exits via ks4's public IP through a
|
||||
WireGuard tunnel to the `wireguard` container on ks4. Kill switch by
|
||||
construction — `eth0` has **no default route**, so with the tunnel down
|
||||
@@ -12,10 +19,10 @@ the container simply has no path to the internet.
|
||||
whitelist (`192.168.0.*` only)
|
||||
- Egress: WG peer `10.8.0.21` → `193.70.35.17:51845`, `AllowedIPs 0.0.0.0/0`
|
||||
(verified: `curl ifconfig.me` from the container returns ks4's IP)
|
||||
- Downloads: `/media/downloads` (= `usb4t/media`, same dataset Jellyfin
|
||||
- Downloads: `/media/downloads` (= `tank/media`, same dataset Jellyfin
|
||||
reads); in-progress files in `/media/.incomplete` so Jellyfin never
|
||||
scans partials
|
||||
- Watch folder: `scp` a `.torrent` into `/srv/media/.watchdir` on the host
|
||||
- Watch folder: `scp` a `.torrent` into `/export/media/.watchdir` on the host
|
||||
and it auto-downloads (see "Watch folder" below)
|
||||
- `transmission-daemon` is `BindsTo=wg-quick@wg0.service` and binds
|
||||
peer traffic to `10.8.0.21` — three independent layers against leaks
|
||||
@@ -90,7 +97,7 @@ EOF'
|
||||
incus exec "$CNAME" -- systemctl enable --now wg-quick@wg0
|
||||
|
||||
# media share (same dataset as jellyfin-server)
|
||||
incus config device add "$CNAME" media disk source=/srv/media path=/media shift=true
|
||||
incus config device add "$CNAME" media disk source=/export/media path=/media shift=true
|
||||
incus exec "$CNAME" -- mkdir -p /media/downloads /media/.incomplete
|
||||
incus exec "$CNAME" -- chown debian-transmission:debian-transmission \
|
||||
/media/downloads /media/.incomplete
|
||||
@@ -129,9 +136,9 @@ incus exec wireguard -- wg-quick save wg0
|
||||
|
||||
## Watch folder (auto-add torrents)
|
||||
|
||||
Drop a `.torrent` into `/srv/media/.watchdir` on the host and Transmission
|
||||
Drop a `.torrent` into `/export/media/.watchdir` on the host and Transmission
|
||||
auto-adds it and starts downloading — no web UI needed. The folder lives on
|
||||
the shared `usb4t/media` dataset (`/media/.watchdir` inside the container).
|
||||
the shared `tank/media` dataset (`/media/.watchdir` inside the container).
|
||||
|
||||
Edit the **active** config only while the daemon is stopped (it rewrites
|
||||
`settings.json` on exit). The active file is
|
||||
@@ -161,7 +168,7 @@ incus exec transmission-bt -- systemctl start transmission-daemon
|
||||
Usage — the `.torrent` is consumed within a few seconds:
|
||||
|
||||
```sh
|
||||
scp some.torrent root@192.168.0.3:/srv/media/.watchdir/
|
||||
scp some.torrent root@192.168.0.4:/export/media/.watchdir/
|
||||
```
|
||||
|
||||
- `watch-dir-force-generic: true` makes Transmission **poll** the folder
|
||||
@@ -179,7 +186,7 @@ incus exec transmission-bt -- wg show wg0 latest-handshakes # non-zero timesta
|
||||
incus exec transmission-bt -- curl -s https://ifconfig.me # must print 193.70.35.17
|
||||
incus exec transmission-bt -- bash -c "ping -c1 -W2 8.8.8.8 || echo kill-switch OK" # with wg0 down
|
||||
# web UI must be tested from a LAN machine — the macvlan quirk means the
|
||||
# nuc host itself cannot reach 192.168.0.7
|
||||
# nas host itself cannot reach 192.168.0.7 (macvlan, by design)
|
||||
```
|
||||
|
||||
## Notes
|
||||
Reference in New Issue
Block a user