Compare commits
53
Commits
4b30b43c7f
...
main
@@ -1,101 +1,33 @@
|
||||
# perso
|
||||
# Personal infrastructure notes
|
||||
|
||||
Personal infrastructure notes.
|
||||
Two repositories back this setup:
|
||||
[**doc**](https://git.lutran.fr/julien/doc) (this one) and
|
||||
[**scripts**](https://git.lutran.fr/julien/scripts) (the cron drivers
|
||||
and their config files, deployed to `/root/scripts` on each host).
|
||||
|
||||
- [`nuc/`](nuc/) — home lab on `nuc` (see below)
|
||||
- [`ks4/`](ks4/) — prod server `ks4` at OVH (see below)
|
||||
**Start here**:
|
||||
- [backup-strategy.md](backup-strategy.md) — what is
|
||||
backed up, by which tool, on what schedule, and how to restore.
|
||||
- [new-container.md](new-container.md) — conventions for adding a
|
||||
service to ks4 (and getting it backed up automatically).
|
||||
|
||||
## Homelab: nuc
|
||||
**Tech notes**
|
||||
- [`nuc/`](nuc/README.md) — home lab on `nuc`: host, iGPU instances
|
||||
- [`nas/`](nas/README.md) — storage + backup host `nas` (Supermicro
|
||||
A1SAi-2750F): the 4 TB on direct SATA, media over NFS, backup pools
|
||||
- [`ks4/`](ks4/README.md) — prod server `ks4` at OVH: host, services, network flows
|
||||
- [`ks2/`](ks2/) — legacy backup server being [decommissioned](ks2/plan.md)
|
||||
- [`archer-c7/`](archer-c7/) — home router (TP-Link Archer C7 v5 running on OpenWrt)
|
||||
|
||||
Incus host on the LAN.
|
||||
## Conventions
|
||||
|
||||
- Host: `nuc` — `192.168.0.3`, Debian 13, Intel Alder Lake-N
|
||||
(iGPU `i915`, shared by both Jellyfin containers) —
|
||||
bare-metal reinstall: [nuc/nuc-install.md](nuc/nuc-install.md)
|
||||
- SSH: `ssh -i id_rsa_claude root@192.168.0.3` (keys at repo root, gitignored)
|
||||
- Instances are bridged onto the LAN (192.168.0.0/24)
|
||||
- USB 4 TB WD Red: ZFS pool `usb4t` — `usb4t/backup` → `/backup`
|
||||
(incus exports, `nuc/` + `ks4/` subdatasets, 1 TB quota) and
|
||||
`usb4t/media` → `/srv/media` (media library, shared into containers
|
||||
via `shift=true` disk devices; works because ZFS ≥ 2.2 supports
|
||||
idmapped mounts)
|
||||
- NAS: `//192.168.0.10/Shared` (CIFS) — original media source, normally
|
||||
offline; not mountable into containers (idmapped mounts unsupported
|
||||
on CIFS)
|
||||
- Backups: all local instances replicated to the USB pool
|
||||
(`/root/scripts/incus-copy.sh -p backup -s nucbackup`; replicas
|
||||
stopped, autostart off) — see
|
||||
[nuc/nuc-install.md](nuc/nuc-install.md); ks4 replicas pulled into
|
||||
pool `ks4backup` — see [ks4/incus-copy.md](ks4/incus-copy.md)
|
||||
|
||||
### Instances
|
||||
|
||||
| Name | IP | Doc | Features |
|
||||
|---|---|---|---|
|
||||
| [jellyfin-server](nuc/jellyfin-server.md) | 192.168.0.5 | ✅ | unprivileged, autostart; iGPU render node (`gpu` device, render gid) for QSV/VAAPI transcoding; `/srv/media` disk device (`shift=true`); proxy device → host :8096 |
|
||||
| [jellyfin-client](nuc/jellyfin-client.md) | 192.168.0.6 | ✅ | **privileged**, autostart; full iGPU (`gpu` device, gid 44) → HDMI kiosk (cage + Jellyfin Media Player); custom `raw.lxc` (bind `/dev/snd`, `/dev/input`, host `/run/udev`); Pioneer USB audio as ALSA default; FR keymap; go-librespot Spotify Connect ("Pioneer A-70") |
|
||||
| [transmission-bt](nuc/transmission-bt.md) | 192.168.0.7 | ✅ | unprivileged, autostart; always-on WireGuard full tunnel → ks4 (egress = 193.70.35.17, kill switch: no default route); `/srv/media` disk device (`shift=true`), downloads to `/media/downloads`; web UI :9091 (LAN only) |
|
||||
| blocky | 192.168.0.254 | — | unprivileged, autostart; DNS ad-blocker |
|
||||
| privoxy | 192.168.0.11 | — | unprivileged, autostart; filtering HTTP proxy |
|
||||
| homeassistant | (stopped) | — | **virtual machine**, 50 GiB root disk on pool `data` |
|
||||
|
||||
## Prod: ks4
|
||||
|
||||
Incus host at OVH — public-facing self-hosted services.
|
||||
|
||||
- Host: `ks4.lutran.fr` — `193.70.35.17`, **SSH on port 2233**,
|
||||
Debian 13, Xeon D-1521 / 32 GiB, 2× 6 TB (OS on mdraid RAID1,
|
||||
ZFS pool `data` on `sda5`) — setup & rebuild:
|
||||
[ks4/install.md](ks4/install.md)
|
||||
- SSH: `ssh -i id_rsa_claude -p 2233 julien@193.70.35.17`
|
||||
(`julien` has passwordless `sudo incus …`)
|
||||
- Instances are on a NAT bridge `incusbr0` (192.168.1.0/24); public
|
||||
ingress via Incus proxy devices. `gateway` (nginx) is the sole HTTP/S
|
||||
entry point and fans out to the app containers.
|
||||
- Services: gateway, mail, nextcloud, seafile, git, freshrss, bitwarden,
|
||||
outline, login (SSO), wireguard, + more — full table in
|
||||
[ks4/install.md](ks4/install.md).
|
||||
- ⚠️ The ZFS `data` pool is single-disk (not mirrored); durability
|
||||
rests on two nightly root cron jobs — `incus copy --refresh` to remote
|
||||
host `ks2` (moving to nuc pool `ks4backup`, see
|
||||
[ks4/incus-copy.md](ks4/incus-copy.md)), and an rsync backup
|
||||
(`scripts/incus-backup.sh`) to `164.132.173.57:/backup/ks4`.
|
||||
|
||||
## Network flows (nuc <-> ks4)
|
||||
|
||||
```
|
||||
nuc — home LAN 192.168.0.0/24 ks4 — OVH 193.70.35.17
|
||||
+-----------------------------------+ +-------------------------------------+
|
||||
| | | |
|
||||
| host: wg-ks4 (10.8.0.20) | | [wireguard] 192.168.1.18 |
|
||||
| incus remote "ks4" ------+--WG-->| wg0 10.8.0.1/24, udp 51845 |
|
||||
| pull ks4:* -> pool ks4backup | udp | | masquerade -> eth0 |
|
||||
| on usb4t [pending FTTH seed] | 51845 | | |
|
||||
| | | +-> incus API 192.168.1.1:8443 |
|
||||
| [transmission-bt] wg0 (10.8.0.21) | | | (ufw: only from .18) |
|
||||
| full tunnel 0.0.0.0/0 ------+--WG-->| | |
|
||||
| kill switch: no default route | udp | +-> WAN egress: torrents + |
|
||||
| downloads -> /srv/media | 51845 | apt of transmission-bt |
|
||||
| (usb4t/media, read by jellyfin) | | exit as 193.70.35.17 |
|
||||
| | | |
|
||||
| 03:00 instance snapshots | | 03:00 instance snapshots |
|
||||
| 03:30 incus-copy: all instances | | 01:00 incus-copy: all instances |
|
||||
| -> project backup, pool | | -> project backup, zpool |
|
||||
| nucbackup (usb4t/backup/nuc) | | backup (sdb5) [planned] |
|
||||
| 05:00 apt upgrade all containers | | 04:00 incus-backup.sh: DB dumps |
|
||||
| | | -> /backup (sdb5) [planned] |
|
||||
+-----------------------------------+ +-------------------------------------+
|
||||
phones/laptops: WG peers 10.8.0.2-3 reach 192.168.1.x through the same endpoint
|
||||
```
|
||||
|
||||
Both tunnels initiate **from** nuc (home NAT, dynamic IP) toward ks4's
|
||||
fixed endpoint; ks4's incus API is never exposed to the internet.
|
||||
|
||||
### Conventions
|
||||
|
||||
- One markdown file per instance in `nuc/`, containing the full
|
||||
install script (idempotent-ish, run as root on the host), first-run
|
||||
configuration, and troubleshooting notes.
|
||||
- Containers are built from `images:ubuntu/24.04` and configured
|
||||
- One markdown file per topic in the correct subdir, containing the full
|
||||
implementation notes, configuration, and troubleshooting notes.
|
||||
- Containers are built from Debian/Ubuntu images and configured
|
||||
exclusively through `incus exec`/`incus config` so the doc is the
|
||||
single source of truth — rebuilding = re-running the script.
|
||||
- Adding a service to ks4: follow
|
||||
[new-container.md](new-container.md) (no Docker in containers —
|
||||
incus runs OCI images natively and `incus-compose` handles stacks;
|
||||
data paths go into
|
||||
[`scripts/restic-paths`](https://git.lutran.fr/julien/scripts/src/branch/main/restic-paths)).
|
||||
|
||||
@@ -0,0 +1,193 @@
|
||||
# Archer C7: upgrade OpenWrt 19.07.4 → 25.12.5
|
||||
|
||||
> **Outcome (2026-08-20):** upgrade done (flashed clean per this doc).
|
||||
> The USB-tether flapping survived the new kernel → root cause is the
|
||||
> C7's USB port lacking charging current (phone dropped off the bus
|
||||
> whenever it drew charge). **WAN is now the phone's Wi-Fi hotspot**:
|
||||
> 2.4 GHz `radio1` in STA mode → interface `wwan` (`phy1-sta0`, in the
|
||||
> wan zone), phone on a mains charger. `usbwan` stays configured for a
|
||||
> future powered-USB-hub wired option. Post-flash gotcha: fw4 compiled
|
||||
> NAT for `eth0.2` only while `usbwan` was down at reload time — LAN
|
||||
> had no internet until `service firewall restart` with the WAN
|
||||
> interface up.
|
||||
|
||||
Home router **TP-Link Archer C7 v5** (`tplink,archer-c7-v5`, target
|
||||
`ath79/generic`, 16 MB flash / 128 MB RAM) at `192.168.0.2`, running
|
||||
OpenWrt 19.07.4 (2020, kernel 4.14, EOL since 2022). Target release:
|
||||
**OpenWrt 25.12.5** (kernel 6.12.94, firewall4/nftables, `apk` package
|
||||
manager instead of `opkg`).
|
||||
|
||||
Context (2026-08-19): after a reboot the router came up with **dnsmasq,
|
||||
firewall and odhcpd disabled** (missing `/etc/rc.d/` symlinks — no DHCP,
|
||||
no NAT). All three were restarted and re-enabled, but the incident plus
|
||||
the EOL release motivated this upgrade. A fresh flash also rules out
|
||||
any other overlay damage.
|
||||
|
||||
Current setup worth knowing:
|
||||
|
||||
- **WAN = OPPO phone (SM8350) USB-tethered** on `usb0` via RNDIS
|
||||
(`kmod-usb-net-rndis`), interface `usbwan`, DHCP from the phone
|
||||
(10.205.61.0/24). The ethernet WAN port is unused — the ISP box at
|
||||
`.1` is dead, which is also why the router LAN IP is `.2`.
|
||||
- SSH: `ssh -i id_rsa_claude root@192.168.0.2` (key at perso repo root;
|
||||
19.07's dropbear needs
|
||||
`-o HostKeyAlgorithms=+ssh-rsa -o PubkeyAcceptedAlgorithms=+ssh-rsa`
|
||||
from a modern OpenSSH; 25.12 won't)
|
||||
- DNS: dnsmasq forwards everything to **blocky** at `192.168.0.254`
|
||||
(`noresolv` + `server=192.168.0.254`), and DHCP option 6 hands blocky
|
||||
directly to clients
|
||||
- Wi-Fi: 5 GHz only (`radio0`, ch 36, VHT80, SSID `HOME`, WPA2-PSK);
|
||||
2.4 GHz radio disabled
|
||||
- 7 static DHCP leases (see script below)
|
||||
- Only user-installed packages on 19.07: `kmod-usb-net-rndis` + deps
|
||||
|
||||
## Prerequisites (done 2026-08-19)
|
||||
|
||||
- [x] Config backup: `sysupgrade -b` → `~/code/perso/backup-archer-c7-20260819.tar.gz`
|
||||
(laptop; contains Wi-Fi PSK and dropbear host keys — never commit)
|
||||
- [x] Firmware `openwrt-25.12.5-ath79-generic-tplink_archer-c7-v5-squashfs-sysupgrade.bin`
|
||||
downloaded, **sha256 verified** (`f15b16b3…`) against the release
|
||||
`sha256sums`, staged in `~/code/perso/archer-c7-upgrade/` and on
|
||||
the router at `/tmp/` (`/tmp` is RAM — gone after any reboot,
|
||||
re-`scp` if needed)
|
||||
- [x] **Offline driver packages** staged in `~/code/perso/archer-c7-upgrade/`:
|
||||
`kmod-mii`, `kmod-usb-net`, `kmod-usb-net-cdc-ether`,
|
||||
`kmod-usb-net-rndis` (all `6.12.94-r1.apk`). Critical: the phone
|
||||
tether is the only WAN and the RNDIS driver is **not** in the
|
||||
default image, so after flashing the router has no internet to
|
||||
install it from — it must be `scp`'d in.
|
||||
|
||||
## Upgrade
|
||||
|
||||
**Do not keep settings across this jump** (19.07 → 25.12 spans the
|
||||
fw3→fw4 and network-syntax migrations). Flash clean with `-n` and
|
||||
reconfigure with the script below.
|
||||
|
||||
You need the **laptop on ethernet** to a LAN port: the fresh image
|
||||
boots at `192.168.1.1` with Wi-Fi disabled.
|
||||
|
||||
```sh
|
||||
# on the router — image already in /tmp, verify then flash
|
||||
sha256sum /tmp/openwrt-25.12.5-ath79-generic-tplink_archer-c7-v5-squashfs-sysupgrade.bin
|
||||
# expect f15b16b36df7fce8a5cf1302320fdb428c3bb6fbc2949a4858afa257f4f364a2
|
||||
sysupgrade -n -v /tmp/openwrt-25.12.5-ath79-generic-tplink_archer-c7-v5-squashfs-sysupgrade.bin
|
||||
```
|
||||
|
||||
The router reboots (~3 min, don't power-cycle). Then from the laptop
|
||||
(ethernet, DHCP gives a 192.168.1.x address):
|
||||
|
||||
```sh
|
||||
# passwordless root at first boot — set a password and install the key
|
||||
ssh root@192.168.1.1 # no password
|
||||
passwd
|
||||
mkdir -p /etc/dropbear && exit
|
||||
scp ~/code/perso/id_rsa_claude.pub root@192.168.1.1:/etc/dropbear/authorized_keys
|
||||
|
||||
# push the offline RNDIS drivers
|
||||
scp ~/code/perso/archer-c7-upgrade/kmod-*.apk root@192.168.1.1:/tmp/
|
||||
```
|
||||
|
||||
## Reconfigure
|
||||
|
||||
Run on the router. Single source of truth for the whole config;
|
||||
`<wifi-psk>` is in the backup tarball (`etc/config/wireless`).
|
||||
|
||||
```sh
|
||||
#!/bin/sh
|
||||
set -eux
|
||||
|
||||
# RNDIS drivers (offline — phone tether is the only WAN)
|
||||
apk add --allow-untrusted /tmp/kmod-mii-*.apk /tmp/kmod-usb-net-6*.apk \
|
||||
/tmp/kmod-usb-net-cdc-ether-*.apk /tmp/kmod-usb-net-rndis-*.apk
|
||||
|
||||
# --- network: keep the generated switch/bridge config, change only what we need
|
||||
uci set network.lan.ipaddr='192.168.0.2'
|
||||
uci set network.usbwan=interface
|
||||
uci set network.usbwan.proto='dhcp'
|
||||
uci set network.usbwan.device='usb0'
|
||||
uci commit network
|
||||
|
||||
# --- firewall: usbwan joins the wan zone (zone[1] = 'wan' in the default config)
|
||||
uci add_list firewall.@zone[1].network='usbwan'
|
||||
uci commit firewall
|
||||
|
||||
# --- dhcp/dns: forward to blocky, hand blocky to clients, static leases
|
||||
uci set dhcp.@dnsmasq[0].noresolv='1'
|
||||
uci add_list dhcp.@dnsmasq[0].server='192.168.0.254'
|
||||
uci add_list dhcp.lan.dhcp_option='6,192.168.0.254'
|
||||
add_host() {
|
||||
uci add dhcp host >/dev/null
|
||||
uci set dhcp.@host[-1].mac="$1"
|
||||
uci set dhcp.@host[-1].ip="$2"
|
||||
uci set dhcp.@host[-1].name="$3"
|
||||
uci set dhcp.@host[-1].dns='1'
|
||||
}
|
||||
add_host 'F4:5C:89:BE:C1:C3' '192.168.0.20' 'LAPTOP719974'
|
||||
add_host '10:66:6A:87:A1:49' '192.168.0.254' 'blocky'
|
||||
add_host '10:66:6A:C8:E1:0A' '192.168.0.5' 'jellyfin-server'
|
||||
add_host '10:66:6A:FF:59:1C' '192.168.0.6' 'jellyfin-client'
|
||||
add_host '10:66:6A:85:20:B7' '192.168.0.11' 'privoxy'
|
||||
add_host '10:66:6A:55:9F:10' '192.168.0.7' 'transmission-bt'
|
||||
add_host '50:9A:4C:20:D9:2B' '192.168.0.21' 'patate'
|
||||
uci commit dhcp
|
||||
|
||||
# --- wifi: 5 GHz AP as before, upgraded to WPA2/WPA3 mixed; 2.4 GHz stays off
|
||||
uci set wireless.radio0.channel='36'
|
||||
uci set wireless.radio0.htmode='VHT80'
|
||||
uci set wireless.radio0.disabled='0'
|
||||
uci set wireless.default_radio0.ssid='HOME'
|
||||
uci set wireless.default_radio0.encryption='sae-mixed'
|
||||
uci set wireless.default_radio0.key='<wifi-psk>'
|
||||
uci set wireless.default_radio0.network='lan'
|
||||
uci commit wireless
|
||||
|
||||
reload_config
|
||||
/etc/init.d/network restart
|
||||
```
|
||||
|
||||
The LAN IP change kicks you off; reconnect at `192.168.0.2` (plug the
|
||||
phone back in for WAN).
|
||||
|
||||
## Verify
|
||||
|
||||
```sh
|
||||
ssh root@192.168.0.2
|
||||
ifstatus usbwan | grep -E 'up|address' # up:true, 10.205.61.x
|
||||
ping -c2 8.8.8.8 && ping -c2 openwrt.org # WAN + DNS via blocky
|
||||
nft list chain inet fw4 srcnat_wan # masquerade on usbwan
|
||||
ls /etc/rc.d/ | grep -E 'dnsmasq|firewall|odhcpd' # all enabled
|
||||
```
|
||||
|
||||
Then from a LAN client: renew DHCP (expect 192.168.0.100–249, DNS
|
||||
`.254`), check internet, check the static leases land on their IPs.
|
||||
|
||||
## Known issue: 5 GHz `ath10k-ct`
|
||||
|
||||
The C7 v2/v5 has a long-standing 5 GHz issue with the default
|
||||
`ath10k-ct` driver: `SWBA overrun on vdev 0` log spam plus client
|
||||
drops under some loads. **Only if it shows up**, swap to the non-ct
|
||||
driver (needs working WAN, or pre-download the same way as the RNDIS
|
||||
apks):
|
||||
|
||||
```sh
|
||||
apk del kmod-ath10k-ct ath10k-firmware-qca988x-ct
|
||||
apk add kmod-ath10k ath10k-firmware-qca988x
|
||||
reboot
|
||||
```
|
||||
|
||||
## Recovery
|
||||
|
||||
If the flash bricks: C7 v5 has TFTP recovery — hold **Reset** while
|
||||
powering on, router pulls `ArcherC7v5_tp_recovery.bin` (the OpenWrt
|
||||
**factory** image renamed to that) from a TFTP server at
|
||||
`192.168.0.66`. Factory image is in the same release directory as the
|
||||
sysupgrade one.
|
||||
|
||||
## Post-upgrade cleanup
|
||||
|
||||
- Delete `/tmp` copies (automatic on reboot)
|
||||
- Keep `~/code/perso/backup-archer-c7-20260819.tar.gz` until the new
|
||||
setup has survived a reboot + a few days
|
||||
- `~/code/perso/archer-c7-upgrade/` can go once the RNDIS drivers are
|
||||
installed and Wi-Fi is confirmed stable (or keep the apks in case of
|
||||
a re-flash)
|
||||
@@ -0,0 +1,150 @@
|
||||
# ks4 backup strategy (2026-08)
|
||||
|
||||
Audience: anyone with root on ks4 who needs to know **what is protected,
|
||||
where, by which tool, and how to check it**. Details live in the linked
|
||||
docs; this page is the map.
|
||||
|
||||
## Two tools, three legs
|
||||
|
||||
| leg | mechanism | protects |
|
||||
|---|---|---|
|
||||
| **local replication** | `incus copy --refresh` → pool `backup` on ks4's second disk (sdb) | instances, against losing the `data` pool |
|
||||
| **remote replication** | **nas** pulls the same replicas over WireGuard → pool `ks4backup` (on `tank`, direct SATA) | instances, against losing ks4 or the site |
|
||||
| **remote backup** | **restic** → S3 bucket `restic-data`: database dumps + selected filesystem trees | the data itself, versioned and encrypted, independent of every disk above (a nightly run walks 1.1 M files / 1.24 TiB in under two minutes) |
|
||||
|
||||
Only two tools are involved: `incus copy` (ZFS-incremental, native to
|
||||
the platform — a restore is `incus start`) and restic (dedup +
|
||||
encryption, local metadata cache, so a night costs only the churn —
|
||||
[ks4/restic-backup.md](ks4/restic-backup.md)).
|
||||
|
||||
The old backup server **ks2 is being retired** (decommission by
|
||||
Sep 30, 2026 — [ks2/plan.md](ks2/plan.md)); nothing is written to it
|
||||
any more.
|
||||
|
||||
**Status 2026-08-31**: local replication and the S3 backup leg are
|
||||
live, and the S3 leg has been **restore-tested** (a file tree came
|
||||
back identical to the live one; a database dump loaded into a scratch
|
||||
server with all its tables). The **nas pull leg waits for FTTH**
|
||||
(expected before end of September) — it moved off nuc on 2026-08-30,
|
||||
onto a host where the 4 TB disk is on direct SATA rather than a USB
|
||||
bridge that suspended the pool weekly
|
||||
([nas/README.md](nas/README.md), [ks2/nas-seed.md](ks2/nas-seed.md)).
|
||||
In the meantime instances have no *fresh* off-site copy: the ks2 push
|
||||
was deliberately not re-enabled (a 3-week-old replica set on a
|
||||
94 %-full pool that is about to be wiped), so off-site protection
|
||||
rests on `restic-data`, which holds the data, the databases and the
|
||||
incus configuration needed to rebuild. Backing up whole
|
||||
instance *images* to S3 was considered and left out — S3 holds the
|
||||
data, the databases and the incus configuration, which is what a
|
||||
rebuild needs.
|
||||
|
||||
## The map
|
||||
|
||||
```
|
||||
ks4 (OVH) off-site
|
||||
┌────────────────────────────────────────────┐
|
||||
│ live instances pool `data` (sda) │
|
||||
│ nextcloud, seafile, mail, git, ... │
|
||||
│ │ │
|
||||
│ │ 01:00 incus copy --refresh │ nas (home LAN, via WireGuard)
|
||||
│ ▼ │ ┌───────────────────────────┐
|
||||
│ replicas (stopped) pool `backup` (sdb) │────▶│ 05:00 incus copy (pull) │
|
||||
│ project `backup` ready to start │ │ ks4backup on `tank` (SATA)│
|
||||
│ │ └───────────────────────────┘
|
||||
│ ──────────────────────────────────────── │
|
||||
│ │ OVH Object Storage (S3, sbg)
|
||||
│ 05:00 restic-backup.sh │ ┌───────────────────────────┐
|
||||
│ • DB dumps (MariaDB/PostgreSQL, auto) │────▶│ bucket restic-data │
|
||||
│ • incus config + global DB dump │ │ everything needed to │
|
||||
│ • data trees (restic-paths) │ │ rebuild: dumps + data │
|
||||
│ │ └───────────────────────────┘
|
||||
│ Sun 14:00 restic-maintenance.sh │
|
||||
│ • prune + check (rotating full verify) │
|
||||
└────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
Two different kinds of protection, on purpose:
|
||||
|
||||
- **instances** (the running systems) are protected by *replication* —
|
||||
a ready-to-start copy on ks4's second disk and, after FTTH, on nas.
|
||||
Restoring one is `incus copy` + `incus start`.
|
||||
- **the data inside them** (files, databases, incus configuration) is
|
||||
protected by *backup* — encrypted, deduplicated, versioned on S3,
|
||||
independent of ks4 and of the disks. Restoring means recreating the
|
||||
container (its config is in the S3 dumps, its install steps are in
|
||||
this repo) and pouring the data back.
|
||||
|
||||
Backing up whole instance images to S3 as well was designed
|
||||
([ks4/restic-backup.md](ks4/restic-backup.md) §6) and shelved: it
|
||||
duplicates data already covered, and the replication legs already give
|
||||
instances two homes.
|
||||
|
||||
## Schedule (root crontab on ks4)
|
||||
|
||||
| when | what | log |
|
||||
|---|---|---|
|
||||
| 01:00 daily | `incus-copy.sh -p backup -s backup` — refresh all replicas onto sdb | `/var/log/incus-copy.log` |
|
||||
| 03:00 daily | instance snapshots (incus profile, 7-day expiry) — what keeps the refreshes incremental | `incus info <inst>` |
|
||||
| 05:00 daily | `restic-backup.sh` — dumps + data trees → S3 | `/var/log/restic-backup.log` |
|
||||
| 05:00 daily (**nas**) | nas pulls all ks4 replicas over WireGuard → `ks4backup` (after FTTH) | nas: `/var/log/incus-copy-ks4.log` |
|
||||
| Sun 14:00 | `restic-maintenance.sh` — prune (capped), structure check, 1/52 data verification | `/var/log/restic-maintenance.log` |
|
||||
| 1st of month 06:00 / quarterly 15th | seafile GC dry-run report / real GC ([ks4/seafile-gc.md](ks4/seafile-gc.md)) | `/var/log/seafile-gc.log` |
|
||||
|
||||
Retention on S3: 14 daily, 8 weekly, 6 monthly snapshots.
|
||||
|
||||
## "Did last night work?" — the 30-second check
|
||||
|
||||
```sh
|
||||
grep -aE "done \(rc=|FAILED" /var/log/incus-copy.log | tail -2
|
||||
grep -aE "snapshot .* saved|done \(rc=|failed" /var/log/restic-backup.log | tail -3
|
||||
```
|
||||
|
||||
`rc=0` everywhere = fine. Any `failed`/`FAILED` line names the culprit
|
||||
(instance, database or path). All scripts exit non-zero on any error,
|
||||
never silently.
|
||||
|
||||
## Restoring
|
||||
|
||||
- **A whole instance, fast (same box)**: `incus copy backup:<inst>`
|
||||
style — copy the replica from project `backup` back into `default`
|
||||
([ks4/incus-copy.md](ks4/incus-copy.md)); from nas the same via the
|
||||
incus remote.
|
||||
- **A file or directory** (any date within retention):
|
||||
`restic -r s3:s3.sbg.io.cloud.ovh.net/restic-data restore <snapshot> --target /backup/restore-x --include <path>`
|
||||
(never restore into `/tmp` — it is RAM). Env: `. /root/.restic-env`.
|
||||
- **A database**: restore the `.sql` dump from `restic-data`
|
||||
(`/backup/dumps/mariadb/<inst>/<db>.sql`, `grants.sql` for users), load
|
||||
it with `incus exec <inst> -- mariadb < dump.sql`.
|
||||
- **An instance when no replica survives** (worst case: ks4 and the
|
||||
replica sites are gone): recreate the container from its page in
|
||||
[ks4/](ks4/) (the doc *is* the install script), restore its data
|
||||
trees and database dumps from `restic-data`, and use the incus
|
||||
configuration captured nightly in the dump tree
|
||||
(`/backup/dumps/incus/incus-global-db.sql`) to check devices,
|
||||
profiles and addresses.
|
||||
|
||||
Secrets you need for any of this: `/root/.restic-passphrase` and the S3
|
||||
keys in `/root/.restic-env` — **both are in the password manager**;
|
||||
without the passphrase the S3 buckets are unreadable.
|
||||
|
||||
## Adding something to the backups
|
||||
|
||||
New instances are picked up automatically by the replica and
|
||||
instance legs (all instances, opt-out only). Databases are
|
||||
auto-discovered. Only **data trees** need one line in
|
||||
[`restic-paths`](https://git.lutran.fr/julien/scripts/src/branch/main/restic-paths) — see [new-container.md](new-container.md).
|
||||
|
||||
## History (reference only)
|
||||
|
||||
The road to the setup above, kept for the measurements rather than for
|
||||
operations: an rsync-based leg to ks2
|
||||
([ks2/plan.md](ks2/plan.md)) and a first S3 implementation with
|
||||
**plakar** ([ks4/plakar-s3-data.md](ks4/plakar-s3-data.md), plus a
|
||||
self-written incus connector,
|
||||
[ks4/plakar-incus-integration.md](ks4/plakar-incus-integration.md)).
|
||||
plakar was dropped because its nightly cost scaled with the size of
|
||||
the tree rather than with the churn
|
||||
([PlakarKorp/plakar#2338](https://github.com/PlakarKorp/plakar/issues/2338)).
|
||||
It is still installed on ks4 and its `plakar-data` bucket still exists
|
||||
— kept only so the upstream issue can be reproduced; nothing schedules
|
||||
it any more.
|
||||
@@ -0,0 +1,62 @@
|
||||
# ks2 decommission — final checklist
|
||||
|
||||
Status: **prepared — execute only when every gate in
|
||||
[plan.md](plan.md) is ticked.** Rental ends **Sep 30, 2026**; leave a
|
||||
few days of margin.
|
||||
|
||||
## Preconditions (the plan's release gates)
|
||||
|
||||
- [ ] restic S3 leg running nightly for ≥ a week, `done (rc=0)`,
|
||||
restore test passed ([restic-backup.md](../ks4/restic-backup.md) §7)
|
||||
- [ ] **nas** pull leg seeded and one instance test-restored
|
||||
([nas-seed.md](nas-seed.md))
|
||||
- [ ] local leg (sdb5) cron green in `/var/log/incus-copy.log`
|
||||
|
||||
## 1. Cut the last flows to ks2 (root on ks4)
|
||||
|
||||
```sh
|
||||
crontab -e # remove any remaining ks2 line (the 02:00 push, if it was
|
||||
# re-enabled; the 04:00 incus-backup.sh line is long gone)
|
||||
incus remote list # if a "ks2" remote is still defined:
|
||||
incus remote remove ks2
|
||||
# drop the rsyncbackup SSH key if it exists only for ks2:
|
||||
ls /root/.ssh/id_rsyncbackup* && rm -i /root/.ssh/id_rsyncbackup*
|
||||
```
|
||||
|
||||
## 2. Last look at ks2 before wiping (root on ks2)
|
||||
|
||||
```sh
|
||||
# anything in the rsync tree newer than the S3 cutover?
|
||||
find /backup/ns3061243 -newermt '2026-08-23' | head
|
||||
# pre-2023 dirs (catc, mythoughts, qcm, scripts): instances still
|
||||
# exist stopped on ks4 — spot-check nothing unique remains:
|
||||
ls /backup/ns3061243/{catc,mythoughts,qcm,scripts}
|
||||
```
|
||||
|
||||
Archive anything unique into the restic dump dir on ks4
|
||||
(`/backup/dumps`) — it then rides the nightly S3 snapshot for free.
|
||||
|
||||
## 3. Wipe and terminate
|
||||
|
||||
```sh
|
||||
# on ks2 — destroy the data (both pools hold copies of personal data):
|
||||
zpool destroy backup && zpool destroy data
|
||||
# optional: shred the OS disks via OVH rescue mode / reinstall
|
||||
```
|
||||
|
||||
Then in the OVH manager: cancel the ks2 (Kimsufi) server before the
|
||||
renewal date.
|
||||
|
||||
## 4. Post-mortem cleanup
|
||||
|
||||
- [ ] Update [doc/README.md](../README.md): remove ks2 from the ks4
|
||||
durability bullet and the index line for `ks2/`s "being
|
||||
decommissioned" wording (keep the folder as history)
|
||||
- [ ] Retire `scripts/incus-backup.sh` + `incus-backup.db` in the repo
|
||||
(superseded by [`restic-backup.sh`](https://git.lutran.fr/julien/scripts/src/branch/main/restic-backup.sh)) —
|
||||
keep for reference, note the retirement in their headers
|
||||
- [ ] `login` container cleanup: its **native** postgres still holds
|
||||
the pre-migration `outline` database (live Outline uses the
|
||||
dockerised `outline-postgres-1`) — `dropdb outline` and remove
|
||||
the native postgres package
|
||||
- [ ] Mark the ks2 line in `plan.md` done, date it
|
||||
+210
@@ -0,0 +1,210 @@
|
||||
# ks4 pull leg — seed after FTTH
|
||||
|
||||
Status: **built and seeding — 2026-09-16.** FTTH is up (gateway
|
||||
`192.168.0.1`, see [nuc/nuc-install.md](../nuc/nuc-install.md)); the
|
||||
tunnel, the incus remote and the cron are in place and the first pass is
|
||||
running. Remaining: let the seed finish, then the test-restore below.
|
||||
|
||||
⚠️ **Changed 2026-08-30: this leg lands on `nas`, not on nuc.** It was
|
||||
originally designed for nuc's USB pool `usb4t`, but that pool proved to
|
||||
be the least reliable device in the setup
|
||||
([nuc/usb4t-dropouts.md](../nuc/usb4t-dropouts.md)) — which is exactly
|
||||
what an off-site copy of ks4 must not be. The 4 TB disk moved to direct
|
||||
SATA on the new host `nas` (`192.168.0.4`,
|
||||
[nas/nas-install.md](../nas/nas-install.md)), and the target pool
|
||||
`ks4backup` moved with it.
|
||||
|
||||
Consequences versus the original plan:
|
||||
|
||||
- Target pool `ks4backup` is now backed by `tank/backup/ks4` on nas.
|
||||
- The **WireGuard tunnel moves too**: nas becomes peer `10.8.0.22`.
|
||||
`transmission-bt`, the only other user, moved to nas and carries its
|
||||
own in-container tunnel (`10.8.0.21`, unchanged — ks4 needs no edit
|
||||
for it).
|
||||
- ⚠️ **nuc's `wg-ks4` was already disabled on 2026-08-31**, *before* the
|
||||
seed, not after. The original plan retired it only once nas was
|
||||
seeded, on the assumption nuc could serve as a fallback target — it
|
||||
cannot: its `ks4backup` pool was deleted and its `data` pool is a
|
||||
512 GB SSD, far too small for the ~1.75 TiB replica set. Keeping a
|
||||
keepalive'd tunnel alive on a machine that is now powered off between
|
||||
uses bought nothing. `wg-quick@wg-ks4` is `disabled`, and the `ks4`
|
||||
incus remote was removed from nuc.
|
||||
`/etc/wireguard/wg-ks4.conf` and its key are **kept**, so it is one
|
||||
`systemctl enable --now wg-quick@wg-ks4` away if ever needed.
|
||||
- ks4's ufw rule is unchanged: traffic arrives masqueraded as the
|
||||
`wireguard` container (`192.168.1.18`) whichever peer sent it.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- [x] `tank` running on SATA for ≥ 7 days with zero pool suspensions —
|
||||
the gate that replaces "fix the USB enclosure".
|
||||
Verified 2026-09-16: `ONLINE`, scrub clean 2026-09-14, uptime 2 w 2 d,
|
||||
zero suspensions in the journal.
|
||||
- [x] FTTH up (the first pass moves ~1.75 TiB)
|
||||
|
||||
## Setup (root on nas)
|
||||
|
||||
```sh
|
||||
# 0. the nas host has no wireguard-tools — transmission-bt carries its own
|
||||
# tunnel *inside* the container, so the host never needed them
|
||||
apt-get install -y wireguard-tools
|
||||
|
||||
# 1. key on nas, then peer it on ks4's wireguard container
|
||||
umask 077; wg genkey > /etc/wireguard/wg-ks4.key
|
||||
wg pubkey < /etc/wireguard/wg-ks4.key # -> <nas-pubkey>
|
||||
# (run on ks4)
|
||||
incus exec wireguard -- wg set wg0 peer <nas-pubkey> allowed-ips 10.8.0.22/32
|
||||
incus exec wireguard -- wg-quick save wg0
|
||||
|
||||
# 2. tunnel on nas: /etc/wireguard/wg-ks4.conf, modelled on nuc's
|
||||
# Address = 10.8.0.22/24, peer pubkey TVs6d7…,
|
||||
# Endpoint = 193.70.35.17:51845,
|
||||
# AllowedIPs = 10.8.0.0/24, 192.168.1.1/32, keepalive 25
|
||||
systemctl enable --now wg-quick@wg-ks4
|
||||
|
||||
# 3. incus remote over the tunnel
|
||||
incus remote add ks4 https://192.168.1.1:8443 --accept-certificate --token '…'
|
||||
incus list ks4: | head # sanity: remote reachable
|
||||
```
|
||||
|
||||
⚠️ **nas needs a managed `incusbr0` of its own, or every copy fails
|
||||
instantly.** All 18 ks4 instances carry an *instance-level* `eth0`
|
||||
(`nictype: bridged`, `parent: incusbr0`, `ipv4.address: 192.168.1.x`).
|
||||
nas has no such bridge, so instance creation dies with:
|
||||
|
||||
```
|
||||
Device validation failed for "eth0": Cannot use manually specified
|
||||
ipv4.address when using unmanaged parent bridge
|
||||
```
|
||||
|
||||
Create a managed network of the same name — but **give it `.254`, never
|
||||
`.1`**: `192.168.1.1` must keep resolving over `wg-ks4` to ks4's incus
|
||||
API, and a local address always beats a route.
|
||||
|
||||
```sh
|
||||
incus network create incusbr0 \
|
||||
ipv4.address=192.168.1.254/24 ipv4.nat=false ipv6.address=none
|
||||
ip route get 192.168.1.1 # must still say: dev wg-ks4
|
||||
```
|
||||
|
||||
The bridge stays inert — replicas are never started here.
|
||||
|
||||
## Seed
|
||||
|
||||
⚠️ **`-p backup` is required**, exactly as for the nuc leg
|
||||
([nas/nas-install.md](../nas/nas-install.md) §9a) — and as the
|
||||
verification command below already assumed. Without it the 18 replicas
|
||||
land in `default` alongside nas's live instances.
|
||||
|
||||
Run it detached rather than in a shell that can drop — the first pass is
|
||||
long: **~1.42 TiB at ~14 MB/s ≈ 29 h** (see the bottleneck section below
|
||||
for why it is 14 MB/s and not more).
|
||||
|
||||
```sh
|
||||
systemd-run --unit=ks4-seed --collect \
|
||||
/bin/bash -c '/root/scripts/incus-copy.sh -r ks4 -s ks4backup -p backup \
|
||||
>> /var/log/incus-copy-ks4.log 2>&1'
|
||||
|
||||
systemctl is-active ks4-seed # progress:
|
||||
tail -f /var/log/incus-copy-ks4.log
|
||||
```
|
||||
|
||||
⚠️ **The seed suppresses the 04:00 `nasbackup` job while it runs.**
|
||||
`incus-copy.sh` takes a single `/run/lock/incus-copy.lock` for every
|
||||
shape, so any run starting while the seed holds it aborts with
|
||||
`another incus-copy run holds …`. Over a ~29 h seed that skips one or two
|
||||
nights of the nas-local copy — accepted; those replicas are small,
|
||||
same-host, and rebuildable. The 03:30 nuc push is unaffected (it runs on
|
||||
nuc, with nuc's own lock).
|
||||
|
||||
### Bottleneck: ks4's source disk, not the network
|
||||
|
||||
Measured 2026-09-16, because ~112 Mbit/s looked far too slow for a 2 Gbit/s
|
||||
FTTH line. **It is not the link, and not WireGuard.** Do not go looking for
|
||||
a network fix:
|
||||
|
||||
| Path | Measured |
|
||||
|---|---|
|
||||
| ks4 upload → internet | 609 Mbit/s |
|
||||
| ks4 download ← internet | 900 Mbit/s |
|
||||
| nas download ← OVH network | 670 Mbit/s |
|
||||
| ks4 → nas through `wg-ks4` | **~112 Mbit/s** |
|
||||
|
||||
WireGuard was ruled out too: `UdpRcvbufErrors` 0 on nas (so the default
|
||||
`net.core.rmem_max` of 208 KB is *not* dropping packets), wg-crypt
|
||||
kworkers ~2 %, both hosts ~85 % idle.
|
||||
|
||||
The limit is **`data` living on `sdb5`, a single 7200 rpm HGST 6 TB
|
||||
spinning disk**. During the send `iostat` showed sdb at **109 r/s /
|
||||
14 MB/s, ~131 KB average request, queue depth ~1.0** — the random-IOPS
|
||||
ceiling of one HDD walking a fragmented 1.42 TiB dataset. 14 MB/s is
|
||||
~112 Mbit/s on the wire, which is exactly the observed rate. The network
|
||||
is idle the whole time.
|
||||
|
||||
**Parallelism is the only real lever, and it is a disk-queue effect**, not
|
||||
a bandwidth one. Running a second instance copy alongside the seed:
|
||||
|
||||
| | sequential | + 1 parallel copy |
|
||||
|---|---|---|
|
||||
| sdb read | 14 MB/s | **21 MB/s** |
|
||||
| avg request size | 131 KB | **514 KB** |
|
||||
| tunnel | 113 Mbit/s | **153 Mbit/s** |
|
||||
|
||||
With two senders queued, ZFS issues larger, more sequential reads instead
|
||||
of seeking one request at a time. Three or four concurrent copies would
|
||||
plausibly reach 25–30 MB/s and roughly halve the seed.
|
||||
|
||||
**Decision 2026-09-16: keep it sequential.** The first pass is a one-off,
|
||||
later refreshes are ZFS-incremental and tiny, and `incus-copy.sh` is
|
||||
shared by all three legs — parallelising means either reworking the script
|
||||
or running copies outside its flock, i.e. two jobs contending for the same
|
||||
dataset. Not worth one overnight. If a full reseed is ever needed and the
|
||||
wall-clock matters, this is the knob; the disk is the floor either way.
|
||||
|
||||
Notes:
|
||||
|
||||
- First pass is a full send per instance; later refreshes are
|
||||
ZFS-incremental **as long as they run at least every
|
||||
`snapshots.expiry` (7 d on ks4)** — same caveat as
|
||||
[ks4's local leg](../ks4/local-backup-cron.md).
|
||||
- Replicas arrive stopped with `boot.autostart=false` (the script does
|
||||
this) — they must never come up on the LAN with ks4's proxy devices.
|
||||
|
||||
## Cron (after the seed)
|
||||
|
||||
Add to nas's root crontab, offset from the 03:30 nuc→nas push, the
|
||||
04:00 nas→nuc push and ks4's own 01:00/05:00 jobs:
|
||||
|
||||
```cron
|
||||
0 5 * * * /root/scripts/incus-copy.sh -r ks4 -s ks4backup -p backup >> /var/log/incus-copy-ks4.log 2>&1
|
||||
```
|
||||
|
||||
`/etc/logrotate.d/incus-copy` already lists `incus-copy-ks4.log`, so
|
||||
nothing to add there.
|
||||
|
||||
## Verification (release gate for ks2)
|
||||
|
||||
```sh
|
||||
incus list --project backup -c ns -f csv # all ks4 instances present
|
||||
# test-restore one instance: copy a replica to the local pool,
|
||||
# start it isolated, check the service answers, then delete it
|
||||
incus copy solar solar-restoretest -s incus
|
||||
incus start solar-restoretest && incus exec solar-restoretest -- systemctl is-system-running
|
||||
incus delete -f solar-restoretest
|
||||
```
|
||||
|
||||
Once verified, tick the nas gate in the [ks2 plan](plan.md).
|
||||
|
||||
Only one piece of nuc's retirement is still outstanding — dropping its
|
||||
now-unused peer on ks4. Harmless to leave (an unused peer costs nothing)
|
||||
and safe to do at any time, since nuc's tunnel is already down:
|
||||
|
||||
```sh
|
||||
# on ks4 — nuc's pubkey is 31Tlgloc…
|
||||
incus exec wireguard -- wg set wg0 peer 31TlglocNJyooDVAO8HWEC0lyCykhbaFIWVWFUCOrmQ= remove
|
||||
incus exec wireguard -- wg-quick save wg0
|
||||
```
|
||||
|
||||
Deleting `/etc/wireguard/wg-ks4.conf` + `.key` on nuc is deliberately
|
||||
**not** done: they cost nothing and regenerating keys would mean
|
||||
re-peering on ks4.
|
||||
+89
@@ -0,0 +1,89 @@
|
||||
# ks2 decommission — summary plan
|
||||
|
||||
Goal: release `ks2.lutran.fr` (`164.132.173.57`, SSH :2233, host
|
||||
`ns3247221`) once ks4 has a real 3-2-1 backup without it.
|
||||
**Deadline: rental ends Sep 30, 2026.**
|
||||
|
||||
## What ks2 does today (updated 2026-08-28)
|
||||
|
||||
**Nothing is written to ks2 any more.** Both feeds are retired: the
|
||||
`01:00` replica push (replaced by the local leg on ks4's sdb5 pool,
|
||||
[local-backup-cron.md](../ks4/local-backup-cron.md)) and the `04:00`
|
||||
`incus-backup.sh` rsync (replaced by restic → S3,
|
||||
[restic-backup.md](../ks4/restic-backup.md)).
|
||||
|
||||
What remains on the box is **cold history**: instance replicas on pool
|
||||
`data` (last refreshed 2026-08-09) and the rsync tree
|
||||
`/backup/ns3061243` on pool `backup` (last refreshed 2026-08-28),
|
||||
snapshotted daily there (`zfs-auto-snapshot.sh`, 2-month expiry).
|
||||
|
||||
⚠️ While the nas leg waits for FTTH, instances have no *fresh*
|
||||
off-site copy. The ks2 push was **deliberately not re-enabled**
|
||||
(2026-08-30): ks2's replicas are three weeks old, its `data` pool is
|
||||
94 % full, no common snapshot survives ks4's 7-day expiry, and the box
|
||||
is wiped within the month — so a full ~1.5 T re-send buys four weeks
|
||||
of freshness on hardware already scheduled for destruction. Off-site
|
||||
protection meanwhile rests on `restic-data` (data, databases and the
|
||||
incus configuration — enough to rebuild). (The leg moved from nuc to
|
||||
the new host `nas` on 2026-08-30 — see [nas-seed.md](nas-seed.md).)
|
||||
|
||||
## Inventory findings (2026-08-22)
|
||||
|
||||
- `biwiki` exists only on ks2, `spot` only on ks2 + ks4's `backup`
|
||||
project — **decision: both abandoned**, safe to delete from backups.
|
||||
- `incus-backup.db` is stale (lists `spot`, misses `livetrail`,
|
||||
`outline`, `login`, …) — forgotten-manifest drift is exactly what
|
||||
the replacement had to eliminate — restic's drivers back up *all*
|
||||
instances and auto-discover *all* databases, opt-out instead of
|
||||
opt-in.
|
||||
- `livetrail` (running on ks4) is **absent from ks4's local `backup`
|
||||
project** — the local leg was run manually once and never again;
|
||||
automating its cron fixes this.
|
||||
- Both ks4 backup crons were **commented out — no ks4 backup ran at
|
||||
all** (hence the stale ks2 rsync data: dirs `Jul 23`, incus dumps
|
||||
`Aug 9`). Fixed 2026-08-22 — see
|
||||
[local-backup-cron.md](../ks4/local-backup-cron.md).
|
||||
- `/backup/ns3061243` also holds pre-2023 dirs (`catc`, `mythoughts`,
|
||||
`qcm`) — those instances still exist stopped on ks4, so nothing
|
||||
unique expected there; spot-check before wiping.
|
||||
- Losing ks2 also loses its 2 months of rsync-snapshot history —
|
||||
acceptable: restic keeps 14 daily / 8 weekly / 6 monthly.
|
||||
|
||||
## Target architecture (3-2-1 for ks4)
|
||||
|
||||
| Leg | Mechanism | Status |
|
||||
|---|---|---|
|
||||
| local, 2nd disk | `incus-copy.sh -p backup -s backup` → sdb5 pool | **live** (01:00) |
|
||||
| off-site, S3 (data+DB) | **restic** → bucket `restic-data` ([restic-backup.md](../ks4/restic-backup.md)) | **live** (05:00) |
|
||||
| off-site, S3 (instances) | restic over `incus file mount` of the `backup`-project replicas | **shelved** 2026-08-28 (replication covers instances) |
|
||||
| off-site, nas | **nas** pulls `ks4:*` → pool `ks4backup` over WG ([nas-seed.md](nas-seed.md)) | waiting FTTH (< Sep 30); moved off nuc 2026-08-30 |
|
||||
| off-site, ks2 (interim) | `incus-copy.sh -d ks2 -m push` | **not re-enabled** 2026-08-30 — see above |
|
||||
|
||||
## Actions (backup work documented in [`../ks4/`](../ks4/), ks2-only tasks here)
|
||||
|
||||
1. ~~salvage biwiki/spot~~ — decided 2026-08-22: both abandoned,
|
||||
**deleted** from ks2 and ks4's `backup` project
|
||||
2. ~~[local-backup-cron.md](../ks4/local-backup-cron.md)~~ — **done
|
||||
2026-08-22**: leg 1 cron re-enabled, ks2 rsync kept as stopgap;
|
||||
`livetrail` verified present in the `backup` project
|
||||
3. ~~first S3 leg~~ → **restic** is the S3 tool
|
||||
([restic-backup.md](../ks4/restic-backup.md), live 2026-08-28; the
|
||||
predecessor's doc is kept as reference)
|
||||
4. ~~instance leg to S3~~ — **shelved 2026-08-28**: instances are
|
||||
protected by replication (sdb + nas/ks2), their data and configs by
|
||||
`restic-data`
|
||||
5. [nas-seed.md](nas-seed.md) — **prepared**; after FTTH: seed the nas
|
||||
pull leg, verify all instances, test-restore one
|
||||
6. [decommission.md](decommission.md) — **prepared**; cut flows,
|
||||
final diff of `/backup/ns3061243`, wipe pools, terminate at OVH
|
||||
|
||||
## Release gates — ks2 can be dropped only when
|
||||
|
||||
- [x] biwiki + spot consciously abandoned (2026-08-22)
|
||||
- [x] local leg cron running since 2026-08-22, **18/18 instances**
|
||||
replicated (verified 2026-08-28)
|
||||
- [ ] **nas** leg fully seeded **and** one instance test-restored
|
||||
- [x] restic S3 backups live (05:00) **and** restore drill passed
|
||||
2026-08-28: tree restored byte-identical to live, dump restored
|
||||
and loaded into a scratch MariaDB (12/12 tables)
|
||||
- [ ] ks4 crons pointing at ks2 disabled
|
||||
@@ -0,0 +1,63 @@
|
||||
# Prod: ks4
|
||||
|
||||
Incus host at OVH — public-facing self-hosted services.
|
||||
|
||||
- Debian 13, Xeon D-1521 / 32 GiB, 2× 6 TB (OS on mdraid RAID1,
|
||||
ZFS pool `data` on `sda5`) — setup & rebuild:
|
||||
[install.md](install.md)
|
||||
- SSH: `ssh -i id_rsa_claude -p 2233 julien@193.70.35.17`
|
||||
(`julien` has passwordless `sudo incus …`, including
|
||||
`sudo incus exec …`; the backup crons themselves run as root)
|
||||
- Instances are on a NAT bridge `incusbr0` (192.168.1.0/24); public
|
||||
ingress via Incus proxy devices. `gateway` (nginx) is the sole HTTP/S
|
||||
entry point and fans out to the app containers.
|
||||
- OS inventory: `incus list -c n,config:user.os,config:user.os-checked`
|
||||
— refreshed by `incus-container-upgrade.sh` (`-o` for metadata only).
|
||||
`image.description` shows the *creation* image (2019 for most), not
|
||||
what the container runs today.
|
||||
- Services: gateway, mail, nextcloud, seafile, git, freshrss, bitwarden,
|
||||
outline, login (SSO), wireguard, + more — full table in
|
||||
[install.md](install.md).
|
||||
- ⚠️ The ZFS `data` pool is single-disk (not mirrored); durability
|
||||
rests on nightly cron jobs — 01:00 `incus copy --refresh` of all
|
||||
instances to the local `backup` pool on sdb5, then the instance leg
|
||||
to S3; 05:00 restic (DB dumps + data trees) to S3; **nas** pulls the
|
||||
replicas over WireGuard (after FTTH). Full picture and restore procedures:
|
||||
**[backup-strategy.md](../backup-strategy.md)**
|
||||
([local-backup-cron.md](local-backup-cron.md),
|
||||
[incus-copy.md](incus-copy.md),
|
||||
[restic-backup.md](restic-backup.md)).
|
||||
|
||||
## Network flows (home ↔ ks4)
|
||||
|
||||
```
|
||||
nas — home LAN 192.168.0.4 ks4 — OVH 193.70.35.17
|
||||
+-----------------------------------+ +-------------------------------------+
|
||||
| | | |
|
||||
| host: wg-ks4 (10.8.0.22) | | [wireguard] 192.168.1.18 |
|
||||
| incus remote "ks4" ------+--WG-->| wg0 10.8.0.1/24, udp 51845 |
|
||||
| pull ks4:* -> pool ks4backup | udp | | masquerade -> eth0 |
|
||||
| on tank (SATA) [pending FTTH] | 51845 | | |
|
||||
| | | +-> incus API 192.168.1.1:8443 |
|
||||
| [transmission-bt] wg0 (10.8.0.21) | | | (ufw: only from .18) |
|
||||
| full tunnel 0.0.0.0/0 ------+--WG-->| | |
|
||||
| kill switch: no default route | udp | +-> WAN egress: torrents + |
|
||||
| downloads -> /export/media | 51845 | apt of transmission-bt |
|
||||
| (NFS-exported to nuc) | | exit as 193.70.35.17 |
|
||||
| | | |
|
||||
| 03:30 nuc pushes its instances | | 03:00 instance snapshots |
|
||||
| -> nucbackup on tank | | 01:00 incus-copy: all instances |
|
||||
| 04:00 nas replicates its own | | -> project backup, zpool sdb5 |
|
||||
| -> nasbackup on tank | | |
|
||||
| 05:00 pull ks4:* -> ks4backup | | 05:00 restic: DB dumps + data |
|
||||
| [pending FTTH] | | trees -> S3 (restic-data) |
|
||||
| | | Sun 14:00 restic maintenance |
|
||||
+-----------------------------------+ +-------------------------------------+
|
||||
phones/laptops: WG peers 10.8.0.2-3 reach 192.168.1.x through the same endpoint
|
||||
nuc (on-demand media box) mounts /export/media from nas over NFSv4
|
||||
```
|
||||
|
||||
Both tunnels initiate **from home** (NAT, dynamic IP) toward ks4's
|
||||
fixed endpoint; ks4's incus API is never exposed to the internet.
|
||||
The pull leg and its tunnel moved from nuc to nas on 2026-08-30
|
||||
([nas/README.md](../nas/README.md)).
|
||||
+38
-21
@@ -1,15 +1,15 @@
|
||||
# incus-copy — ks4 instance replication
|
||||
|
||||
Incremental replication of all ks4 Incus instances via
|
||||
[`scripts/incus-copy.sh`](../scripts/incus-copy.sh)
|
||||
[`scripts/incus-copy.sh`](https://git.lutran.fr/julien/scripts/src/branch/main/incus-copy.sh)
|
||||
(`incus copy --refresh`: ZFS-incremental, produces ready-to-start
|
||||
replicas instead of tarballs). Two legs replace the historical nightly
|
||||
push to `ks2` (decommissioning):
|
||||
|
||||
1. **local** — replicas + dumps on a dedicated `backup` zpool on ks4's
|
||||
second disk (`sdb5`), survives `sda` death
|
||||
2. **off-site** — replicas pulled by **nuc** into pool `ks4backup`
|
||||
(dataset `usb4t/backup/ks4`), survives losing ks4 entirely
|
||||
2. **off-site** — replicas pulled by **nas** into pool `ks4backup`
|
||||
(dataset `tank/backup/ks4`), survives losing ks4 entirely
|
||||
|
||||
## The script
|
||||
|
||||
@@ -35,7 +35,7 @@ incus-copy.sh -p <dest-project> -s <dest-pool> # local
|
||||
`--refresh-exclude-older` keeps expired snapshots from being re-sent.
|
||||
- Consistency: copies of running instances are crash-consistent;
|
||||
databases may need recovery on restore. Snapshot history mitigates,
|
||||
and [`incus-backup.sh`](../scripts/incus-backup.sh) (proper
|
||||
and [`incus-backup.sh`](https://git.lutran.fr/julien/scripts/src/branch/main/incus-backup.sh) (proper
|
||||
`mariadb-dump`s + selected paths, manifest-driven) covers the DBs.
|
||||
- **VMs**: `--refresh` re-sends a VM's **full block volume** every run
|
||||
unless source and replica share a common snapshot — containers diff
|
||||
@@ -63,7 +63,9 @@ Cron (root on ks4) — replaces both ks2 jobs:
|
||||
|
||||
```cron
|
||||
0 1 * * * /root/scripts/incus-copy.sh -p backup -s backup >> /var/log/incus-copy.log 2>&1
|
||||
0 4 * * * /root/scripts/incus-backup.sh -f /root/scripts/incus-backup.db -s data >> /var/log/incus-backup.log 2>&1
|
||||
# retired 2026-08-28 — dumps + data trees are restic's job (05:00),
|
||||
# see ../backup-strategy.md and restic-backup.md
|
||||
#0 4 * * * /root/scripts/incus-backup.sh -f /root/scripts/incus-backup.db -s data >> /var/log/incus-backup.log 2>&1
|
||||
```
|
||||
|
||||
(`incus-backup.sh` without `-d` now writes locally to
|
||||
@@ -72,24 +74,31 @@ Cron (root on ks4) — replaces both ks2 jobs:
|
||||
⚠️ Replicas in the `backup` project must stay **stopped** — they keep
|
||||
the live containers' static `192.168.1.x` addresses.
|
||||
|
||||
## Leg 2 — off-site pull from nuc
|
||||
## Leg 2 — off-site pull from nas (was nuc until 2026-08-30)
|
||||
|
||||
Storage pool on nuc (done 2026-08-09): `ks4backup`, backed by the
|
||||
dataset `usb4t/backup/ks4` on the USB 4 TB pool (quota on `usb4t/backup`
|
||||
removed 2026-08-09 — full replica set is ~1.75 TiB):
|
||||
Storage pool on **nas** (recreated 2026-08-30): `ks4backup`, backed by
|
||||
the dataset `tank/backup/ks4` — the same 4 TB disk, now on **direct
|
||||
SATA** instead of the USB enclosure whose bridge kept suspending the pool
|
||||
([nuc/usb4t-dropouts.md](../nuc/usb4t-dropouts.md)). Full replica set is
|
||||
~1.75 TiB.
|
||||
|
||||
```sh
|
||||
incus storage create ks4backup zfs source=usb4t/backup/ks4
|
||||
incus storage create ks4backup zfs source=tank/backup/ks4
|
||||
```
|
||||
|
||||
Replicas live only on the USB drive — if it fails, only backups are
|
||||
lost; nuc's own instances (pool `data` on the SSD) are unaffected.
|
||||
⚠️ Originally this lived on nuc as `usb4t/backup/ks4`. That pool no
|
||||
longer exists — the disk moved to nas and the pool was renamed on import
|
||||
([nas/nas-install.md](../nas/nas-install.md) §5b). Following the old
|
||||
command fails with "no such pool".
|
||||
|
||||
**Direction: nuc pulls, through the WireGuard tunnel.** Verified
|
||||
`tank` is a single vdev, so if the disk fails only backups are lost —
|
||||
nas's own instances live on the `incus` SSD mirror and are unaffected.
|
||||
|
||||
**Direction: nas pulls, through the WireGuard tunnel.** Verified
|
||||
2026-08-09: ks4's API listens on wildcard `:8443` (so it answers on
|
||||
`192.168.1.1`, the `incusbr0` host address) but is **firewalled from
|
||||
the internet** — the VPN path keeps it that way, needs no inbound port
|
||||
at home, and doesn't care that nuc's public IP is dynamic. The
|
||||
at home, and doesn't care that the home public IP is dynamic. The
|
||||
`wireguard` container (`192.168.1.18`, `wg0` `10.8.0.1/24`) is exposed
|
||||
via a proxy device on public UDP `51845`.
|
||||
|
||||
@@ -97,13 +106,16 @@ Setup (✅ **done 2026-08-09**, verified end-to-end with
|
||||
`incus list ks4:` from nuc):
|
||||
|
||||
- **wireguard container** (ks4): forwards + masquerades wg0→eth0
|
||||
(pre-existing); nuc added as peer `10.8.0.20/32`
|
||||
(`wg set wg0 peer 31Tlgloc… allowed-ips 10.8.0.20/32` +
|
||||
`wg-quick save wg0`).
|
||||
(pre-existing). **nas** is the peer for this leg —
|
||||
`wg set wg0 peer <nas-pubkey> allowed-ips 10.8.0.22/32` +
|
||||
`wg-quick save wg0`. (`10.8.0.20/32` was nuc's peer for the same leg
|
||||
and is retired once nas is seeded; `10.8.0.21` is transmission-bt's
|
||||
own in-container tunnel and is unrelated.)
|
||||
- **ufw** (ks4): `ufw allow in on incusbr0 from 192.168.1.18 to any
|
||||
port 8443 proto tcp` — the API stays firewalled from the internet
|
||||
and the connection arrives masqueraded as the WG container.
|
||||
- **nuc**: `/etc/wireguard/wg-ks4.conf` (`wg-quick@wg-ks4` enabled;
|
||||
- **nas**: `/etc/wireguard/wg-ks4.conf` (`wg-quick@wg-ks4` enabled,
|
||||
`Address = 10.8.0.22/32`;
|
||||
peer = container pubkey `TVs6d7…`, endpoint `193.70.35.17:51845`,
|
||||
`AllowedIPs = 10.8.0.0/24, 192.168.1.1/32`, keepalive 25s) and
|
||||
`incus remote add ks4 https://192.168.1.1:8443 --accept-certificate
|
||||
@@ -115,7 +127,8 @@ container over its own tunnel is fine (crash-consistent, tiny, no
|
||||
interruption); if the tunnel is down the cron job fails loudly instead
|
||||
of hanging.
|
||||
|
||||
Then cron (root on nuc) — stagger after ks4's local leg:
|
||||
Then cron (root on **nas**) — stagger after ks4's local leg and after
|
||||
nas's own 04:00 local replication:
|
||||
|
||||
```cron
|
||||
30 2 * * * /root/scripts/incus-copy.sh -r ks4 -s ks4backup >> /var/log/incus-copy.log 2>&1
|
||||
@@ -128,7 +141,7 @@ cron once it completes.
|
||||
## Cutover checklist (then kill ks2)
|
||||
|
||||
1. First full cycle of all three jobs clean (logs above).
|
||||
2. Restore test: on nuc, start a small replica (e.g. `freshrss`) with
|
||||
2. Restore test: on nas, start a small replica (e.g. `freshrss`) with
|
||||
its NIC detached, check app data, then stop it.
|
||||
3. Remove both ks2 cron lines on ks4, `incus remote remove ks2`,
|
||||
cancel the server (`164.132.173.57` = ks2, rsync target of the old
|
||||
@@ -137,11 +150,15 @@ cron once it completes.
|
||||
## Restore
|
||||
|
||||
```sh
|
||||
# from nuc (off-site replica):
|
||||
# from nas (off-site replica):
|
||||
incus copy <instance> ks4:<instance> --mode push
|
||||
# from the local backup project (sda replaced, pool data rebuilt):
|
||||
incus copy <instance> <instance> --project backup --target-project default -s data
|
||||
```
|
||||
|
||||
The step-by-step seed procedure, including the WireGuard move and the
|
||||
gate it depends on, is [ks2/nas-seed.md](../ks2/nas-seed.md) — that is
|
||||
the authoritative version for this leg.
|
||||
|
||||
Remember replicas have `boot.autostart=false`; re-enable after a real
|
||||
failover, and re-check it after copying back to ks4.
|
||||
|
||||
+23
-12
@@ -142,22 +142,33 @@ All in the `default` project, on the `default` profile
|
||||
|
||||
## Backups
|
||||
|
||||
> **Migration in progress** (2026-08-09): the `ks2` destination below is
|
||||
> being replaced by a local `backup` zpool on `sdb5` + off-site pull
|
||||
> from nuc — plan and setup in [incus-copy.md](incus-copy.md). The
|
||||
> cron blocks here describe what runs **today**; update on cutover.
|
||||
|
||||
Two **root** cron jobs (this repo's [`scripts/`](../scripts/) are the
|
||||
copies that run here) — the primary safety net for the non-mirrored
|
||||
`data` pool:
|
||||
Full picture, schedule and restore procedures:
|
||||
**[../backup-strategy.md](../backup-strategy.md)**. Drivers live in the
|
||||
[scripts repo](https://git.lutran.fr/julien/scripts) and are deployed
|
||||
to `/root/scripts`. Current **root** cron jobs — the safety net for the
|
||||
non-mirrored `data` pool:
|
||||
|
||||
```cron
|
||||
0 1 * * * /root/scripts/incus-copy.sh -d ks2 -m push
|
||||
0 4 * * * /root/scripts/incus-backup.sh -d 164.132.173.57 -u rsyncbackup \
|
||||
-i /root/.ssh/id_rsyncbackup -p 2233 \
|
||||
-f /root/scripts/incus-backup.db -s data
|
||||
0 1 * * * /root/scripts/incus-copy.sh -p backup -s backup >> /var/log/incus-copy.log 2>&1
|
||||
0 5 * * * /root/scripts/restic-backup.sh >> /var/log/restic-backup.log 2>&1
|
||||
0 14 * * 0 /root/scripts/restic-maintenance.sh >> /var/log/restic-maintenance.log 2>&1
|
||||
```
|
||||
|
||||
- **01:00 `incus-copy.sh`** → replicas of every instance in project
|
||||
`backup` on the sdb5 pool ([incus-copy.md](incus-copy.md),
|
||||
[local-backup-cron.md](local-backup-cron.md)); the S3 instance leg
|
||||
chains here once seeded.
|
||||
- **05:00 `restic-backup.sh`** → auto-discovered DB dumps + the data
|
||||
trees listed in
|
||||
[`scripts/restic-paths`](https://git.lutran.fr/julien/scripts/src/branch/main/restic-paths),
|
||||
into the `restic-data` bucket ([restic-backup.md](restic-backup.md)).
|
||||
- **Sun 14:00 `restic-maintenance.sh`** → prune + integrity checks.
|
||||
|
||||
A `ks2` interim replica push (`incus-copy.sh -d ks2 -m push`, 02:00)
|
||||
runs until the nuc pull leg is seeded. Retired 2026-08: the
|
||||
`incus-backup.sh` rsync leg and the first S3 implementation
|
||||
([plakar-s3-data.md](plakar-s3-data.md) — reference only).
|
||||
|
||||
- **01:00 — `incus-copy.sh` → `ks2`**: differential `incus copy … \
|
||||
--refresh --refresh-exclude-older --mode push` of every *running*
|
||||
instance to remote incus host `ks2` (a near-live replica). `ks2` must
|
||||
|
||||
@@ -0,0 +1,51 @@
|
||||
# local backup cron on ks4 (leg 1)
|
||||
|
||||
Enabled 2026-08-22 (both ks4 backup crons had been commented out — no
|
||||
backup of any kind was running). Overview of all legs:
|
||||
[../backup-strategy.md](../backup-strategy.md).
|
||||
|
||||
## What runs now (root crontab on ks4)
|
||||
|
||||
```cron
|
||||
0 1 * * * /root/scripts/incus-copy.sh -p backup -s backup >> /var/log/incus-copy.log 2>&1
|
||||
```
|
||||
|
||||
- `01:00` — **local leg**: `incus copy --refresh` of all instances
|
||||
into project `backup` on the sdb5 `backup` zpool (replaces the old
|
||||
push to ks2; see [incus-copy.md](incus-copy.md)). An S3 leg for the
|
||||
instance filesystems was designed and shelved
|
||||
([restic-backup.md](restic-backup.md) §6).
|
||||
- The rsync stopgap to ks2 (`incus-backup.sh`, 04:00) and the first
|
||||
S3 implementation (04:30) are both **retired**; S3 backups are
|
||||
restic's job at 05:00 ([restic-backup.md](restic-backup.md)).
|
||||
|
||||
## Cleanup done the same day
|
||||
|
||||
- ks2: `incus delete biwiki spot` (abandoned instances — freed space
|
||||
on the 95 %-full `data` pool)
|
||||
- ks4: `incus delete spot --project backup`
|
||||
|
||||
## Verification
|
||||
|
||||
```bash
|
||||
# all instances present, incl. livetrail (first copy is a full send)
|
||||
sudo incus list --project backup -c ns -f csv
|
||||
# cron runs land here (manual screen runs don't):
|
||||
sudo tail /var/log/incus-copy.log /var/log/incus-backup.log
|
||||
# rsync leg freshness, on ks2:
|
||||
ls -lat /backup/ns3061243/ | head
|
||||
```
|
||||
|
||||
Notes:
|
||||
|
||||
- `flock` in both scripts makes overlapping cron/manual runs abort
|
||||
safely instead of racing.
|
||||
- Replicas get `boot.autostart=false` set by the script — they must
|
||||
never fight the live instances for their static IPs.
|
||||
- Keep `snapshots.schedule` set on the sources so refreshes stay
|
||||
ZFS-incremental (a refresh without common snapshots falls back to
|
||||
rsync). Corollary: the cron must run at least every
|
||||
`snapshots.expiry` (7 d) — a longer gap expires every common
|
||||
snapshot and the next refresh silently degrades to a **full
|
||||
re-send** (observed 2026-08-22: seafile re-sent 933 G after the
|
||||
weeks-long gap while the crons were commented out).
|
||||
@@ -0,0 +1,185 @@
|
||||
# plakar incus integration — design notes (RETIRED)
|
||||
|
||||
> **Historical.** The importer works (smoke-tested against ks4's
|
||||
> `backup` project) but production moved to restic, which needs no
|
||||
> plugin: `incus file mount` exposes any instance — including stopped
|
||||
> replicas — over the same per-instance sftp API
|
||||
> ([restic-backup.md](restic-backup.md) §6). Code lives on at
|
||||
> <https://github.com/jlutran/plakar-integration-incus>; the analysis
|
||||
> below (per-file vs image, replica-vs-live consistency, the
|
||||
> seek-bound wall) is what shaped the restic design.
|
||||
|
||||
|
||||
Status: **importer scaffolded and compiling 2026-08-23** in
|
||||
`perso/plakar/integration-incus` (walker modeled on the official sftp
|
||||
integration; enumerate project → per-instance sftp walk → per-file
|
||||
records + synthetic `.incus.yaml`; allow/deny lists; VM sftp failures
|
||||
reported as record errors, not fatal). See its README for
|
||||
build/install/TODO. Untested against a live incus yet.
|
||||
|
||||
Goal: plakar source connector for Incus so instance backups land in a
|
||||
plakar kloset on S3 (leg 4 of the ks4 3-2-1 plan — see
|
||||
[plan.md](../ks2/plan.md)).
|
||||
|
||||
## Decision 1 — per-file importer, not tarball streaming
|
||||
|
||||
The proxmox integration streams one full vzdump archive per VM
|
||||
(`FLAG_STREAM`). For ks4 that model reads ~1.7 TB nightly (seafile
|
||||
933 G, nextcloud 725 G): the incus backup/export API produces **full
|
||||
tarballs only** — `shared/api/instance_backup.go` has just
|
||||
`instance_only` and `optimized_storage`, no incremental option.
|
||||
|
||||
Instead the importer emits **one record per file** with real stat info
|
||||
and a lazy reader. kloset's engine then does the incremental work for
|
||||
us: `snapshot/backup.go` (`checkVFSCache`) reuses the cached object
|
||||
when a path's stat matches the previous run and **never opens the
|
||||
file**. Nightly cost after the initial seed = stat-walk + changed
|
||||
files only; S3 receives only new chunks. Bonus: per-file browse and
|
||||
restore of any snapshot.
|
||||
|
||||
File access is native incus REST: `GET /1.0/instances/{name}/sftp`
|
||||
(a real SFTP session, also available per storage volume). Works over
|
||||
the local unix socket or remote :8443, on stopped containers (the
|
||||
daemon mounts the volume on demand), no ZFS assumptions.
|
||||
|
||||
## Decision 2 — do not reuse `incus copy --refresh` / migration API
|
||||
|
||||
Considered and rejected. `copy --refresh` is `POST /1.0/instances`
|
||||
with `source.refresh=true` (`shared/api/instance.go:373`) and is
|
||||
**not** zfs-only — transports are negotiated per storage driver
|
||||
(`MigrationTypes()`: zfs/btrfs/ceph native streams, universal RSYNC
|
||||
fallback, `driver_common.go:230`). But:
|
||||
|
||||
- the receiver must be another incus daemon speaking the migration
|
||||
websocket protocol (`internal/migration/migrate.proto`);
|
||||
- the optimized payload is an opaque `zfs send` stream — storing
|
||||
those in plakar means chain-of-increments restore onto a real
|
||||
zpool, no per-file browse, no pruning, one corrupt link breaks the
|
||||
chain;
|
||||
- the RSYNC transport needs a materialized previous copy on the
|
||||
receiver — a full-size local mirror, i.e. leg 1 rebuilt inside a
|
||||
plugin;
|
||||
- gotcha: `driver_zfs.go:801` — refresh **without** snapshot copying
|
||||
forces RSYNC even zfs↔zfs (keep `snapshots.schedule` on sources).
|
||||
|
||||
`copy --refresh` stays what legs 1–2 use (host↔host replication);
|
||||
plakar is content-addressed archiving to dumb storage. Complementary,
|
||||
not competing.
|
||||
|
||||
## Decision 3 — read the `backup` project replicas, not live instances
|
||||
|
||||
Reading a live rootfs is fuzzy. Leg 1 (`incus-copy.sh -p backup -s
|
||||
backup`, 01:00) already produces quiescent, crash-consistent stopped
|
||||
replicas — the importer targets those (`?project=backup`), scheduled
|
||||
after the copy completes. DB consistency remains the mariadb-dump
|
||||
job's responsibility (plakar `mysql` importer / dump step).
|
||||
|
||||
## Decision 4 — enumerate everything by default
|
||||
|
||||
`incus-backup.db` manifest drift (stale `spot`, missing `livetrail`,
|
||||
`outline`, `login`…) is the failure mode to kill: the importer backs
|
||||
up **all instances in the project by default**, opt-out via config,
|
||||
never opt-in.
|
||||
|
||||
## Other design points
|
||||
|
||||
- Location scheme `incus://<remote>/<project>/<instance>/...` —
|
||||
stable pathnames keep the kloset VFS cache effective.
|
||||
- Synthetic records per instance: config + profiles
|
||||
(`GET /1.0/instances/{name}`, expanded) so restore can recreate the
|
||||
instance before pushing files back.
|
||||
- Optional later `mode=image`: full tarball via the backups API for
|
||||
occasional exact-image restores (`incus import`); plakar dedups
|
||||
unchanged chunks between runs. Not the nightly path.
|
||||
- Restore path (exporter): create instance from stored config → push
|
||||
file tree back via sftp. Weaker than `incus import` but granular.
|
||||
|
||||
## Upstreaming (decided 2026-08-23: Route B — community integration)
|
||||
|
||||
Repo: `github.com/jlutran/plakar-integration-incus` (remote set,
|
||||
initial commit done). Path to publication, in order:
|
||||
|
||||
1. unit tests + first real run against ks4's `backup` project
|
||||
2. open a PlakarKorp/plakar issue proposing the integration
|
||||
(disclose the Apache-2.0 `lxc/incus` client dependency)
|
||||
3. push the repo, then PR a recipe against `PlakarKorp/hub` →
|
||||
installable as `plakar pkg add incus`
|
||||
4. (optional, later) adoption into the `integrations` monorepo:
|
||||
module rename to `github.com/PlakarKorp/integrations/incus`,
|
||||
orphan `integration/incus` branch, `incus/incus` layout
|
||||
|
||||
## Packaging & deployment
|
||||
|
||||
Plugins are standalone executables (gRPC over stdio) — no rebuild of
|
||||
plakar, fully compatible with the APT-installed binary on ks4:
|
||||
|
||||
```sh
|
||||
cd plakar/integration-incus && make # build importer/exporter
|
||||
plakar pkg create manifest.yaml v0.1.0 # → incus_v0.1.0_linux_amd64.ptar
|
||||
# on ks4, as root (plugins are per-user, cron runs as root):
|
||||
plakar pkg add ./incus_v0.1.0_linux_amd64.ptar
|
||||
plakar source add ks4-incus incus://... # then list it in /root/scripts/plakar-sources
|
||||
```
|
||||
|
||||
Installing a *local* `.ptar` file needs no `plakar login` (the plugin
|
||||
registry is only for fetching by name). Model the `manifest.yaml` on
|
||||
integration-proxmox (`tier: third-party`, `api_version: v1.1.0`).
|
||||
Compatibility contract = the go-kloset-sdk wire protocol: pin the SDK
|
||||
to the installed plakar line (v1.1.x), rebuild the ptar on a plakar
|
||||
major upgrade; a mismatch surfaces at backup time, so run a manual
|
||||
`plakar backup @ks4-incus` after plakar upgrades.
|
||||
|
||||
## The giant-instance problem (2026-08-24, seed attempt #1)
|
||||
|
||||
The per-file S3 seed of the whole `backup` project stalled at
|
||||
~1 MiB/s once it reached the big replicas: sdb at 96 % util doing
|
||||
~275 × 4 KiB reads/s — **cold random tiny-file reads on an HDD are
|
||||
seek-bound**, regardless of importer (the fs importer hit the same
|
||||
wall on the live pool; sftp changes nothing). At ~275 IOPS,
|
||||
nextcloud (725 G, ~500 k files) + seafile (933 G, millions of block
|
||||
files) need *weeks* to seed. The 16 small instances are unaffected
|
||||
(solar smoke test: 5 GiB in 2m38s).
|
||||
|
||||
Options considered for nextcloud/seafile:
|
||||
|
||||
1. **backups-API tarball (`mode=image`)** — sequential-ish but incus
|
||||
materializes the tarball server-side first (no room on `/`; would
|
||||
need `storage.backups_volume` on the backup pool) and the tar walk
|
||||
itself seeks like any tree walk. Double IO nightly. Weak.
|
||||
2. **`zfs send` of the replica dataset** — the only truly sequential
|
||||
read (disk block order, ~100+ MiB/s, no temp space): seed in
|
||||
hours. Full send nightly is ~4 h read for both giants; CDC chunking
|
||||
dedups unchanged stream regions. Restore = `zfs receive` +
|
||||
`incus admin recover` (document!). Backend-specific — acceptable as
|
||||
an opt-in mode for exactly these two.
|
||||
3. **Per-file + warm metadata** — after a seed, nightly walks are
|
||||
stat-only; keeping dnodes resident (`zfs_arc_min` ≈ 8 G) could make
|
||||
them fast. Doesn't solve the *seed*.
|
||||
4. **Exclude giants from this leg** — leaves seafile without any
|
||||
fresh off-site copy (data leg already excludes it). Not acceptable
|
||||
long-term.
|
||||
|
||||
Direction (pending decision): seed the 16 small instances per-file
|
||||
(`exclude_instances=nextcloud,seafile`), then implement option 2 as a
|
||||
`zfs+send://` mode or side-channel for the two giants, and revisit 3
|
||||
for nightly stat-walk speed.
|
||||
|
||||
## Open questions
|
||||
|
||||
- uid/gid view through the instance sftp endpoint (idmap handling for
|
||||
unprivileged containers) — verify what stat returns and what restore
|
||||
must remap.
|
||||
- xattrs / ACLs / device nodes over sftp — coverage and fidelity.
|
||||
- VMs: file access needs a running incus-agent — out of scope for ks4
|
||||
(containers only); document the limitation.
|
||||
- ~~Scheduling/locking~~ decided 2026-08-23: the S3 instance leg is
|
||||
only as fresh as the last **completed** local incus-copy, so both
|
||||
run chained in one cron entry —
|
||||
`incus-copy.sh -p backup -s backup ; <plakar incus leg>` — never at
|
||||
a fixed offset (a long refresh, e.g. a post-gap full re-send, would
|
||||
race it). `;` not `&&`: a partially failed copy leaves stale but
|
||||
internally consistent replicas, still worth uploading. Same rule
|
||||
for manual test runs: check `/var/log/incus-copy.log` completed
|
||||
first.
|
||||
- S3 hardening: scoped credentials (no delete), versioning/object
|
||||
lock, plakar client-side encryption passphrase storage.
|
||||
@@ -0,0 +1,247 @@
|
||||
# plakar data + DB backups on ks4 (leg 3) — SUPERSEDED
|
||||
|
||||
> **Historical.** This leg ran 2026-08-22 → 08-28 and was replaced by
|
||||
> restic: see **[restic-backup.md](restic-backup.md)**. plakar's
|
||||
> incremental cost scaled with tree size rather than churn
|
||||
> ([PlakarKorp/plakar#2338](https://github.com/PlakarKorp/plakar/issues/2338)).
|
||||
> The `plakar-data` kloset and the plakar install are deliberately kept
|
||||
> until that issue concludes (the report offers debug runs against that
|
||||
> repository); the `plakar-incus` bucket can be deleted. Kept here for
|
||||
> the measurements and the design rationale, which the restic doc
|
||||
> builds on.
|
||||
|
||||
|
||||
Status: **live since 2026-08-24** — S3 kloset
|
||||
(`lutran-ks4-plakar-data`) seeded (dumps + 9 fs sources), restore
|
||||
test passed (file tree + 10 GiB dumps snapshot), 04:30 nightly cron +
|
||||
Sunday `check -latest` installed. Replaces `incus-backup.sh` (its
|
||||
04:00 ks2 rsync cron is disabled). Open items: `seafile-fs` excluded
|
||||
(see §3 warning), `nextcloud-fs` costs 3–4 h/night (cost scales with
|
||||
tree size, not churn — reported upstream 2026-08-25:
|
||||
[PlakarKorp/plakar#2338](https://github.com/PlakarKorp/plakar/issues/2338);
|
||||
giants to move to a restic leg meanwhile).
|
||||
|
||||
Architecture: plakar runs **on ks4** (decision recorded in
|
||||
[plan.md](../ks2/plan.md)) and backs up **directly into an S3 kloset**
|
||||
(`plakar at @s3 backup @src`). A local staging kloset on sdb5 was
|
||||
considered and rejected: it would double-store ~1.5 T that the leg-1
|
||||
replicas already hold on the same disk, and local restore is already
|
||||
leg 1's job. Direct-to-S3 keeps the incremental behavior — the
|
||||
skip-unchanged baseline is the parent snapshot's VFS read from the
|
||||
kloset (plakar caches repository metadata locally; v1.1.4's dirpack
|
||||
prefetcher targets exactly this remote-backend walk). Driver script:
|
||||
[`scripts/plakar-backup.sh`](https://git.lutran.fr/julien/scripts/src/branch/main/plakar-backup.sh)
|
||||
(flock, incus DB dumps, iterates a sources file, prune + maintenance;
|
||||
its `-k` takes any `plakar at` target, so `-k @s3` — a throwaway
|
||||
local kloset, e.g. `-k /backup/plakar-test`, works for testing before
|
||||
credentials arrive and can be deleted after).
|
||||
|
||||
## 1. Install plakar via the official APT repo (root on ks4)
|
||||
|
||||
Repo verified reachable 2026-08-22 (suite `stable`, amd64/arm64);
|
||||
upgrades then ride the normal `apt upgrade` flow.
|
||||
|
||||
```sh
|
||||
curl -fsSL https://plakar.io/dist/keys/community-v1.1.0.gpg \
|
||||
| gpg --dearmor -o /usr/share/keyrings/plakar.gpg
|
||||
echo "deb [signed-by=/usr/share/keyrings/plakar.gpg] https://plakar.io/dist/repos/deb/ stable main" \
|
||||
> /etc/apt/sources.list.d/plakar.list
|
||||
apt update && apt install plakar
|
||||
plakar version # v1.1.4 at time of writing
|
||||
```
|
||||
|
||||
The mysql integration is a plugin; official plugins install from the
|
||||
plugin server after `plakar login`, or build locally (needs make + Go):
|
||||
|
||||
```sh
|
||||
plakar pkg add mysql # or: clone integrations, cd mysql && make install
|
||||
```
|
||||
|
||||
## 2. Kloset + passphrase
|
||||
|
||||
```sh
|
||||
head -c 32 /dev/urandom | base64 > /root/.plakar-passphrase
|
||||
chmod 600 /root/.plakar-passphrase
|
||||
# ⚠️ copy the passphrase to the password manager NOW — without it
|
||||
# the S3 kloset is unreadable garbage.
|
||||
export PLAKAR_PASSPHRASE=$(cat /root/.plakar-passphrase)
|
||||
|
||||
# when S3 credentials are available:
|
||||
plakar store add s3 s3://<endpoint>/<bucket> access_key=<AK> secret_access_key=<SK> use_tls=true
|
||||
plakar at @s3 create
|
||||
# meanwhile, a throwaway kloset lets everything below be tested now:
|
||||
plakar at /backup/plakar-test create
|
||||
```
|
||||
|
||||
## 3. Database dumps — `incus exec`, no DB users needed
|
||||
|
||||
The plakar mysql integration was considered and rejected: it connects
|
||||
over TCP, which would have required a bind-address change plus a
|
||||
dedicated backup user in every DB container. Instead the driver
|
||||
script keeps the old `incus-backup.sh` method — `incus exec <ct> --
|
||||
mariadb-dump --single-transaction ...` (container-local root over the
|
||||
unix socket) — with one upgrade: **databases are auto-discovered**,
|
||||
so there is no manifest to go stale (the failure mode that bit
|
||||
`incus-backup.db`). Three probes per *running* container:
|
||||
|
||||
- native MariaDB/MySQL (either binary naming — `mail` has a
|
||||
hand-installed `mariadb-dump` + mysql-named client; every
|
||||
non-system DB) → `/backup/plakar-dumps/mariadb/<ct>/<db>.sql`,
|
||||
plus a replayable users/grants dump (`SHOW GRANTS` per user, incl.
|
||||
password hashes) → `grants.sql` — so the S3 dump path is
|
||||
self-sufficient without the replicas, mirroring pg's `globals.sql`
|
||||
- native PostgreSQL (`pg_dump` per DB + `pg_dumpall --globals-only`
|
||||
for roles, peer auth as `postgres`) →
|
||||
`/backup/plakar-dumps/postgres/<ct>/`
|
||||
- PostgreSQL **inside docker** (covers `outline`/`login`): docker
|
||||
containers whose image name contains "postgres", dump user from
|
||||
`$POSTGRES_USER` → `/backup/plakar-dumps/postgres/<ct>/<docker-name>/`
|
||||
(a pg bundled in an image *not* named `*postgres*` would be missed —
|
||||
check the nightly log's `dump` lines after adding such a service)
|
||||
|
||||
Stopped containers (e.g. `databap`) are skipped —
|
||||
their static state is covered by the leg-1 replicas.
|
||||
|
||||
Opt-out: `/root/scripts/plakar-db-exclude` (optional) lists dump
|
||||
paths to skip, one per line, exactly as the `dump` log lines print
|
||||
them (`<ct>/<db>` or `<ct>/<docker-name>/<db>`). Every skip is logged
|
||||
(`SKIP <path>`), so unlike the old manifest the list can't rot
|
||||
silently. Applied 2026-08-22: `livetrail/geo` (10 GiB static map
|
||||
data, 3 h dump). Its last dump deliberately stays in the dump dir:
|
||||
unchanged → skipped by the VFS cache, so every nightly `dumps`
|
||||
snapshot keeps carrying it to S3 at zero cost. To refresh after a
|
||||
re-import: drop the exclude line for one night.
|
||||
|
||||
Dumps are staged **plain, not gzipped**: plakar chunks and compresses
|
||||
anyway, and CDC dedup across near-identical consecutive dumps only
|
||||
works on uncompressed input. The dump tree (incus's own DBs +
|
||||
MariaDB) is backed up as one snapshot tagged `dumps`. The nightly log
|
||||
lists every `dump <ct>/<db>` line — a database disappearing from the
|
||||
log is visible. (The installed mysql plugin is now unused; harmless
|
||||
to keep.)
|
||||
|
||||
Beyond databases, `outline`/`login` may hold non-DB state (uploaded
|
||||
files, docker volumes) — decide per instance whether an fs source is
|
||||
needed in §4.
|
||||
|
||||
⚠️ **seafile-fs excluded from the data leg (2026-08-24).** Its S3
|
||||
seed ran 12 h with no visible upload and was killed. Successive
|
||||
theories (EINTR livelock — `GODEBUG=asyncpreemptoff=1` changed
|
||||
nothing; kernel reclaim stall — PSI counters too low; ARC-starved
|
||||
lstats — disproved by `find` walking nextcloud's whole tree in 8m20
|
||||
cold / 19 s warm) all fell to evidence; cause **unknown**. Next
|
||||
diagnostic: bisect by seeding seafile-fs into a *local* scratch
|
||||
kloset (`plakar at /backup/plakar-scratch create` + timed backup) —
|
||||
completes fine → S3 store path implicated; crawls → importer-side on
|
||||
that tree shape (millions of tiny block files); then the
|
||||
maintainer-suggested `-trace=all` (writes to stdout). Until resolved,
|
||||
seafile's fresh off-site coverage waits on the incus leg reading the
|
||||
sdb replica.
|
||||
|
||||
Findings from the first dry-run (2026-08-22):
|
||||
|
||||
- `livetrail`'s `/root/.my.cnf` had a stale `database = spot` default
|
||||
that broke discovery — fixed 2026-08-22 (its DB is `livetrail`).
|
||||
- `login` still runs a **native** postgres holding the pre-migration
|
||||
`outline` DB (live Outline uses `outline-postgres-1`). Dumped
|
||||
harmlessly for now; cleanup once the S3 leg is verified:
|
||||
`dropdb outline` + remove the native postgres package.
|
||||
|
||||
## 4. Filesystem sources
|
||||
|
||||
Derived from the old `incus-backup.db` manifest (minus `spot`; DBs
|
||||
are handled by §3's auto-discovery). FS paths are read from the live
|
||||
containers' rootfs on the host — crash-consistent, same fidelity as
|
||||
the old rsync (the per-file
|
||||
[incus integration](plakar-incus-integration.md) supersedes this
|
||||
later). Verify the pool mount prefix first:
|
||||
`ls /var/lib/incus/storage-pools/data/containers/mail/rootfs/`.
|
||||
|
||||
`outline`/`login` non-DB state identified 2026-08-22 via
|
||||
`docker inspect` (Outline's local file storage volume; authentik's
|
||||
data + custom templates — `/opt/authentik` also catches the compose
|
||||
file). ⚠️ Still to review: whether `livetrail` holds state.
|
||||
|
||||
Register sources (root on ks4):
|
||||
|
||||
```sh
|
||||
R=/var/lib/incus/storage-pools/data/containers
|
||||
plakar source add bitwarden-fs $R/bitwarden/rootfs/opt/bitwarden
|
||||
plakar source add gateway-fs $R/gateway/rootfs/var/www
|
||||
plakar source add git-fs $R/git/rootfs/home/git/projects
|
||||
plakar source add login-fs $R/login/rootfs/opt/authentik
|
||||
plakar source add mail-vmail $R/mail/rootfs/var/vmail
|
||||
plakar source add mail-www $R/mail/rootfs/var/www
|
||||
plakar source add nextcloud-fs $R/nextcloud/rootfs/nextcloud
|
||||
plakar source add outline-fs $R/outline/rootfs/var/lib/docker/volumes/outline_storage-data/_data
|
||||
plakar source add seafile-fs $R/seafile/rootfs/opt/seafile
|
||||
plakar source add solar-fs $R/solar/rootfs/var/www/html/solar
|
||||
```
|
||||
|
||||
Then list the names (one per line) in `/root/scripts/plakar-sources` —
|
||||
the script iterates that file.
|
||||
|
||||
## 5. Script + cron
|
||||
|
||||
```sh
|
||||
# copy scripts/plakar-backup.sh from this repo to /root/scripts/, then:
|
||||
chmod +x /root/scripts/plakar-backup.sh
|
||||
/root/scripts/plakar-backup.sh -k /backup/plakar-test # dry-run the pipeline now
|
||||
/root/scripts/plakar-backup.sh -k @s3 # first real run (once creds exist)
|
||||
plakar at @s3 ls # inspect snapshots
|
||||
```
|
||||
|
||||
Cron (04:30, after the 04:00 stopgap so they don't hammer the same
|
||||
DBs simultaneously; move to 04:00 once the rsync stopgap is dropped).
|
||||
Inline `backup -check` was removed 2026-08-22: against S3 it re-reads
|
||||
every written chunk (~2.6 MB/s latency-bound; gateway-fs: 7 min
|
||||
backup, 23 min check — the giants would add 10–20 h nightly).
|
||||
Verification is a weekly deep check of the latest snapshots instead:
|
||||
|
||||
```cron
|
||||
30 4 * * * /root/scripts/plakar-backup.sh -k @s3 >> /var/log/plakar-backup.log 2>&1
|
||||
0 12 * * 0 PLAKAR_PASSPHRASE=$(cat /root/.plakar-passphrase) plakar -quiet at @s3 check -latest >> /var/log/plakar-check.log 2>&1
|
||||
```
|
||||
|
||||
Once validated against @s3, delete the throwaway kloset:
|
||||
`rm -rf /backup/plakar-test`.
|
||||
|
||||
## 6. S3 hardening
|
||||
|
||||
Scoped credentials (no delete permission if the provider allows it),
|
||||
bucket versioning or object lock, and prefer an OVH region — ks4 →
|
||||
OVH Object Storage stays on the OVH backbone, so backup and restore
|
||||
speed is LAN-like.
|
||||
|
||||
## 7. Restore test (release gate)
|
||||
|
||||
Two restores from `@s3`, exercising both data types:
|
||||
|
||||
```sh
|
||||
export PLAKAR_PASSPHRASE=$(cat /root/.plakar-passphrase)
|
||||
plakar at @s3 ls # expect 11 snapshots: dumps + 10 fs sources
|
||||
|
||||
# (a) file tree — restore solar-fs and diff against the live tree
|
||||
plakar at @s3 restore -to /backup/restore-test <solar-fs-snap-id>
|
||||
diff -r /backup/restore-test/var/lib/incus/storage-pools/data/containers/solar/rootfs/var/www/html/solar \
|
||||
/var/lib/incus/storage-pools/data/containers/solar/rootfs/var/www/html/solar
|
||||
rm -rf /backup/restore-test
|
||||
|
||||
# (b) database — restore a dump and load it into a scratch container
|
||||
plakar at @s3 restore -to /backup/restore-db <dumps-snap-id>
|
||||
incus launch images:ubuntu/24.04 dbtest
|
||||
incus exec dbtest -- apt-get install -y mariadb-server
|
||||
incus file push /backup/restore-db/backup/plakar-dumps/mariadb/freshrss/freshrss.sql dbtest/root/
|
||||
incus exec dbtest -- sh -c 'mariadb < /root/freshrss.sql' # dump embeds CREATE DATABASE
|
||||
incus exec dbtest -- mariadb -e 'SELECT COUNT(*) FROM information_schema.tables WHERE table_schema="freshrss"'
|
||||
incus delete -f dbtest && rm -rf /backup/restore-db
|
||||
```
|
||||
|
||||
Pass = diff empty (modulo files changed since the snapshot) and the
|
||||
scratch DB loads with a sane table count.
|
||||
|
||||
## Retirement
|
||||
|
||||
Once this leg is verified (and once more from @s3): remove the 04:00
|
||||
`incus-backup.sh` cron to ks2 — precondition for
|
||||
[decommissioning ks2](../ks2/plan.md).
|
||||
@@ -0,0 +1,311 @@
|
||||
# restic backups on ks4 — the S3 leg
|
||||
|
||||
Status: **live since 2026-08-28.** This is the S3 backup leg of the
|
||||
[strategy](../backup-strategy.md): database dumps + selected
|
||||
filesystem trees into the `restic-data` bucket, nightly at 05:00.
|
||||
Instances themselves are handled by replication (sdb, nuc after FTTH),
|
||||
not by this leg — an instance-filesystem leg (`restic-incus`, §6) was
|
||||
written and **shelved**; its script and empty repo stay in place
|
||||
should that change.
|
||||
|
||||
Why restic: its comparison state lives in a **local cache**, so a
|
||||
night costs a local stat-walk plus the churn — the rsync cost model,
|
||||
with dedup, compression and encryption on top. (The predecessor and
|
||||
the measurements that motivated the switch are in
|
||||
[plakar-s3-data.md](plakar-s3-data.md) — reference only.)
|
||||
|
||||
## 1. Repositories — two buckets, two drivers (decided 2026-08-25)
|
||||
|
||||
| bucket = repo | contents | driver script |
|
||||
|---|---|---|
|
||||
| `restic-data` | SQL dumps + **all** fs sources (8 small + nextcloud + seafile) | `restic-backup.sh` |
|
||||
| `restic-incus` | *(shelved — empty repo, see §6)* the `backup`-project replicas, per-file via mount | `restic-incus-backup.sh` |
|
||||
|
||||
Two repos so each can be pruned and checked independently. Accepted
|
||||
trade-off: `restic-data` carries the giants' blobs (~3–5 M once
|
||||
seafile is in), so **every operation on it loads a ~1.5–2 GiB
|
||||
index** — mitigated by running the whole nightly as a *single*
|
||||
`restic backup` invocation (one index load, one snapshot covering
|
||||
all paths), `GOGC=20` in reserve, rustic as the escape hatch.
|
||||
Buckets (not prefixes) for per-bucket OVH metrics and independent
|
||||
versioning/lock decisions later.
|
||||
|
||||
Passphrase: `/root/.restic-passphrase` (mode 600), one for all repos,
|
||||
**copy to the password manager immediately**. S3 creds + env in
|
||||
`/root/.restic-env` (mode 600).
|
||||
|
||||
## 2. Option analysis (our numbers, not defaults-worship)
|
||||
|
||||
### `--pack-size` (default 16 MiB, max 128)
|
||||
|
||||
Packs are the S3 objects; blobs (~1 MiB avg for data) live inside
|
||||
them. Pack size does **not** change index RAM (that scales with blob
|
||||
count) — it changes object count, request count, and rewrite
|
||||
amplification:
|
||||
|
||||
| pack-size | S3 objects for ~1.2 T stored | trade-off |
|
||||
|---|---|---|
|
||||
| 16 MiB | ~75 000 | slow `check`/listing, more requests; prune rewrites are fine-grained |
|
||||
| **64 MiB (chosen)** | ~19 000 | 4× fewer objects/requests; assembly RAM ≈ pack×connections ≈ 64 M × 5 ≈ 320 MiB — fine |
|
||||
| 128 MiB | ~9 500 | fewer still, but prune rewrite amplification doubles and upload buffers grow |
|
||||
|
||||
`--pack-size 64` on **every backup/prune invocation** (not a repo
|
||||
property).
|
||||
|
||||
### Memory (the real constraint on this 23/31 GiB-used box)
|
||||
|
||||
- Index RAM ≈ 200–300 B/blob → combined `restic-data` index
|
||||
~1 GiB per operation; prune peaks 2–3×. The seafile side of that
|
||||
estimate collapsed after its garbage collection (24.9 M → 0.5 M
|
||||
objects, [seafile-gc.md](seafile-gc.md)). Measure peak RSS with
|
||||
`/usr/bin/time -v` if the box ever feels tight.
|
||||
- `GOGC=20` trades CPU for a smaller Go heap — enable if the seafile
|
||||
prune ever pressures the box.
|
||||
|
||||
### Cache location
|
||||
|
||||
`~/.cache/restic` would land on `/` (14 G free) and can reach a few
|
||||
GiB at this scale → `--cache-dir /backup/restic-cache` (sdb pool;
|
||||
cache reads are ARC-warm after first touch). tmpfs was considered and
|
||||
rejected: the cache exists to avoid *network* metadata re-fetches;
|
||||
tmpfs loses it at reboot and eats the RAM the ARC needs.
|
||||
|
||||
### Concurrency & compression
|
||||
|
||||
- `--read-concurrency 8` (default 2): more outstanding reads lets the
|
||||
HDD elevator help on seek-bound trees (our 275-IOPS wall).
|
||||
- `-o s3.connections=8` (default 5): mild upload parallelism bump.
|
||||
- Compression: repo-v2 default `auto` is right.
|
||||
|
||||
### Excludes — the cheapest optimisation of all
|
||||
|
||||
Nextcloud's `appdata_*/preview` (and `dav-photocache`) are
|
||||
regenerable caches holding a large fraction of the 500k files:
|
||||
|
||||
```
|
||||
# /root/scripts/restic-exclude
|
||||
/var/lib/incus/storage-pools/data/containers/nextcloud/rootfs/nextcloud/data/appdata_*/preview
|
||||
/var/lib/incus/storage-pools/data/containers/nextcloud/rootfs/nextcloud/data/appdata_*/dav-photocache
|
||||
```
|
||||
|
||||
Verify before the seed: `find .../data/appdata_* -path '*/preview/*' | wc -l`
|
||||
for the win size, then `restic backup --dry-run -vv --exclude-file …`
|
||||
prints every decision without uploading. Per-user `<user>/cache/`
|
||||
dirs are a further candidate. Seafile: no excludes — the block store
|
||||
*is* the data.
|
||||
|
||||
### Escape hatch: rustic
|
||||
|
||||
[rustic](https://github.com/rustic-rs/rustic) speaks the same
|
||||
repository format — switching later is a binary swap on the same
|
||||
repos (decided 2026-08-25: restic first, rustic if performance
|
||||
issues rise; manual upgrades accepted). It would bring lock-free
|
||||
prune and a lower RAM footprint. Don't run both concurrently on one
|
||||
repo (rustic ignores restic's locks); a clean switchover is fine.
|
||||
|
||||
## 3. Install + init (root on ks4)
|
||||
|
||||
Debian 13 ships restic **0.18.0**, which misses fixes we specifically
|
||||
need (checked 2026-08-25 against upstream changelogs): the 0.18.1
|
||||
crash fix for directories deleted mid-walk (#5421 — our live trees do
|
||||
that nightly), 0.19.0's much faster index loading (#5713 — the merged
|
||||
restic-data repo loads its index every op) and lower check memory
|
||||
(#5610), index-repair robustness (#21827/28), clean exit codes
|
||||
(#4467/#5363 — our drivers key off rc), and 0.19.1's
|
||||
skip-inaccessible-source fix (#5667 — an unmounted source must not
|
||||
commit an empty "successful" snapshot). → install upstream **0.19.1**;
|
||||
future upgrades via `restic self-update` (signature-verified).
|
||||
|
||||
```sh
|
||||
apt install sshfs bzip2 # sshfs: needed by `incus file mount` (§6)
|
||||
curl -LO https://github.com/restic/restic/releases/download/v0.19.1/restic_0.19.1_linux_amd64.bz2
|
||||
bunzip2 restic_0.19.1_linux_amd64.bz2
|
||||
install -m755 restic_0.19.1_linux_amd64 /usr/local/bin/restic
|
||||
restic version
|
||||
head -c 32 /dev/urandom | base64 > /root/.restic-passphrase && chmod 600 /root/.restic-passphrase
|
||||
# ⚠️ password manager, NOW.
|
||||
|
||||
cat > /root/.restic-env <<'EOF'
|
||||
export AWS_ACCESS_KEY_ID=<AK>
|
||||
export AWS_SECRET_ACCESS_KEY=<SK>
|
||||
export RESTIC_PASSWORD_FILE=/root/.restic-passphrase
|
||||
export RESTIC_CACHE_DIR=/backup/restic-cache
|
||||
EOF
|
||||
chmod 600 /root/.restic-env && . /root/.restic-env && mkdir -p /backup/restic-cache
|
||||
|
||||
for r in restic-data restic-incus; do
|
||||
restic -r s3:s3.sbg.io.cloud.ovh.net/$r init
|
||||
done
|
||||
```
|
||||
|
||||
## 4. Seed plan (executed 2026-08-25 → 08-28)
|
||||
|
||||
1. **ARC floor first** (helps every walk on the box):
|
||||
```sh
|
||||
echo 8589934592 > /sys/module/zfs/parameters/zfs_arc_min
|
||||
echo "options zfs zfs_arc_min=8589934592" > /etc/modprobe.d/zfs-arc.conf
|
||||
update-initramfs -u
|
||||
awk '/^c_min/ {printf "%.1f GiB\n", $3/1073741824}' /proc/spl/kstat/zfs/arcstats
|
||||
```
|
||||
⚠️ the floor is defended even under pressure — real headroom is
|
||||
~31 G − services − 8 G; revisit before adding a fat service.
|
||||
2. `restic-data`, staged (all into the same repo; separate
|
||||
invocations so each stage is independently restartable —
|
||||
`restic-backup.sh -s` runs the same thing):
|
||||
- dumps + 8 small sources (~50 G — an hour)
|
||||
- nextcloud path with the exclude file (proven ~22 MiB/s from the
|
||||
live pool → ~8 h, screen)
|
||||
- seafile — the one-time debt: 1–2 days, restartable at any point
|
||||
(committed packs dedup on retry)
|
||||
3. `restic-incus` (16 small replicas via §6 — a few hours).
|
||||
|
||||
## 5. Nightly driver — `scripts/restic-backup.sh`
|
||||
|
||||
Shape: flock, env file, loud logging. Phases:
|
||||
|
||||
1. **dumps** — incus DB
|
||||
dumps + auto-discovered MariaDB/PostgreSQL (native and docker)
|
||||
dumps into the dump dir (during the parallel week restic dumps
|
||||
into its own `/backup/dumps`; carry the archived `geo.sql` over
|
||||
once with `cp -a`).
|
||||
2. **one backup invocation** into `restic-data`: dump dir + all fs
|
||||
paths from `/root/scripts/restic-paths`
|
||||
(`--files-from-verbatim`), nextcloud excludes applied globally
|
||||
(patterns only match nextcloud paths). One index load, one
|
||||
snapshot per night.
|
||||
3. **retention** — `restic forget --group-by host --keep-daily 14
|
||||
--keep-weekly 8 --keep-monthly 6` (instant without `--prune`;
|
||||
`--group-by host` because seed-era snapshots have different path
|
||||
sets than nightly ones and must age in one group).
|
||||
|
||||
The instances leg (`restic-incus-backup.sh`, §6) is shelved — nothing
|
||||
chains onto the 01:00 line today.
|
||||
|
||||
```cron
|
||||
0 5 * * * /root/scripts/restic-backup.sh >> /var/log/restic-backup.log 2>&1
|
||||
```
|
||||
|
||||
⚠️ cron gotcha (bit us 2026-08-28): cron's PATH lacks `/usr/local/bin`,
|
||||
so the scripts call `/usr/local/bin/restic` by absolute path.
|
||||
|
||||
```cron
|
||||
```
|
||||
|
||||
Weekly maintenance (Sunday; prune takes an **exclusive lock** — never
|
||||
overlap the 05:00 run):
|
||||
|
||||
```cron
|
||||
0 14 * * 0 /root/scripts/restic-maintenance.sh >> /var/log/restic-maintenance.log 2>&1
|
||||
```
|
||||
|
||||
Per repo: `restic prune --max-unused 10% --max-repack-size 4G
|
||||
--pack-size 64` — our dead-data rate is ~0.1–0.4 GiB/day against
|
||||
~700 GiB repos (seafile append-mostly ≈ nothing dies), so 10 %
|
||||
tolerated slack lets prune skip repacking for months, and the 4 G cap
|
||||
bounds any single Sunday to ~10–15 min even after a mass deletion.
|
||||
Then `restic check` (structure, cheap) + `check
|
||||
--read-data-subset=1/52` rotating — full data verification of every
|
||||
byte once a year, ~25 GiB read per week.
|
||||
|
||||
## 6. Instances without a plugin — `incus file mount` (SHELVED)
|
||||
|
||||
> Decided 2026-08-28: **not deployed.** Instances are replicated (sdb,
|
||||
> nuc after FTTH, ks2 in the meantime) and everything needed to
|
||||
> rebuild one is in `restic-data`, so a third copy of their
|
||||
> filesystems was judged not worth the nightly moving parts. The
|
||||
> design and the script remain, ready if that changes.
|
||||
> If the `restic-incus` bucket is ever deleted, drop it from `REPOS`
|
||||
> in `restic-maintenance.sh` too.
|
||||
|
||||
Neither restic nor rustic accepts source plugins, and forking the
|
||||
backup tool is the wrong place to carry a patch. Instead, incus
|
||||
exposes any instance (including **stopped replicas**) as a FUSE mount
|
||||
over the per-instance sftp API:
|
||||
|
||||
```sh
|
||||
for inst in $(incus list --project backup -c n -f csv); do
|
||||
case ",$EXCLUDE_INSTANCES," in *",$inst,"*) continue;; esac # nextcloud,seafile
|
||||
mnt=/run/restic-incus/$inst # per-instance mountpoint: restic
|
||||
mkdir -p "$mnt" # picks parents by host+path, so
|
||||
incus config show "$inst" --project backup --expanded > "$mnt.yaml"
|
||||
incus file mount "$inst/" "$mnt" --project backup & # foreground proc
|
||||
MPID=$!; sleep 2
|
||||
restic -r s3:…/restic-incus backup --pack-size 64 --tag "$inst" "$mnt" "$mnt.yaml"
|
||||
kill $MPID; wait $MPID 2>/dev/null; umount "$mnt" 2>/dev/null
|
||||
done
|
||||
```
|
||||
|
||||
⚠️ **per-instance mountpoints are load-bearing**: a shared mountpoint
|
||||
would make every snapshot's parent the previous *other* instance's
|
||||
snapshot (restic matches parents by host+path) → nightly full
|
||||
re-reads. Distinct stable paths give correct parents and stat-skip.
|
||||
Mount lifecycle hardened in `restic-incus-backup.sh` (`incus file
|
||||
mount` runs foreground; needs `sshfs`). All instances by default,
|
||||
opt-out via a variable — the no-manifest-drift rule survives the tool
|
||||
change. Chained after the 01:00 incus-copy once validated, same
|
||||
freshness reasoning as before.
|
||||
|
||||
One shared repo would dedup the Ubuntu base across all replicas. Restore granularity: per-file; instance definitions ride
|
||||
the dump tree (`incus/<inst>.yaml` + `incus-global-db.sql`).
|
||||
|
||||
## 7. Restore drill (gate, then quarterly)
|
||||
|
||||
Scripted — [`restic-restore-test.sh`](https://git.lutran.fr/julien/scripts/src/branch/main/restic-restore-test.sh):
|
||||
|
||||
```sh
|
||||
/root/scripts/restic-restore-test.sh # tree + dump, ~2 min
|
||||
/root/scripts/restic-restore-test.sh -d # also loads the dump into a
|
||||
# throwaway container (strongest)
|
||||
```
|
||||
|
||||
It restores a filesystem tree from the latest snapshot and diffs it
|
||||
against the live one, restores a database dump and checks it is
|
||||
well-formed (`CREATE TABLE` count + `Dump completed` marker), then
|
||||
cleans up (`-k` keeps the files, `-w` changes the work dir). Never
|
||||
restore into `/tmp` — it is a 16 GiB tmpfs on ks4; the script writes
|
||||
under `/backup`.
|
||||
|
||||
A backup leg that has never been restored from is a hypothesis, not a
|
||||
backup: run this at the cutover gate and once a quarter after.
|
||||
|
||||
## 8. Cutover (done) + what is left
|
||||
|
||||
Legacy on the box: plakar is still installed and its `plakar-data`
|
||||
bucket still exists, kept only to reproduce
|
||||
[#2338](https://github.com/PlakarKorp/plakar/issues/2338) if upstream
|
||||
asks; no cron references it. The empty `plakar-incus` bucket can be
|
||||
deleted.
|
||||
|
||||
- **Cutover 2026-08-28**: the previous S3 leg's crons are off and
|
||||
restic's 05:00 run is the S3 backup. The formal parallel-week
|
||||
comparison was cut short — the evidence had already settled it
|
||||
(predecessor: ~800 MiB metadata re-read and 3–4 h per night on the
|
||||
nextcloud tree, churn-independent; restic: seed at 60–115 MiB/s,
|
||||
local-cache incrementals). Details in
|
||||
[plakar-s3-data.md](plakar-s3-data.md).
|
||||
- **Done 2026-08-28**: initial sync (1,114,212 files / 1.239 TiB in
|
||||
4h10m, only 2.4 GiB added thanks to dedup + the seafile GC), restore
|
||||
drill passed (§7), interim ks2 push re-enabled.
|
||||
- **Steady state, measured 2026-08-29** (first full nightly run):
|
||||
**1,114,343 files / 1.239 TiB walked in 1 min 43 s**, 201 MiB added
|
||||
(63 MiB stored), `rc=0`. For comparison, the previous S3 tool needed
|
||||
3–4 h per night for *one* of those trees. This is the local-cache
|
||||
cost model working as intended: the walk is stat-only, and only the
|
||||
churn is read, chunked and uploaded.
|
||||
- **First maintenance run 2026-08-30**: `prune` 0 blobs / 0 B removed,
|
||||
`check` and the rotating `--read-data-subset` slice both clean, 7 min,
|
||||
`rc=0`. Nothing to reclaim because the killed seed's packs were
|
||||
*adopted* by the successful seed's dedup rather than orphaned — worth
|
||||
knowing before assuming an interrupted backup wastes storage.
|
||||
- **Left**: the nuc leg after FTTH, then the ks2 decommission
|
||||
([../ks2/plan.md](../ks2/plan.md)). The instance leg (§6) is
|
||||
shelved, not pending.
|
||||
- **Old tool**: see the legacy note above; the `integration-incus`
|
||||
repo retires from production (optionally still published to
|
||||
PlakarKorp/hub as a community contribution).
|
||||
- **Interim ks2 push** (until the nuc leg seeds after FTTH, expected
|
||||
before end of September): re-enable the replica push so instances
|
||||
keep an off-site copy meanwhile:
|
||||
```cron
|
||||
0 2 * * * /root/scripts/incus-copy.sh -d ks2 -m push >> /var/log/incus-copy.log 2>&1
|
||||
```
|
||||
@@ -0,0 +1,152 @@
|
||||
# seafile: garbage collection + history retention
|
||||
|
||||
Status: **GC executed 2026-08-26/27.** Results:
|
||||
|
||||
| metric | before | after |
|
||||
|---|---|---|
|
||||
| dataset inodes | 24,885,504 | **515,842** (after Documents recreation + final `-r`, 2026-08-28) |
|
||||
| `storage/blocks/` | 867 G | **632 G** (−235 G) |
|
||||
| fs objects | ~19.3 M | ~100 k (19,207,400 removed) |
|
||||
| ghost library stores | 13 | 0 (blocks/fs eaten by GC; 4 empty commit husks removed manually — `--rm-deleted` couldn't see them, they predate `GarbageRepos`) |
|
||||
| wall time | | 16 min (blocks) + 0.2 s (`-r`) + **21 h 33 m** (`-R`) |
|
||||
|
||||
**Root-root-cause found**: one library — Documents
|
||||
(`c4ec7247`, 2.9 G data) — held **19.02 M fs objects and ~4.9 M
|
||||
commits**: a client has been committing at extreme frequency for
|
||||
years. The ~4.9 M commit objects survive GC (CE never deletes commit
|
||||
objects) and are most of the remaining inodes. Remedy: identify and
|
||||
fix the committing client, then **recreate the library** (sync down
|
||||
current state → new library → delete old → GC `-r`, which works for
|
||||
12.x-era deletions) — that retires the commit store entirely and
|
||||
takes the dataset to ~600 k inodes, at which point per-file backup of
|
||||
seafile becomes ordinary.
|
||||
|
||||
(The ks4 backup crons were paused during the GC window and are back
|
||||
on since 2026-08-27.)
|
||||
|
||||
## Root cause of the 24.9 M-inode / 933 G dataset
|
||||
|
||||
Investigated after per-file backups of `/opt/seafile` proved
|
||||
unwalkable (restic seed: ~20 M objects remaining at ~66 files/s).
|
||||
Findings (seafile-server **12.0.14**, binary install, upgraded from
|
||||
8.0 → 9.0 → 12.0):
|
||||
|
||||
- **GC has never run** — no `gc.log` ever, no cron in the container.
|
||||
`seaf-gc` is the *only* mechanism that deletes blocks/fs/commits
|
||||
objects; without it nothing is ever reclaimed.
|
||||
- **13 ghost libraries**: `storage/commits/` holds 30 library dirs;
|
||||
the DB has 17 live repos and an empty trash. Deleted libraries'
|
||||
object stores persist until `seaf-gc --rm-deleted`.
|
||||
- **Unlimited history on 16 of 17 repos**: only one library (the
|
||||
240 G one) has a retention limit (30 days, `RepoHistoryLimit`).
|
||||
Everything else keeps every version of every file since creation.
|
||||
- Sizes: live repo data ≈ 710 G (DB `RepoSize`) vs **867 G** in
|
||||
`blocks/` → ≥150 G dead blocks; the rest of the dataset is the
|
||||
fs/commits object swarm (~5 M commits objects counted; fs likely
|
||||
15 M+, sub-4K each) — the thing that makes every per-file walk
|
||||
take days.
|
||||
|
||||
References: <https://manual.seafile.com/12.0/maintain/seafile_gc/>
|
||||
|
||||
## Procedure
|
||||
|
||||
### 0. Safety net (host, instant)
|
||||
|
||||
```sh
|
||||
zfs snapshot data/containers/seafile@pre-gc # rollbackable; delete after validation
|
||||
```
|
||||
|
||||
Leg-1 replica on sdb is the second net (crontab currently disabled —
|
||||
the replica is pre-GC state, which is exactly what we want here).
|
||||
|
||||
### 1. Default history retention (inside the container)
|
||||
|
||||
`/opt/seafile/conf/seafile.conf`:
|
||||
|
||||
```ini
|
||||
[history]
|
||||
keep_days = 90
|
||||
```
|
||||
|
||||
Applies as the default for all libraries without a per-library
|
||||
setting (the existing per-library 30 d row stays). Users can still
|
||||
override per library in the web UI if
|
||||
`ENABLE_REPO_HISTORY_SETTING = True` in `seahub_settings.py`.
|
||||
|
||||
### 2. GC — maintenance window
|
||||
|
||||
⚠️ Superseded by experience: with the **MySQL backend seafile runs
|
||||
*online* GC** (the runs print "use online GC"), so stopping the
|
||||
services is not required — the recurring cron below does not. The
|
||||
2026-08-26 run stopped them anyway, out of caution. What follows is
|
||||
that run's procedure.
|
||||
|
||||
CE's GC traditionally requires seafile stopped. On this pathological store expect
|
||||
**hours** (it traverses fs/commits on a cold HDD); the dry-run sizes
|
||||
the window. Inside the container:
|
||||
|
||||
```sh
|
||||
cd /opt/seafile/seafile-server-latest
|
||||
./seahub.sh stop && ./seafile.sh stop # or the container's systemd units — check systemctl
|
||||
|
||||
./seaf-gc.sh --dry-run # per-repo report of what would be reclaimed
|
||||
./seaf-gc.sh # blocks GC, honors keep_days
|
||||
./seaf-gc.sh --rm-deleted # purge the 13 ghost libraries
|
||||
./seaf-gc.sh --rm-fs # unreferenced fs objects — the inode bulk
|
||||
# (verify flag names: ./seaf-gc.sh --help; manual 12.0)
|
||||
|
||||
./seafile.sh start && ./seahub.sh start
|
||||
```
|
||||
|
||||
### 3. Measure the result (host)
|
||||
|
||||
```sh
|
||||
df -i /var/lib/incus/storage-pools/data/containers/seafile # was 24,885,504 inodes
|
||||
du -sh .../seafile/rootfs/opt/seafile/seafile-data/storage/blocks # was 867 G
|
||||
```
|
||||
|
||||
### 4. Aftermath
|
||||
|
||||
- validate seafile works (client sync, web, download a versioned
|
||||
file), then `zfs destroy data/containers/seafile@pre-gc`
|
||||
- re-enable the crontab (incus-copy leg first; its next refresh
|
||||
transfers the shrunken dataset)
|
||||
- **recurring GC** (done 2026-08-27 design): the MySQL backend runs
|
||||
GC *online* (verified — the runs printed "use online GC" and `-r`
|
||||
ran with services up), so no stop/start is needed. Host crontab:
|
||||
```cron
|
||||
# monthly dry-run report (cheap health check, read-only)
|
||||
0 6 1 * * incus exec seafile -- sh -c "cd /opt/seafile/seafile-server-latest && ./seaf-gc.sh -D" >> /var/log/seafile-gc.log 2>&1
|
||||
# quarterly real GC: blocks + trashed repos + fs objects
|
||||
0 6 15 1,4,7,10 * incus exec seafile -- sh -c "cd /opt/seafile/seafile-server-latest && ./seaf-gc.sh && ./seaf-gc.sh -r && ./seaf-gc.sh -R" >> /var/log/seafile-gc.log 2>&1
|
||||
```
|
||||
At the post-cleanup scale (~120 k live files) the full sequence is
|
||||
minutes, not hours. Keep it clear of the 01:00/05:00 backup
|
||||
slots.
|
||||
- 2026-08-27 follow-ups executed: 8 old libraries deleted by user →
|
||||
trash emptied → `seaf-gc -r` purged their stores (18 m, blocks now
|
||||
590 G, −277 G total); 4 pre-`GarbageRepos` husks removed manually;
|
||||
Documents to be recreated from a local re-sync (retires its ~4.9 M
|
||||
fossil commit objects → expected final ~600 k inodes)
|
||||
- re-decide seafile's S3 backup with real post-GC numbers:
|
||||
per-file restic redux if the object count collapses far enough,
|
||||
`zfs send | restic --stdin-from-command` otherwise
|
||||
(see [restic-backup.md](restic-backup.md))
|
||||
- then plan the 13.x upgrade (Docker-only — see the incus-compose
|
||||
notes and
|
||||
<https://manual.seafile.com/13.0/upgrade/upgrade_notes_for_13.0.x/>)
|
||||
against a much smaller estate
|
||||
|
||||
## Related cleanup while in there
|
||||
|
||||
- `logs/seafevents.log.2024-*` and friends — years of rotated logs
|
||||
- the old `seafile-server-8.0.7` / `9.0.10` dirs once 12.x is proven
|
||||
|
||||
## Closure (2026-08-28)
|
||||
|
||||
Documents recreated from a local re-sync, old library trashed → trash
|
||||
emptied → `seaf-gc.sh -r` (78 m) retired its ~4.9 M fossil commit
|
||||
objects. **Final: 515,842 inodes (from 24,885,504, −98 %), blocks
|
||||
590 G (from 867 G), 8 libraries / ~120 k live files / 703 GB.** The
|
||||
`@pre-gc` snapshot can be destroyed once the new Documents library has
|
||||
proven itself for a few days (it pins the freed space until then).
|
||||
@@ -0,0 +1,109 @@
|
||||
# seafile — GCID table loss, 6-month CPU burn (fixed 2026-08-22)
|
||||
|
||||
`seaf-server` in the `seafile` container sat at 40–47 % CPU for months
|
||||
while appearing to do nothing: negligible client traffic on `eth0`
|
||||
(344 MB sent over 102 days) but 82 GB and 385 M packets over loopback,
|
||||
and ~11 **days** of accumulated CPU time.
|
||||
|
||||
## Root cause
|
||||
|
||||
The `seafile-db` MariaDB database was missing the `GCID` and
|
||||
`LastGCID` tables. Since Seafile 11 every client sync operation checks
|
||||
a per-repo GC id (`SELECT gc_id FROM GCID WHERE repo_id = ? FOR
|
||||
UPDATE`); with the table gone the query failed, the client retried,
|
||||
and the cycle repeated forever — no sync ever completed:
|
||||
|
||||
- 6 of seaf-server's fileserver worker threads spinning in pure
|
||||
user-space, ~3 new MySQL connections/s (167 loopback `TIME_WAIT`
|
||||
to :3306 at any moment),
|
||||
- `seafile.log` grew to **1.8 GB** of the single warning
|
||||
`Failed to prepare sql SELECT gc_id FROM GCID … Table
|
||||
'seafile-db.GCID' doesn't exist`, several times per second.
|
||||
|
||||
First occurrence **2026-03-01 22:17** — under 11.0.9, i.e. *before*
|
||||
both the 2026-03-16 restart and the 2026-05-11 upgrade to 12.0.14.
|
||||
`GCID` is an old core table (the 6.3.0 upgrade SQL already ALTERs
|
||||
it), so it existed and was lost. **What dropped it around 2026-03-01
|
||||
was never identified** — if tables vanish again, start there.
|
||||
|
||||
## Diagnosis trail (what worked without container root)
|
||||
|
||||
From the host as `julien` (sudo grant covers read-only `incus`
|
||||
subcommands only):
|
||||
|
||||
```sh
|
||||
sudo incus info seafile # PID 2410, 1005 GiB, 2 GiB RAM
|
||||
ps -eo pid,%cpu,args --sort=-%cpu # seaf-server pid 10217 top hog
|
||||
# per-thread CPU from /proc (world-readable, no root needed):
|
||||
for t in /proc/10217/task/*; do awk '{print $1, $14+$15}' $t/stat; done
|
||||
# utime vs stime split → pure user-space compute, ~0 syscalls
|
||||
# container netns TCP table via the process:
|
||||
awk 'NR>1 && $4=="06"' /proc/10217/net/tcp | wc -l # 167 TIME_WAIT → :3306
|
||||
```
|
||||
|
||||
Reading `/opt/seafile/logs/seafile.log` needed a temporary sudoers
|
||||
extension (`/usr/bin/incus exec *`, `/usr/bin/incus file pull *` in a
|
||||
drop-in — **removed again after the fix**). The log line then made the
|
||||
cause obvious.
|
||||
|
||||
## Fix
|
||||
|
||||
The exact DDL ships inside the server binary:
|
||||
|
||||
```sh
|
||||
strings /opt/seafile/seafile-server-12.0.14/seafile/bin/seaf-server \
|
||||
| grep 'CREATE TABLE IF NOT EXISTS.*GCID'
|
||||
```
|
||||
|
||||
Applied (root in the container; additive, no restart needed):
|
||||
|
||||
```sql
|
||||
CREATE TABLE IF NOT EXISTS GCID (
|
||||
id BIGINT NOT NULL PRIMARY KEY AUTO_INCREMENT,
|
||||
repo_id CHAR(36), gc_id CHAR(36),
|
||||
UNIQUE INDEX(repo_id)) ENGINE=INNODB;
|
||||
CREATE TABLE IF NOT EXISTS LastGCID (
|
||||
id BIGINT NOT NULL PRIMARY KEY AUTO_INCREMENT,
|
||||
repo_id CHAR(36), client_id VARCHAR(128), gc_id CHAR(36),
|
||||
UNIQUE INDEX(repo_id, client_id)) ENGINE=INNODB;
|
||||
```
|
||||
|
||||
The warning storm stopped the same second, seaf-server dropped to
|
||||
0 % CPU, host load fell from ~8–12 toward normal. `seafile.log` was
|
||||
truncated (`truncate -s 0`).
|
||||
|
||||
## Log rotation (added 2026-08-22)
|
||||
|
||||
Seafile ships **no rotation** — that's how one warning reached 1.8 GB
|
||||
unnoticed. `/etc/logrotate.d/seafile` in the container:
|
||||
|
||||
```
|
||||
/opt/seafile/logs/*.log {
|
||||
weekly
|
||||
maxsize 100M
|
||||
rotate 8
|
||||
missingok
|
||||
notifempty
|
||||
compress
|
||||
delaycompress
|
||||
copytruncate
|
||||
}
|
||||
```
|
||||
|
||||
`copytruncate` avoids pid/SIGUSR1 handling for the various daemons
|
||||
(seaf-server, seahub, seafevents…) at the cost of possibly losing a
|
||||
few lines at rotation — fine here. `maxsize 100M` caps a future spam
|
||||
storm at the next daily logrotate run instead of next week.
|
||||
|
||||
## Related cleanup / leftovers
|
||||
|
||||
- Legacy 4 AM cron on ks4 rsyncing seafile's rootfs to old **ks2**
|
||||
(`164.132.173.57`) removed 2026-08-22 — redundant with the
|
||||
[incus-copy](incus-copy.md) legs and it burned another ~47 % CPU
|
||||
per run.
|
||||
- Still missing from `seafile-db` (harmless, nothing queries them
|
||||
unless per-folder permissions are used): `FolderUserPerm`,
|
||||
`FolderGroupPerm`, `FolderPermTimestamp`. Same `strings`/upgrade-SQL
|
||||
trick provides the DDL if ever needed.
|
||||
- Host swap (1 GB) was 100 % full during the incident; unrelated but
|
||||
worth keeping an eye on.
|
||||
@@ -0,0 +1,64 @@
|
||||
# Storage: nas
|
||||
|
||||
Storage + backup host on the LAN, added 2026-08.
|
||||
|
||||
- **Always-on host.** nuc is now an on-demand media box (see
|
||||
[nuc/README.md](../nuc/README.md)), so everything that must stay up —
|
||||
LAN DNS, the HTTP proxy, torrents — lives here.
|
||||
- Debian 13, Supermicro **A1SAi-2750F** / Intel Atom C2750 (8 c, 20 W,
|
||||
ECC DDR3) — build procedure: [nas-install.md](nas-install.md)
|
||||
- SSH: `ssh -i id_rsa_claude root@192.168.0.4`
|
||||
- **No iGPU** (Avoton is headless; video is the AST2400 BMC). Anything
|
||||
needing hardware transcoding stays on nuc.
|
||||
- Pools:
|
||||
- `incus` — ZFS mirror across the last partition of both 120 GB SSDs;
|
||||
holds this host's container roots. OS itself is on mdraid RAID1 +
|
||||
ext4 across the same disks (rationale in
|
||||
[nas-install.md](nas-install.md) §2).
|
||||
- `tank` — the 4 TB WD Red, **on direct SATA**. This is the whole
|
||||
point of the box: the disk used to hang off a JMicron USB bridge on
|
||||
nuc that suspended the pool 61 times in 30 days
|
||||
([nuc/usb4t-dropouts.md](../nuc/usb4t-dropouts.md)). Single vdev,
|
||||
accepted — nothing on it is irreplaceable.
|
||||
- `tank/media` → `/export/media`: the media library. Exported
|
||||
**read-only over NFSv4 to nuc**, where `jellyfin-server` reads it;
|
||||
written locally only by `transmission-bt`.
|
||||
- Backups: nuc ↔ nas **cross-replication** (each host's instances live
|
||||
on the other), plus the ks4 pull leg —
|
||||
[nas-install.md](nas-install.md) §9,
|
||||
[ks2/nas-seed.md](../ks2/nas-seed.md).
|
||||
|
||||
## Instances
|
||||
|
||||
| Name | IP | Doc | Features |
|
||||
|---|---|---|---|
|
||||
| blocky | 192.168.0.254 | — | unprivileged, autostart; DNS ad-blocker for the LAN. Moved from nuc 2026-08-30 so it survives nuc being powered off |
|
||||
| privoxy | 192.168.0.11 | — | unprivileged, autostart; filtering HTTP proxy, listens on **:3128** (not privoxy's default 8118); static config in `/etc/systemd/network/eth0.network` (`Gateway=192.168.0.1`), `DNS=192.168.0.254`. Moved from nuc 2026-08-30 |
|
||||
| [transmission-bt](transmission-bt.md) | 192.168.0.7 | ✅ | unprivileged, autostart; always-on WireGuard full tunnel → ks4 (egress = 193.70.35.17, kill switch: no default route in `main`, wg-quick's `fwmark`/`suppress_prefixlength` rules send traffic to table 51820 — **`netplan apply` wipes those rules, so always `systemctl restart wg-quick@wg0` after it**); **IPv6 disabled** (`/etc/sysctl.d/99-no-ipv6.conf`) since the tunnel is `AllowedIPs = 0.0.0.0/0` only and the FTTH box's native IPv6 RA bypassed the kill switch entirely. Extending the tunnel to `::/0` is **not currently possible**: ks4 has a global v6 address and a default v6 route but **no working v6 egress** (verified 2026-09-16 — both ICMP and TCP to the v6 internet fail while v4 is fine), so it cannot act as a v6 exit. Fix OVH v6 on ks4 first if v6 peers are ever wanted; `/export/media` disk device (`shift=true`), downloads to `/media/downloads`; web UI :9091 (LAN only). Moved from nuc 2026-08-30 |
|
||||
|
||||
## Host tunnel
|
||||
|
||||
`wg-ks4` — `10.8.0.22/24`, peer = the `wireguard` container on ks4,
|
||||
endpoint `193.70.35.17:51845`, `AllowedIPs = 10.8.0.0/24, 192.168.1.1/32`.
|
||||
It exists only to reach ks4's incus API at `192.168.1.1:8443` for the
|
||||
05:00 pull. Key at `/etc/wireguard/wg-ks4.key`, unit
|
||||
`wg-quick@wg-ks4` (enabled).
|
||||
|
||||
⚠️ nas also runs a **managed `incusbr0` on `192.168.1.254/24`** — no
|
||||
uplink, nothing attached, inert. It exists purely so the ks4 replicas'
|
||||
instance-level `eth0` (`parent: incusbr0`, static `192.168.1.x`) passes
|
||||
validation on arrival. It must never take `192.168.1.1`: that address has
|
||||
to keep resolving over `wg-ks4`, and a local address beats a route.
|
||||
|
||||
## Backup pools hosted here
|
||||
|
||||
| incus pool | dataset | receives |
|
||||
|---|---|---|
|
||||
| `nucbackup` | `tank/backup/nuc` | nuc's instances (pushed nightly, 03:30) |
|
||||
| `ks4backup` | `tank/backup/ks4` | ks4's instances (pulled over `wg-ks4`, 05:00; built 2026-09-16, see [ks2/nas-seed.md](../ks2/nas-seed.md)) |
|
||||
| `nasbackup` | `tank/backup/nas` | **nas's own** instances (local copy, 04:00) |
|
||||
|
||||
nas's own instances are replicated **locally** rather than to nuc: nuc is
|
||||
an on-demand box and usually powered off, so it is not a usable backup
|
||||
target. `nasbackup` lives on `tank`, a different pool from the `incus`
|
||||
SSD mirror the instances run on.
|
||||
@@ -0,0 +1,858 @@
|
||||
# nas — build procedure
|
||||
|
||||
New storage host on the LAN (`192.168.0.4`), built 2026-08 from a
|
||||
Supermicro A1SAi-2750F. It exists to solve one specific problem:
|
||||
[usb4t-dropouts.md](../nuc/usb4t-dropouts.md) ruled out the cable and
|
||||
USB power management and left the **JMicron 152d:0578 bridge** as the
|
||||
cause of 61 disconnects in 30 days. The durable fix named there is a
|
||||
direct SATA connection — this box provides six of them.
|
||||
|
||||
Why it matters beyond the media library: `usb4t` is the intended home
|
||||
of the ks4 off-site replicas, and *"the pool holding the off-site copy
|
||||
of ks4 must not be the least reliable device in the setup"*. That makes
|
||||
this build a **release gate for the ks2 decommission**
|
||||
([ks2/plan.md](../ks2/plan.md), deadline Sep 30, 2026).
|
||||
|
||||
It also takes over `transmission-bt` from nuc, so nuc keeps only what
|
||||
needs its iGPU.
|
||||
|
||||
## Hardware
|
||||
|
||||
- Supermicro **A1SAi-2750F** mini-ITX, Intel Atom **C2750** (8 cores,
|
||||
2.4 GHz Silvermont, 20 W SoC), AES-NI, no AVX
|
||||
- RAM: **2× 4 GB DDR3-1600 ECC SO-DIMM fitted = 8 GB** (`Single-bit ECC`
|
||||
confirmed), in DIMMA1/DIMMB1; **2 slots free**. Board takes 32 GB
|
||||
officially, 64 GB with 16 GB modules.
|
||||
⚠️ 8 GB is modest for a 3.6 TB pool — ARC lands around 4 GB. Fine for
|
||||
streaming and replication (neither benefits much from cache), but the
|
||||
first thing to raise if metadata-heavy operations feel slow.
|
||||
- SATA: **2× SATA3 + 4× SATA2** (6 total)
|
||||
- BIOS **2.2** (2019-11-22) as shipped by the RMA
|
||||
- NIC: 4× GbE (Intel i354) + dedicated IPMI LAN
|
||||
- Video: **ASPEED AST2400 BMC only — there is no iGPU.** Avoton is a
|
||||
headless server SoC; `/dev/dri` is empty. That is why `jellyfin-server`
|
||||
stays on nuc (§8).
|
||||
|
||||
### ⚠️ AVR54 — already handled
|
||||
|
||||
The C2750 is on the list of Atom C2000 parts affected by Intel's
|
||||
**AVR54** erratum: the SoC's `LPC_CLKOUT0/1` signals degrade and stop,
|
||||
after which the board never boots again — typically after ~18 months of
|
||||
power-on, i.e. exactly an always-on duty cycle. **This board was RMA'd
|
||||
by Supermicro for that issue and replaced**, so it carries the fix
|
||||
(C0 stepping or the LPC pull-up rework). Recorded here so a dead C2000
|
||||
board is not re-diagnosed from scratch later.
|
||||
|
||||
### Disk plan
|
||||
|
||||
| Port | Device | Role |
|
||||
|---|---|---|
|
||||
| SATA3-0 | Intel `SSDSC2CT120A3` (120 GB) | md mirror + `incus` pool |
|
||||
| SATA3-1 | Toshiba `Q300.` (120 GB) | md mirror + `incus` pool |
|
||||
| SATA2-0 | WD Red 4 TB (moved off the USB enclosure) | pool `tank` |
|
||||
| SATA2-1/2/3 | free | a second 4 TB to mirror `tank`, later |
|
||||
|
||||
The SSDs get the SATA3 ports because they are the only devices that can
|
||||
use them: both negotiate 6 Gb/s and do ~450–500 MB/s, while the WD Red
|
||||
tops out near 180 MB/s and cannot saturate SATA2's ~270 MB/s. A future
|
||||
SLOG would also be fine on SATA2 — it is latency-bound on small sync
|
||||
writes, not bandwidth-bound.
|
||||
|
||||
⚠️ **Disconnect the 4 TB before partitioning the SSDs.** It carries the
|
||||
media library and nuc's replicas, nothing in the OS install needs it,
|
||||
and it keeps the two 120 GB disks unambiguous in the installer's list.
|
||||
Reconnect it before §5b.
|
||||
|
||||
Device letters shift with enumeration order and mean nothing — mdraid
|
||||
assembles from superblock UUIDs, ZFS imports by GUID, fstab and GRUB use
|
||||
UUIDs. **Identify disks by model and serial**
|
||||
(`lsblk -o NAME,SIZE,MODEL,SERIAL`), never by letter: both SSDs are
|
||||
120 GB, so size alone does not tell them apart.
|
||||
|
||||
SSD health measured 2026-08-30 (both read over the JMicron bridge on
|
||||
nuc, `smartctl -d sat`), before deployment:
|
||||
|
||||
| | Intel 330 (25 nm MLC) | Toshiba Q300 (15 nm TLC) |
|
||||
|---|---|---|
|
||||
| host writes | 16.0 TiB | 7.5 TiB |
|
||||
| endurance consumed | **0 %** (`Media_Wearout_Indicator` 100) | **7 %** (`Percentage Used Endurance Indicator`) |
|
||||
| power-on hours | **unreadable** — attr 9 decodes to 914,563 h on this family | 3,877 |
|
||||
| power cycles | 98 | 432 |
|
||||
| defects | 0 reallocated / program-fail / erase-fail | 0 reported uncorrectable |
|
||||
| SMART error log | **not supported** | supported, clean |
|
||||
| device statistics log | absent | full ACS-2 set |
|
||||
| short self-test | passed | passed |
|
||||
| interface CRC errors | n/a | 16 (baseline — watch for growth) |
|
||||
|
||||
Both are healthy and far from wear-out; at OS-disk write rates endurance
|
||||
is not the binding constraint for either. They were previously a
|
||||
**matched pair** — identical layouts, `bpool` / `rpool` / `ubuntu:0`
|
||||
labels from an Ubuntu ZFS-on-root mirror — which is why they go back
|
||||
into a mirror here.
|
||||
|
||||
⚠️ **The Intel is effectively unmonitorable**: no error log, no device
|
||||
statistics, no temperature, no usable hours counter. `smartd` can watch
|
||||
the Toshiba properly and can only ask the Intel "are you still there".
|
||||
Expect the Intel to be found dead rather than found degrading. That
|
||||
asymmetry is the reason for the mirror.
|
||||
|
||||
### `tank` is a single vdev — accepted
|
||||
|
||||
The 4 TB holds the media library, transmission's downloads and the ks4
|
||||
replicas, with no redundancy. A single vdev can *detect* corruption but
|
||||
only self-heal metadata, not data — as seen in the usb4t incident.
|
||||
|
||||
**Decision 2026-08-30: accepted, nothing on `tank` is irreplaceable.**
|
||||
Media and instances are re-fetchable from their sources; the ks4
|
||||
replicas are leg 3 of ks4's 3-2-1 (local `sdb5` + restic/S3 remain —
|
||||
[backup-strategy.md](../backup-strategy.md)). Losing `tank` costs time,
|
||||
not data.
|
||||
|
||||
Three SATA ports stay free, so `zpool attach tank <existing> <new>`
|
||||
turns it into a mirror whenever a spare 4 TB turns up. Not a
|
||||
prerequisite for anything. Scrub weekly regardless — on a single vdev
|
||||
the scrub is the only thing that *tells* you a file has rotted.
|
||||
|
||||
## 1. IPMI and BIOS first
|
||||
|
||||
The AST2400 stack is old and has known vulnerabilities.
|
||||
|
||||
- **Decision 2026-08-30: the BMC is never cabled.** The dedicated IPMI
|
||||
port stays unplugged, so the AST2400's default credentials and its
|
||||
known vulnerabilities are not reachable from anything. This is the
|
||||
simplest correct answer for a box that sits on a flat home LAN — no
|
||||
management VLAN needed, nothing to harden, nothing to patch.
|
||||
- ⚠️ Consequence: **there is no out-of-band console.** A boot that fails
|
||||
before sshd needs a physical monitor and keyboard. Worth knowing before
|
||||
changing anything that affects booting (GRUB, the md arrays, fstab).
|
||||
- BIOS: enable **restore-on-AC-loss**, enable C-states, and disable the
|
||||
three unused i354 NICs (each costs about a watt).
|
||||
|
||||
### Switching this board to UEFI
|
||||
|
||||
There is **no "Boot Mode Select" entry** in this BIOS — that option
|
||||
exists on later Supermicro generations, not here. What works
|
||||
(verified 2026-08-30):
|
||||
|
||||
- **CSM → Disabled**
|
||||
- **All OpROM policies → UEFI** (storage, video *and* network)
|
||||
|
||||
Setting the **video** OpROM to UEFI is safe despite the console being
|
||||
the AST2400 BMC framebuffer — output survives, both on the BMC console
|
||||
and over IPMI KVM.
|
||||
|
||||
⚠️ **Confirm the mode before you partition anything**, from the
|
||||
installer (`Ctrl+Alt+F2`):
|
||||
|
||||
```sh
|
||||
ls /sys/firmware/efi # directory exists = UEFI. Missing = legacy
|
||||
```
|
||||
|
||||
This single check is what makes the difference between a working
|
||||
install and an afternoon lost — see the trap in §3.
|
||||
|
||||
## 2. Prepare the SSDs
|
||||
|
||||
Both report `ATA Security is: Disabled, NOT FROZEN`, so a real secure
|
||||
erase is available — do that rather than just repartitioning. It
|
||||
restores the full spare-block pool on ten-year-old NAND.
|
||||
|
||||
```sh
|
||||
# per disk, from a live system where the disk is NOT the running OS
|
||||
hdparm --user-master u --security-set-pass Eins /dev/sdX
|
||||
hdparm --user-master u --security-erase Eins /dev/sdX
|
||||
hdparm -I /dev/sdX | grep -A2 Security # expect "not enabled"
|
||||
```
|
||||
|
||||
Identical GPT layout on both, sized so ~27 GB (22 %) stays
|
||||
**unallocated** as over-provisioning:
|
||||
|
||||
| Part | Size | Type | Device | Use |
|
||||
|------|------|------|--------|-----|
|
||||
| `sdX1` | 1 GB | EFI System Partition | — | `/boot/efi` (two independent ESPs) |
|
||||
| `sdX2` | 2 GB | Linux RAID | `md0` | `/boot` ext4 |
|
||||
| `sdX3` | 24 GB | Linux RAID | `md1` | `/` ext4 |
|
||||
| `sdX4` | 2 GB | Linux RAID | `md2` | swap |
|
||||
| `sdX5` | 64 GB | Solaris root (bf00) | — | **zpool `incus`** (mirror) |
|
||||
| — | ~27 GB | **unallocated** | — | over-provisioning |
|
||||
|
||||
- ⚠️ **The ESP is deliberately not a RAID1 array.** ks4 has `md1 →
|
||||
/boot/efi` because OVH's installer builds it with mdadm metadata
|
||||
**1.0** (superblock at the *end*, so firmware still sees plain FAT).
|
||||
`debian-installer` only creates metadata **1.2** arrays, whose
|
||||
superblock sits at the start and makes the ESP unreadable to
|
||||
firmware. So: one plain ESP per disk, only one mounted, the second
|
||||
filled by hand (§3). Deviation from ks4 is intentional.
|
||||
- A legacy-BIOS variant of this layout was tried first — 1 MB
|
||||
`bios_grub` instead of the ESP, which makes the mirror simpler
|
||||
(`grub-install` to both disks, nothing to keep in sync). It was
|
||||
abandoned because this board's firmware has no way to prefer legacy
|
||||
targets once CSM is off, and it kept falling through to the UEFI
|
||||
shell. Recorded so it is not retried: **UEFI is the working path
|
||||
here.**
|
||||
- `/` at 24 GB matches the other hosts (nuc 46 GB, ks4 40 GB) — this box
|
||||
has no desktop and no container roots on `/`.
|
||||
- Mixing md partitions and a ZFS partition on the same disks is exactly
|
||||
what ks4 does (`md1/2/3` + ZFS on `sda5`/`sdb5`).
|
||||
|
||||
### Why not full root-on-ZFS
|
||||
|
||||
Decision 2026-08-30. These SSDs previously ran Ubuntu 20.04's
|
||||
experimental ZFS-root installer (hence the leftover `bpool` / `rpool`
|
||||
labels), so the option was on the table. Rejected because:
|
||||
|
||||
- **Debian's installer cannot do it.** ZFS is CDDL, shipped only in
|
||||
`contrib` as `zfs-dkms`; `debian-installer` can neither partition nor
|
||||
boot from ZFS. Root-on-ZFS means the manual
|
||||
[OpenZFS Debian Trixie HOWTO](https://openzfs.github.io/openzfs-docs/Getting%20Started/Debian/Debian%20Trixie%20Root%20on%20ZFS.html)
|
||||
— ~60 steps from a live ISO. Per the repo convention the doc *is* the
|
||||
rebuild procedure, and that is a bad thing to be executing during an
|
||||
actual failure.
|
||||
- **DKMS failure mode.** `zfs-dkms` rebuilds on every kernel upgrade.
|
||||
If that build fails, root-on-ZFS means the box **does not boot**;
|
||||
with an ext4 md root it boots normally and only the pools are
|
||||
missing — recoverable over IPMI with a shell.
|
||||
- GRUB's ZFS support lags OpenZFS, which is why every root-on-ZFS guide
|
||||
needs a separate feature-limited `bpool`; `zpool upgrade bpool` is a
|
||||
known way to make a machine unbootable.
|
||||
- Ubuntu's version of this is a dead end anyway: the installer option
|
||||
was nearly dropped in 22.04 and `zsys`, which made boot environments
|
||||
useful, is abandoned.
|
||||
|
||||
What root-on-ZFS would buy — snapshot and roll back a bad upgrade — is
|
||||
already covered where the state actually lives: container roots get
|
||||
incus snapshots plus nightly replication (§9). The host is 24 GB of
|
||||
packages reproducible from this file. Accepted cost: no pre-upgrade
|
||||
rollback of the host itself (`etckeeper` covers `/etc` if wanted).
|
||||
|
||||
## 3. Install Debian 13 (trixie)
|
||||
|
||||
Netinst ISO (burned 2026-08-30, sha256 `65273bee…664e7`, verified
|
||||
against `cdimage.debian.org/debian-cd/13.6.0/amd64/iso-cd/SHA256SUMS`).
|
||||
|
||||
- Manual partitioning per the table above: `sdX1` as ESP, `sdX2..4` as
|
||||
RAID1 members (three arrays), `sdX5` left untouched.
|
||||
- Tasks: **SSH server + standard utilities only**. No desktop — there is
|
||||
no GPU and the console is a BMC framebuffer.
|
||||
- Sources: `main contrib non-free-firmware` (`contrib` is required by
|
||||
`zfs-dkms`; the installer does not offer it — add it after first boot).
|
||||
|
||||
Confirm d-i picked the right bootloader once installed:
|
||||
`dpkg -l | grep grub-efi` — `grub-efi-amd64`, not `grub-pc`.
|
||||
|
||||
### ⚠️ Trap: installer boot mode decides the bootloader (hit 2026-08-30)
|
||||
|
||||
`debian-installer` chooses `grub-pc` or `grub-efi-amd64` from **how the
|
||||
installer itself booted**, not from what the disks look like. Booting the
|
||||
USB stick in legacy mode while the firmware prefers UEFI produces:
|
||||
|
||||
1. d-i installs `grub-pc`, targeting the MBR;
|
||||
2. on a **GPT** disk that needs a 1 MB `bios_grub` partition — absent
|
||||
here, so `grub-install` fails, easy to click past;
|
||||
3. the firmware then tries the SSDs as UEFI targets, finds no `.efi`
|
||||
binary, and drops to the **UEFI shell**.
|
||||
|
||||
Nothing is corrupt; the halves simply disagree. Symptoms and checks:
|
||||
|
||||
```sh
|
||||
[ -d /sys/firmware/efi ] && echo UEFI || echo legacy # in the installer
|
||||
lsblk -no PTTYPE,PARTTYPENAME /dev/sdX # gpt + "EFI System"?
|
||||
dd if=/dev/sdX bs=440 count=1 2>/dev/null | od -c | head -3 # all \0 = no boot code
|
||||
dpkg -l | grep -E '^ii.*grub-(pc|efi)' # which flavour got installed
|
||||
```
|
||||
|
||||
The board's hybrid ISO offers both paths, so the F11 boot menu usually
|
||||
lists the stick twice — picking the **`UEFI:`** entry avoids the whole
|
||||
thing. Checking `/sys/firmware/efi` before partitioning is the one step
|
||||
that prevents it.
|
||||
|
||||
### Second ESP — do this before trusting the mirror
|
||||
|
||||
The installer populates only the ESP it mounted. Until the second one is
|
||||
written, losing that disk means the box does not boot, mirror or no
|
||||
mirror.
|
||||
|
||||
⚠️ **Use `/dev/disk/by-id/`, never `/dev/sdX`.** Reconnecting the 4 TB
|
||||
after the install shifts every letter — observed 2026-08-30: the HDD on
|
||||
SATA2-0 takes `sda` even with the SSDs on SATA3, because the SATA2
|
||||
controller enumerates first on this SoC. A bare `/dev/sdb1` written
|
||||
during the install then points at a *different disk*, and here that
|
||||
would mean reformatting the ESP the system actually boots from.
|
||||
|
||||
```sh
|
||||
ESP2=/dev/disk/by-id/ata-TOSHIBA_Q300._36OB318OK1KU-part1 # the one NOT at /boot/efi
|
||||
mkfs.vfat -F32 "$ESP2"
|
||||
mkdir -p /boot/efi2 && mount "$ESP2" /boot/efi2
|
||||
grub-install --target=x86_64-efi --efi-directory=/boot/efi2 \
|
||||
--bootloader-id=debian-b --recheck
|
||||
efibootmgr -v # expect: debian, debian-b
|
||||
echo "UUID=$(blkid -s UUID -o value $ESP2) /boot/efi2 vfat umask=0077 0 1" >> /etc/fstab
|
||||
mount -a && findmnt /boot/efi2
|
||||
```
|
||||
|
||||
Re-run the `grub-install` after any GRUB or kernel change — the second
|
||||
ESP is **not** kept in sync automatically. **Verify by pulling one disk
|
||||
and booting**; the checklist item exists because an untested mirror is a
|
||||
guess, and this is the component most likely to be silently wrong.
|
||||
|
||||
Do not fix the letter ordering by moving cables: the only arrangement
|
||||
that makes the SSDs `sda`/`sdb` puts them on SATA2 and the HDD on SATA3,
|
||||
which caps the only devices that can use 6 Gb/s and gives the bandwidth
|
||||
to a disk that tops out near 180 MB/s.
|
||||
|
||||
## 4. Base system
|
||||
|
||||
```sh
|
||||
apt update && apt full-upgrade -y
|
||||
apt install -y \
|
||||
linux-headers-amd64 zfs-dkms zfsutils-linux zfs-zed \
|
||||
mdadm smartmontools nfs-kernel-server \
|
||||
msmtp msmtp-mta bsd-mailx \
|
||||
curl vim htop ripgrep sysstat dmidecode pciutils usbutils \
|
||||
stress-ng fio
|
||||
```
|
||||
|
||||
Static network — `/etc/network/interfaces` (ifupdown, matching nuc):
|
||||
|
||||
```
|
||||
source /etc/network/interfaces.d/*
|
||||
|
||||
auto lo
|
||||
iface lo inet loopback
|
||||
|
||||
allow-hotplug enp0s20f0
|
||||
iface enp0s20f0 inet static
|
||||
address 192.168.0.4
|
||||
netmask 255.255.255.0
|
||||
gateway 192.168.0.1
|
||||
dns-nameservers 1.1.1.1 9.9.9.9
|
||||
```
|
||||
|
||||
(Gateway is **`192.168.0.1`** — the FTTH box, since 2026-09. The host
|
||||
uses public resolvers, never blocky, to avoid a bootstrap loop.
|
||||
Interface name is a guess until the board is up — check `ip -br link`.)
|
||||
|
||||
Restore `/root/.ssh/authorized_keys` (incl. `id_rsa_claude.pub`) and
|
||||
`timedatectl set-timezone Europe/Paris`.
|
||||
|
||||
## 5. Pools
|
||||
|
||||
### 5a. `incus` — SSD mirror
|
||||
|
||||
```sh
|
||||
zpool create -o ashift=12 \
|
||||
-O compression=zstd -O atime=off -O xattr=sa -O acltype=posixacl \
|
||||
incus mirror \
|
||||
/dev/disk/by-id/ata-INTEL_SSDSC2CT120A3_CVMP250400ES120BGN-part5 \
|
||||
/dev/disk/by-id/ata-TOSHIBA_Q300._36OB318OK1KU-part5
|
||||
zpool set autotrim=on incus
|
||||
```
|
||||
|
||||
Weekly scrubs come from the packaged systemd timers rather than cron:
|
||||
|
||||
```sh
|
||||
systemctl enable --now zfs-scrub-weekly@tank.timer zfs-scrub-weekly@incus.timer
|
||||
systemctl list-timers 'zfs-scrub*'
|
||||
```
|
||||
|
||||
`autotrim` matters on ten-year-old NAND — it is what keeps the
|
||||
unallocated 22 % actually available to the controller as spare.
|
||||
|
||||
### 5b. `tank` — move the 4 TB off USB onto SATA
|
||||
|
||||
The risky step. The pool holds `usb4t/backup/nuc` (nuc's replicas),
|
||||
`usb4t/backup/ks4` (empty, awaiting the FTTH seed) and `usb4t/media`.
|
||||
|
||||
⚠️ **Between export and import, nuc has no replica target and Jellyfin
|
||||
has no media.** Plan a maintenance window and disable nuc's 03:30 cron
|
||||
first, so it fails loudly rather than half-running.
|
||||
|
||||
```sh
|
||||
# --- on nuc, first ---
|
||||
zpool scrub usb4t # start clean; wait for it
|
||||
zpool status usb4t
|
||||
zpool export usb4t
|
||||
```
|
||||
|
||||
Move the disk to SATA2-0 (reconnect it now if you unplugged it for the
|
||||
install), then:
|
||||
|
||||
```sh
|
||||
# --- on nas ---
|
||||
zpool import # confirm it is seen
|
||||
zpool import usb4t tank # rename: it is not USB any more
|
||||
zpool set cachefile=/etc/zfs/zpool.cache tank
|
||||
zpool status tank
|
||||
```
|
||||
|
||||
Properties survive from creation (`ashift=12`, `compression=zstd`,
|
||||
`atime=off`, `xattr=sa`, `acltype=posixacl`). Re-point the mountpoints
|
||||
and add the two new backup datasets:
|
||||
|
||||
```sh
|
||||
zfs set mountpoint=/export/media tank/media # NFSv4 export root (§8)
|
||||
zfs set mountpoint=none tank/backup
|
||||
zfs list -o name,used,avail,mountpoint
|
||||
```
|
||||
|
||||
Resulting layout:
|
||||
|
||||
```
|
||||
incus mirror, 2× SSD — nas's own container roots
|
||||
tank 4 TB, single vdev
|
||||
├── tank/media → /export/media NFS ro → nuc; local device → transmission-bt
|
||||
└── tank/backup
|
||||
├── tank/backup/nuc → incus pool `nucbackup` (nuc pushes here)
|
||||
└── tank/backup/ks4 → incus pool `ks4backup` (nas pulls from ks4)
|
||||
|
||||
nas's own replicas live on **nuc** (`data/backup/nas` → pool `nasbackup`),
|
||||
not here — see §9.
|
||||
```
|
||||
|
||||
Backup pools are named after the **source** host, matching
|
||||
[ks4/incus-copy.md](../ks4/incus-copy.md). nas's *own* instances are not
|
||||
backed up here — they cross-replicate to nuc (§9b), so neither host's
|
||||
instances depend on that host surviving.
|
||||
|
||||
Once `tank` has run a week on SATA with **zero** pool suspensions and no
|
||||
CRC errors, the `usb4t-dropouts` gate is cleared — record that in
|
||||
[ks2/plan.md](../ks2/plan.md). Day 1 was clean (2026-08-31).
|
||||
|
||||
### Clearing the inherited `<metadata>` errors — order matters
|
||||
|
||||
The pool imported carrying `<metadata>:<0x0>` and `<metadata>:<0x3d>` from
|
||||
the 2026-08-29 USB dropout. A scrub found **0 errors and repaired 0B**,
|
||||
yet the entries stayed, and a plain `zpool clear` afterwards did not drop
|
||||
them either. ZFS flushes its persistent error log on a scrub that runs
|
||||
**after** the clear — so the working order is:
|
||||
|
||||
```sh
|
||||
zpool clear tank
|
||||
zpool scrub tank # this is the run that flushes the log
|
||||
```
|
||||
|
||||
Result 2026-08-31: `scrub repaired 0B in 02:34:24 with 0 errors`,
|
||||
`errors: No known data errors`, `all pools are healthy`. They were
|
||||
artefacts of interrupted writes, not corruption — matching the
|
||||
[2026-08-28 incident](../nuc/usb4t-dropouts.md).
|
||||
|
||||
⚠️ This matters for monitoring, not just tidiness: while those entries
|
||||
stand, `zpool status -x` reports the pool unhealthy permanently, so
|
||||
`zpool-health.sh` sits in the alarm state and **cannot signal a new
|
||||
problem**. Clear them before trusting the watchdog.
|
||||
|
||||
## 6. Incus
|
||||
|
||||
Same Zabbly stable repo as nuc and ks4:
|
||||
|
||||
```sh
|
||||
mkdir -p /etc/apt/keyrings
|
||||
curl -fsSL https://pkgs.zabbly.com/key.asc -o /etc/apt/keyrings/zabbly.asc
|
||||
cat > /etc/apt/sources.list.d/zabbly-incus-stable.sources <<EOF
|
||||
Enabled: yes
|
||||
Types: deb
|
||||
URIs: https://pkgs.zabbly.com/incus/stable
|
||||
Suites: trixie
|
||||
Components: main
|
||||
Architectures: amd64
|
||||
Signed-By: /etc/apt/keyrings/zabbly.asc
|
||||
EOF
|
||||
apt update && apt install -y incus
|
||||
```
|
||||
|
||||
macvlan like nuc, so instances get real LAN addresses — `transmission-bt`
|
||||
keeps `192.168.0.7` when it moves:
|
||||
|
||||
```sh
|
||||
cat <<EOF | incus admin init --preseed
|
||||
config:
|
||||
core.https_address: :8443
|
||||
storage_pools:
|
||||
- name: incus
|
||||
driver: zfs
|
||||
config:
|
||||
source: incus
|
||||
networks:
|
||||
- name: macvlan
|
||||
type: macvlan
|
||||
config:
|
||||
parent: enp0s20f0
|
||||
profiles:
|
||||
- name: default
|
||||
devices:
|
||||
eth0: {name: eth0, network: macvlan, type: nic}
|
||||
root: {path: /, pool: incus, type: disk}
|
||||
EOF
|
||||
|
||||
incus profile set default snapshots.schedule="0 3 * * *" snapshots.expiry=7d
|
||||
```
|
||||
|
||||
Same macvlan quirk as nuc: **the host cannot talk to its own instances**,
|
||||
and vice versa. Test container services from another LAN host or from
|
||||
inside the container, never from `nas`.
|
||||
|
||||
Backup pools and the replica project:
|
||||
|
||||
```sh
|
||||
incus storage create nucbackup zfs source=tank/backup/nuc
|
||||
incus storage create ks4backup zfs source=tank/backup/ks4
|
||||
incus project create backup -c features.images=false -c features.profiles=false
|
||||
```
|
||||
|
||||
⚠️ **`tank/backup/nuc` already contains nuc's replicas** — they came
|
||||
across with the pool. Re-register them so refreshes stay
|
||||
ZFS-incremental instead of re-sending everything (the homeassistant VM
|
||||
alone is a 50 GiB volume):
|
||||
|
||||
```sh
|
||||
incus admin recover # point it at pool nucbackup; project backup
|
||||
incus list --project backup
|
||||
```
|
||||
|
||||
Same call nuc-install.md §5 uses after a rebuild. If `recover` is
|
||||
skipped, the first push in §9a silently becomes a full re-send of every
|
||||
nuc instance.
|
||||
|
||||
Order incus after the ZFS mounts, as on nuc —
|
||||
`/etc/systemd/system/incus.service.d/after-zfs.conf`:
|
||||
|
||||
```ini
|
||||
[Unit]
|
||||
After=zfs-mount.service zfs.target
|
||||
```
|
||||
|
||||
## 7. Move `transmission-bt` from nuc
|
||||
|
||||
Its WireGuard tunnel is **entirely inside the container** (`wg0`,
|
||||
`10.8.0.21`, `wg-quick@wg0`, `BindsTo=` on the daemon), so the container
|
||||
carries its own keys and **ks4 needs no change at all** — the peer stays
|
||||
`10.8.0.21/32`. The kill-switch `/32` route points at the gateway
|
||||
`192.168.0.1`, which is the same from here.
|
||||
|
||||
```sh
|
||||
# on nuc — remote already added in §9a
|
||||
incus stop transmission-bt
|
||||
incus move transmission-bt nas: --storage incus
|
||||
```
|
||||
|
||||
Then on nas, re-point the media device at the local dataset — this is a
|
||||
plain `shift=true` device again, because the data is local ZFS:
|
||||
|
||||
```sh
|
||||
incus config device remove transmission-bt media
|
||||
incus config device add transmission-bt media disk \
|
||||
source=/export/media path=/media shift=true
|
||||
incus config set transmission-bt boot.autostart=true
|
||||
incus start transmission-bt
|
||||
```
|
||||
|
||||
Verify the tunnel and the kill switch before trusting it:
|
||||
|
||||
```sh
|
||||
incus exec transmission-bt -- wg show
|
||||
incus exec transmission-bt -- curl -s ifconfig.me # must print 193.70.35.17
|
||||
incus exec transmission-bt -- ip route # must have NO default route
|
||||
```
|
||||
|
||||
⚠️ **The watch-folder workflow moves with it** —
|
||||
[transmission-bt.md](transmission-bt.md) says
|
||||
`scp some.torrent root@192.168.0.3:/srv/media/.watchdir/`; it is now
|
||||
`root@192.168.0.4:/export/media/.watchdir/`.
|
||||
|
||||
## 8. Media over NFS — `jellyfin-server` stays on nuc
|
||||
|
||||
`jellyfin-server` needs the Alder Lake-N iGPU for QSV/VAAPI; the C2750
|
||||
has no render device at all, and software transcoding on Silvermont
|
||||
manages 1–2 concurrent 1080p H.264 streams at best. So the container
|
||||
stays on nuc and reaches the library over NFS, **read-only** —
|
||||
`transmission-bt` is the only writer and it now lives here.
|
||||
|
||||
```sh
|
||||
# --- nas: NFSv4 export, read-only, nuc only ---
|
||||
cat >> /etc/exports <<'EOF'
|
||||
/export 192.168.0.3(ro,fsid=0,crossmnt,no_subtree_check)
|
||||
/export/media 192.168.0.3(ro,no_subtree_check,all_squash,anonuid=65534,anongid=65534)
|
||||
EOF
|
||||
exportfs -ra && exportfs -v
|
||||
```
|
||||
|
||||
```sh
|
||||
# --- nuc: mount at the SAME path, so jellyfin-server.md still applies ---
|
||||
mkdir -p /srv/media
|
||||
echo '192.168.0.4:/media /srv/media nfs4 ro,_netdev,soft,timeo=100,retrans=3 0 0' >> /etc/fstab
|
||||
mount /srv/media && ls /srv/media
|
||||
```
|
||||
|
||||
The container device changes only in losing the shift — per
|
||||
[nuc/jellyfin-server.md](../nuc/jellyfin-server.md)'s own troubleshooting
|
||||
note, *"on CIFS files are world-readable synthetic ownership, enough for
|
||||
a read-only library"*; the same holds for NFS with `all_squash`:
|
||||
|
||||
```sh
|
||||
incus stop jellyfin-server # shift cannot be hot-applied
|
||||
incus config device set jellyfin-server media shift=false
|
||||
incus config device set jellyfin-server media readonly=true
|
||||
incus start jellyfin-server
|
||||
incus exec jellyfin-server -- ls /media # must list the library
|
||||
```
|
||||
|
||||
Requirements for that to work: media files must be **world-readable**
|
||||
(`find /export/media -type f ! -perm -o=r`), and transmission must keep
|
||||
creating them that way (it does — `umask`/`0775` per its doc).
|
||||
|
||||
⚠️ Boot ordering on nuc: incus is already ordered after
|
||||
`zfs-mount.service`; add `remote-fs.target` to that drop-in, or
|
||||
`jellyfin-server` starts against an empty mountpoint and shows an empty
|
||||
library.
|
||||
|
||||
⚠️ `soft` mount is deliberate: a hung NAS should fail Jellyfin's reads,
|
||||
not wedge nuc's processes in uninterruptible sleep the way the suspended
|
||||
`usb4t` pool did.
|
||||
|
||||
## 9. Backup legs
|
||||
|
||||
The driver is unchanged —
|
||||
[`incus-copy.sh`](https://git.lutran.fr/julien/scripts/src/branch/main/incus-copy.sh),
|
||||
deployed to `/root/scripts` as everywhere else.
|
||||
|
||||
### 9a. nuc's instances -> nas
|
||||
|
||||
Root crontab on nuc:
|
||||
|
||||
```cron
|
||||
30 3 * * * /root/scripts/incus-copy.sh -d nas -m push -s nucbackup -p backup >> /var/log/incus-copy.log 2>&1
|
||||
```
|
||||
|
||||
⚠️ **nuc is an on-demand media box** (see
|
||||
[nuc/README.md](../nuc/README.md)): since 2026-08-30 it only runs when
|
||||
watching Jellyfin or using the Spotify kiosk, so it is often powered
|
||||
off at 03:30 and **that night's push is simply skipped** — cron does
|
||||
not catch up missed windows. Accepted deliberately (2026-08-31):
|
||||
nuc's instances change rarely and the next time it is up the refresh
|
||||
is incremental anyway.
|
||||
This ran briefly as a `systemd` timer with `Persistent=true` (which
|
||||
*does* catch up after boot); the units are still on disk, disabled, at
|
||||
`/etc/systemd/system/incus-copy.{service,timer}` if that behaviour is
|
||||
ever wanted back:
|
||||
`systemctl enable --now incus-copy.timer` (and remove the cron line).
|
||||
|
||||
⚠️ **`-p backup` is not optional.** Without it the replicas land in
|
||||
`default` on nas and collide with nas's *live* instances — both hosts are
|
||||
on the same macvlan LAN and the replicas carry the same static IPs.
|
||||
|
||||
### 9b. nas's own instances -> local pool `nasbackup`
|
||||
|
||||
**Decision 2026-08-30: local, not cross-replicated.** nuc is powered off
|
||||
most of the time, so it is not a usable backup target — a nightly push to
|
||||
it would fail noisily and, once mail works, alarm every morning. The
|
||||
replicas instead go to `tank/backup/nas`, which is a **different pool**
|
||||
from the instances themselves (`incus`, the SSD mirror), so it survives
|
||||
losing that mirror. All three instances here are rebuildable from their
|
||||
docs, so same-host is proportionate.
|
||||
|
||||
```sh
|
||||
zfs create tank/backup/nas
|
||||
zfs set mountpoint=legacy tank/backup/nas # see the trap below
|
||||
incus storage create nasbackup zfs source=tank/backup/nas
|
||||
```
|
||||
|
||||
```cron
|
||||
# nas, root crontab
|
||||
0 4 * * * root /root/scripts/incus-copy.sh -p backup -s nasbackup >> /var/log/incus-copy.log 2>&1
|
||||
```
|
||||
|
||||
⚠️ **Trap: `incus storage create` hangs forever on a `mountpoint=none`
|
||||
dataset.** `tank/backup` is set to `mountpoint=none`, so any child
|
||||
created afterwards inherits it, and the pool create then blocks with no
|
||||
error and no entry in `incus operation list` — it looks exactly like I/O
|
||||
contention (a scrub was running, which sent me down that path for 20
|
||||
minutes). Set the child to `legacy` to match its siblings first, and it
|
||||
completes instantly.
|
||||
|
||||
### 9c. Cleanup on nuc after the pool move
|
||||
|
||||
|
||||
Exporting `usb4t` leaves nuc with an incus storage pool whose backing
|
||||
dataset is gone, plus replica records in the `backup` project pointing
|
||||
at it. Remove them once §6's `incus admin recover` has re-registered the
|
||||
same volumes on nas — **verify there first, then delete here**:
|
||||
|
||||
```sh
|
||||
# on nas: confirm the replicas are registered
|
||||
incus list --project backup -c ns -f csv
|
||||
|
||||
# on nuc: only then
|
||||
incus delete --project backup --force <each-replica>
|
||||
incus storage delete nucbackup
|
||||
zpool status # only `data` should remain
|
||||
```
|
||||
|
||||
### 9d. ks4 pull leg moves from nuc to nas
|
||||
|
||||
The leg [ks2/nas-seed.md](../ks2/nas-seed.md) prepared. The target pool
|
||||
moves, so the **host** WireGuard tunnel moves too — nuc drops `wg-ks4`
|
||||
entirely once this works (its only other tunnel user, `transmission-bt`,
|
||||
is now here and carries its own).
|
||||
|
||||
- **ks4**: add nas as a peer on the `wireguard` container —
|
||||
`wg set wg0 peer <nas-pubkey> allowed-ips 10.8.0.22/32 && wg-quick save wg0`.
|
||||
The existing ufw rule (`allow in on incusbr0 from 192.168.1.18 to any
|
||||
port 8443 proto tcp`) already covers it: traffic arrives masqueraded as
|
||||
the WG container whichever peer sent it.
|
||||
- **nas**: `/etc/wireguard/wg-ks4.conf` modelled on nuc's —
|
||||
`Address = 10.8.0.22/32`, peer pubkey `TVs6d7…`,
|
||||
`Endpoint = 193.70.35.17:51845`,
|
||||
`AllowedIPs = 10.8.0.0/24, 192.168.1.1/32`, keepalive 25 — then
|
||||
`systemctl enable --now wg-quick@wg-ks4`.
|
||||
- `incus remote add ks4 https://192.168.1.1:8443 --accept-certificate --token '…'`
|
||||
(cross-check the fingerprint against the token).
|
||||
- **nuc, after verification**: `systemctl disable --now wg-quick@wg-ks4`,
|
||||
remove `/etc/wireguard/wg-ks4.conf`, and drop the `10.8.0.20/32` peer
|
||||
on ks4.
|
||||
|
||||
Built 2026-09-16 — full runbook, corrections and gotchas:
|
||||
[ks2/nas-seed.md](../ks2/nas-seed.md). Two things that block the copy if
|
||||
missed: nas needs `wireguard-tools` installed and a **managed `incusbr0`
|
||||
on `192.168.1.254/24`** (never `.1`), and the seed needs **`-p backup`**.
|
||||
|
||||
The pull runs at ~14 MB/s (~112 Mbit/s) and that is **ks4's single
|
||||
spinning source disk, not the link or the tunnel** — measured, with the
|
||||
numbers, in [ks2/nas-seed.md](../ks2/nas-seed.md) §Bottleneck. Nothing to
|
||||
fix on the network side.
|
||||
|
||||
```sh
|
||||
systemd-run --unit=ks4-seed --collect \
|
||||
/bin/bash -c '/root/scripts/incus-copy.sh -r ks4 -s ks4backup -p backup \
|
||||
>> /var/log/incus-copy-ks4.log 2>&1'
|
||||
```
|
||||
|
||||
Then test-restore one instance before ticking the gate in
|
||||
[ks2/plan.md](../ks2/plan.md).
|
||||
|
||||
### Resulting schedule
|
||||
|
||||
| When | Host | What |
|
||||
|---|---|---|
|
||||
| 03:00 | nuc, nas | instance snapshots (profile) |
|
||||
| 03:30 | nuc | push all instances → `nas:nucbackup` (root crontab; skipped when nuc is off) |
|
||||
| 04:00 | nas | local copy of nas instances → `nasbackup` (`tank/backup/nas`) |
|
||||
| 05:00 | nas | pull `ks4:*` → `ks4backup` (after FTTH) |
|
||||
| 05:30 | nuc | apt upgrade all containers |
|
||||
| 06:00 | nas | apt upgrade all containers (`incus-container-upgrade.sh`, added 2026-09-01; also refreshes `user.os`) |
|
||||
| Mon ~00:12 | nas | `zfs-scrub-weekly@tank.timer` / `@incus.timer` (systemd, not cron) |
|
||||
|
||||
Staggered around ks4's own 01:00 / 05:00 jobs.
|
||||
|
||||
⚠️ Keep `snapshots.schedule` on every source. A refresh with no common
|
||||
snapshot degrades to a **full re-send** — the failure mode that cost
|
||||
933 G on ks4 ([ks4/local-backup-cron.md](../ks4/local-backup-cron.md)).
|
||||
The homeassistant **VM** on nuc re-sends its whole volume without them.
|
||||
Note `incus-copy.sh` takes a **global** `flock`: an overrunning ks4 pull
|
||||
aborts that night's other run loudly rather than racing it.
|
||||
|
||||
## 10. Monitoring
|
||||
|
||||
Do not repeat nuc's 11-day blind spot
|
||||
([usb4t-dropouts.md](../nuc/usb4t-dropouts.md) — zed was running, but the
|
||||
host had no MTA).
|
||||
|
||||
**Status: live since 2026-08-30**, verified end to end (`smtpstatus=250`).
|
||||
|
||||
- **msmtp**, same shape as nuc: this box shares the dynamic home IP with
|
||||
no PTR and no SPF alignment, so it must use **submission (587) with
|
||||
auth**, not port 25 — rspamd rejects the port-25 path as spam.
|
||||
`/etc/msmtprc` mode 600, host `mail.lutran.fr`, STARTTLS. Give SMTP
|
||||
tests ≥ 30 s: the missing PTR delays the greeting.
|
||||
The `zed@lutran.fr` account is the **same credential as nuc** — it is
|
||||
SMTP-AUTH, not IP-bound, so `/etc/msmtprc` can simply be copied between
|
||||
hosts (mode 600, root:root). The password lives in the password
|
||||
manager; no host-specific setup is needed.
|
||||
- **zed**: `/etc/zfs/zed.d/zed.rc` mode 600 with
|
||||
`ZED_EMAIL_ADDR="julien@lutran.fr"`, `ZED_EMAIL_PROG="mail"`,
|
||||
`ZED_NOTIFY_VERBOSE=1`, **`ZED_NOTIFY_DATA=1`**,
|
||||
`ZED_NOTIFY_INTERVAL_SECS=3600`.
|
||||
- **[`zpool-health.sh`](https://git.lutran.fr/julien/scripts/src/branch/main/zpool-health.sh)**
|
||||
every 15 min from root's crontab — zed does *not* report a
|
||||
suspended pool (the vdev stays `ONLINE`, so `statechange-notify.sh`
|
||||
never fires). That watchdog is the only thing that catches the exact
|
||||
failure this box was built to prevent.
|
||||
- **smartd** — `/etc/smartd.conf`. The Toshiba carries the useful
|
||||
attributes; the Intel is a liveness check only:
|
||||
|
||||
```
|
||||
# Toshiba Q300 — endurance + temperature are real here
|
||||
/dev/disk/by-id/ata-TOSHIBA_Q300._36OB318OK1KU -a -o on -S on -s (S/../.././02|L/../../6/03) -W 4,50,55 -m julien@lutran.fr
|
||||
# Intel 330 — no error log, no devstat, no temperature: liveness only
|
||||
/dev/disk/by-id/ata-INTEL_SSDSC2CT120A3_CVMP250400ES120BGN -H -s (S/../.././03|L/../../6/04) -m julien@lutran.fr
|
||||
# WD Red
|
||||
/dev/disk/by-id/ata-WDC_WD40EFRX-68WT0N0_WD-WCC4E6NLPJJE -a -o on -S on -s (S/../.././04|L/../../6/05) -W 4,45,50 -m julien@lutran.fr
|
||||
```
|
||||
|
||||
Verify the whole chain the day you build it, not the day you need it:
|
||||
|
||||
```sh
|
||||
/root/scripts/zpool-health.sh -m julien@lutran.fr -t
|
||||
tail -2 /var/log/msmtp.log # expect smtpstatus=250
|
||||
smartctl -d sat -l devstat /dev/sdX | grep -i endurance
|
||||
```
|
||||
|
||||
## 11. Power baseline
|
||||
|
||||
Take the measurement **before** the box goes into service, so later
|
||||
readings mean something. Expect roughly 25–35 W idle with three disks:
|
||||
the 20 W SoC plus a BMC drawing several watts continuously, even at
|
||||
soft-off.
|
||||
|
||||
```sh
|
||||
# baseline: 10 min idle, everything settled
|
||||
# CPU in steps — the interesting curve for an always-on box
|
||||
for l in 25 50 75 100; do echo "=== ${l}% $(date +%s)"; stress-ng --cpu 0 --cpu-load $l --timeout 120s; done
|
||||
# disk: random I/O is what moves an HDD's power, not throughput
|
||||
fio --name=rr --directory=/export/media --size=20G --rw=randread --bs=4k \
|
||||
--iodepth=32 --numjobs=4 --ioengine=libaio --direct=1 --runtime=300 --time_based
|
||||
```
|
||||
|
||||
Measure at the wall (PDU or smart plug) — RAPL is unreliable on Avoton
|
||||
and sees neither disks nor fans. Log epoch timestamps per step so the
|
||||
trace can be recut against the meter's series afterwards.
|
||||
|
||||
## 12. Post-install checklist
|
||||
|
||||
- [x] IPMI **left unplugged by decision** (2026-08-30) — no BMC on the
|
||||
LAN, therefore no out-of-band console either
|
||||
- [x] `cat /proc/mdstat` — all three arrays `[UU]`; GRUB written to
|
||||
**both** ESPs, both mounted, `debian` + `debian-b` boot entries
|
||||
present (verified across a cold boot 2026-08-31)
|
||||
- [ ] boot still untested with **one disk physically unplugged** — the
|
||||
mirror is a guess until that is done
|
||||
- [x] `zpool status` healthy for `incus` and `tank`; weekly scrubs
|
||||
scheduled; **cold boot verified 2026-08-31** — both pools imported
|
||||
from `/etc/zfs/zpool.cache`, all instances autostarted, NFS exports
|
||||
republished, 0 failed units
|
||||
- [ ] **7 days with zero pool suspensions and zero CRC errors** — the
|
||||
gate that closes [usb4t-dropouts.md](../nuc/usb4t-dropouts.md)
|
||||
- [x] `smartd` monitoring all 3 disks; `zpool-health.sh -t` mail
|
||||
delivered (`smtpstatus=250` in `/var/log/msmtp.log`) — done
|
||||
2026-08-30
|
||||
- [ ] `transmission-bt` on nas: egress is `193.70.35.17`, **no default
|
||||
route**, downloads land in `/export/media/downloads`, watch folder
|
||||
works from the new path
|
||||
- [ ] `jellyfin-server` on nuc lists the library over NFS after a **cold
|
||||
reboot of both hosts** (the boot-ordering trap)
|
||||
- [ ] `incus admin recover` ran on nas **before** the first push, so the
|
||||
inherited `tank/backup/nuc` replicas refresh incrementally instead
|
||||
of re-sending (check the first run's duration, not just `rc=0`)
|
||||
- [ ] nuc's 03:30 leg → `nas:nucbackup`, `rc=0`, all nuc instances
|
||||
present (`incus list --project backup -c ns -f csv` on nas)
|
||||
- [ ] nas's 04:00 local leg → `nasbackup`, `rc=0`, and **blocky,
|
||||
privoxy and transmission-bt all appear by name** in
|
||||
`incus list --project backup`
|
||||
- [ ] stale `nucbackup` pool removed from nuc (§9c), `zpool status`
|
||||
shows only `data`
|
||||
- [ ] replicas are **stopped** with `boot.autostart=false` — they hold
|
||||
the live containers' LAN addresses
|
||||
- [ ] ks4 pull leg seeded + one instance test-restored → tick the gate
|
||||
in [ks2/plan.md](../ks2/plan.md); then retire nuc's `wg-ks4`
|
||||
- [ ] power baseline recorded above, with the meter reading
|
||||
- [ ] this file updated with what was built (RAM fitted, NIC name, WD Red
|
||||
serial)
|
||||
@@ -1,6 +1,13 @@
|
||||
# transmission-bt
|
||||
|
||||
BitTorrent client in an unprivileged Incus container on `nuc`, with an
|
||||
|
||||
> Moved from nuc to `nas` on 2026-08-30, together with the media
|
||||
> dataset ([nas-install.md](nas-install.md) §7). Its WireGuard tunnel is
|
||||
> entirely in-container, so ks4 needed no change — the peer is still
|
||||
> `10.8.0.21`. The watch folder moved with it:
|
||||
> `/export/media/.watchdir` on nas, not `/srv/media/.watchdir` on nuc.
|
||||
|
||||
BitTorrent client in an unprivileged Incus container on `nas`, with an
|
||||
**always-on VPN**: all peer traffic exits via ks4's public IP through a
|
||||
WireGuard tunnel to the `wireguard` container on ks4. Kill switch by
|
||||
construction — `eth0` has **no default route**, so with the tunnel down
|
||||
@@ -12,10 +19,10 @@ the container simply has no path to the internet.
|
||||
whitelist (`192.168.0.*` only)
|
||||
- Egress: WG peer `10.8.0.21` → `193.70.35.17:51845`, `AllowedIPs 0.0.0.0/0`
|
||||
(verified: `curl ifconfig.me` from the container returns ks4's IP)
|
||||
- Downloads: `/media/downloads` (= `usb4t/media`, same dataset Jellyfin
|
||||
- Downloads: `/media/downloads` (= `tank/media`, same dataset Jellyfin
|
||||
reads); in-progress files in `/media/.incomplete` so Jellyfin never
|
||||
scans partials
|
||||
- Watch folder: `scp` a `.torrent` into `/srv/media/.watchdir` on the host
|
||||
- Watch folder: `scp` a `.torrent` into `/export/media/.watchdir` on the host
|
||||
and it auto-downloads (see "Watch folder" below)
|
||||
- `transmission-daemon` is `BindsTo=wg-quick@wg0.service` and binds
|
||||
peer traffic to `10.8.0.21` — three independent layers against leaks
|
||||
@@ -59,17 +66,17 @@ network:
|
||||
addresses: [192.168.0.254]
|
||||
routes:
|
||||
- to: 193.70.35.17/32
|
||||
via: 192.168.0.2
|
||||
via: 192.168.0.1
|
||||
EOF
|
||||
chmod 600 /etc/netplan/10-lxc.yaml
|
||||
netplan apply'
|
||||
|
||||
# packages need a temporary default route (removed right after)
|
||||
incus exec "$CNAME" -- ip route add default via 192.168.0.2
|
||||
incus exec "$CNAME" -- ip route add default via 192.168.0.1
|
||||
incus exec "$CNAME" -- apt-get update
|
||||
incus exec "$CNAME" -- apt-get install -y --no-install-recommends \
|
||||
transmission-daemon wireguard-tools iptables curl
|
||||
incus exec "$CNAME" -- ip route del default via 192.168.0.2
|
||||
incus exec "$CNAME" -- ip route del default via 192.168.0.1
|
||||
|
||||
# WireGuard full tunnel (generate key, print pubkey for the ks4 side)
|
||||
incus exec "$CNAME" -- bash -c 'umask 077
|
||||
@@ -90,7 +97,7 @@ EOF'
|
||||
incus exec "$CNAME" -- systemctl enable --now wg-quick@wg0
|
||||
|
||||
# media share (same dataset as jellyfin-server)
|
||||
incus config device add "$CNAME" media disk source=/srv/media path=/media shift=true
|
||||
incus config device add "$CNAME" media disk source=/export/media path=/media shift=true
|
||||
incus exec "$CNAME" -- mkdir -p /media/downloads /media/.incomplete
|
||||
incus exec "$CNAME" -- chown debian-transmission:debian-transmission \
|
||||
/media/downloads /media/.incomplete
|
||||
@@ -129,9 +136,9 @@ incus exec wireguard -- wg-quick save wg0
|
||||
|
||||
## Watch folder (auto-add torrents)
|
||||
|
||||
Drop a `.torrent` into `/srv/media/.watchdir` on the host and Transmission
|
||||
Drop a `.torrent` into `/export/media/.watchdir` on the host and Transmission
|
||||
auto-adds it and starts downloading — no web UI needed. The folder lives on
|
||||
the shared `usb4t/media` dataset (`/media/.watchdir` inside the container).
|
||||
the shared `tank/media` dataset (`/media/.watchdir` inside the container).
|
||||
|
||||
Edit the **active** config only while the daemon is stopped (it rewrites
|
||||
`settings.json` on exit). The active file is
|
||||
@@ -161,7 +168,7 @@ incus exec transmission-bt -- systemctl start transmission-daemon
|
||||
Usage — the `.torrent` is consumed within a few seconds:
|
||||
|
||||
```sh
|
||||
scp some.torrent root@192.168.0.3:/srv/media/.watchdir/
|
||||
scp some.torrent root@192.168.0.4:/export/media/.watchdir/
|
||||
```
|
||||
|
||||
- `watch-dir-force-generic: true` makes Transmission **poll** the folder
|
||||
@@ -179,7 +186,7 @@ incus exec transmission-bt -- wg show wg0 latest-handshakes # non-zero timesta
|
||||
incus exec transmission-bt -- curl -s https://ifconfig.me # must print 193.70.35.17
|
||||
incus exec transmission-bt -- bash -c "ping -c1 -W2 8.8.8.8 || echo kill-switch OK" # with wg0 down
|
||||
# web UI must be tested from a LAN machine — the macvlan quirk means the
|
||||
# nuc host itself cannot reach 192.168.0.7
|
||||
# nas host itself cannot reach 192.168.0.7 (macvlan, by design)
|
||||
```
|
||||
|
||||
## Notes
|
||||
@@ -192,3 +199,70 @@ incus exec transmission-bt -- bash -c "ping -c1 -W2 8.8.8.8 || echo kill-switch
|
||||
- The image server check can make `incus launch` hang on slow WAN —
|
||||
launching from the cached image fingerprint (`incus image list`)
|
||||
bypasses it.
|
||||
|
||||
## Jellyfin library scan on completion (2026-08-31)
|
||||
|
||||
Jellyfin cannot notice finished downloads by itself any more. It watches
|
||||
libraries with **inotify**, but since the media moved to nas the writer
|
||||
(transmission, here) and the reader (`jellyfin-server` on nuc, over NFS)
|
||||
are on different machines — an inotify event never crosses that. Before
|
||||
the move both shared one local dataset on nuc, so it just worked.
|
||||
|
||||
So transmission tells Jellyfin explicitly, via
|
||||
`script-torrent-done`:
|
||||
|
||||
```json
|
||||
"script-torrent-done-enabled": true,
|
||||
"script-torrent-done-filename": "/usr/local/bin/jellyfin-scan.sh"
|
||||
```
|
||||
|
||||
The hook POSTs to Jellyfin's `/Library/Refresh`:
|
||||
|
||||
```sh
|
||||
#!/bin/sh
|
||||
KEY_FILE=/etc/jellyfin-scan.key
|
||||
JF=http://192.168.0.5:8096
|
||||
NAME="${TR_TORRENT_NAME:-unknown}"
|
||||
[ -r "$KEY_FILE" ] || { logger -t jellyfin-scan "no readable key file; skipped ($NAME)"; exit 0; }
|
||||
KEY=$(tr -d " \t\r\n" < "$KEY_FILE")
|
||||
if curl -fsS -m 15 -X POST -H "X-Emby-Token: $KEY" "$JF/Library/Refresh" >/dev/null 2>&1; then
|
||||
logger -t jellyfin-scan "library scan requested after: $NAME"
|
||||
else
|
||||
logger -t jellyfin-scan "library scan request FAILED (nuc off?) after: $NAME"
|
||||
fi
|
||||
exit 0
|
||||
```
|
||||
|
||||
Design points, each of which matters:
|
||||
|
||||
- **Always `exit 0`, never block.** transmission runs the hook
|
||||
synchronously; a hanging or failing hook stalls the daemon. Both paths
|
||||
are tested — success and unreadable-key both exit 0.
|
||||
- **nuc is usually powered off.** The request then fails, logs
|
||||
`FAILED (nuc off?)`, and Jellyfin picks the file up on its next
|
||||
scheduled scan. Not an error worth alerting on.
|
||||
- **The key file is `640 root:debian-transmission`** — the hook runs as
|
||||
`debian-transmission`, so it must be group-readable, and nothing wider.
|
||||
- Reachable despite the kill switch: `192.168.0.5` is on the directly
|
||||
connected LAN, so it needs no default route.
|
||||
|
||||
Verify:
|
||||
|
||||
```sh
|
||||
incus exec transmission-bt -- su -s /bin/sh debian-transmission \
|
||||
-c 'TR_TORRENT_NAME=selftest /usr/local/bin/jellyfin-scan.sh'
|
||||
incus exec transmission-bt -- journalctl -t jellyfin-scan -n 3
|
||||
```
|
||||
|
||||
**Rotating the key**: create a new one in Jellyfin (Dashboard → API Keys),
|
||||
then
|
||||
|
||||
```sh
|
||||
printf %s '<new-key>' | incus exec transmission-bt -- sh -c \
|
||||
'umask 027; cat > /etc/jellyfin-scan.key; chown root:debian-transmission /etc/jellyfin-scan.key'
|
||||
```
|
||||
|
||||
⚠️ Edit `settings.json` only while the daemon is **stopped** —
|
||||
transmission rewrites the whole file on shutdown and will silently
|
||||
discard changes made underneath it.
|
||||
|
||||
@@ -0,0 +1,143 @@
|
||||
# Creating a new service on ks4 — conventions
|
||||
|
||||
Audience: you have root on ks4 and want to add a service. Follow this
|
||||
and the service is automatically replicated, backed up and
|
||||
restorable, without touching the backup machinery.
|
||||
Companion page: [backup-strategy.md](backup-strategy.md).
|
||||
|
||||
## 1. One service = one incus container
|
||||
|
||||
```sh
|
||||
incus launch images:debian/13 myservice
|
||||
incus config device add myservice eth0 nic nictype=bridged parent=incusbr0 \
|
||||
ipv4.address=192.168.1.30 # pick a free IP, note it in the doc
|
||||
incus exec myservice -- bash # install and configure from here
|
||||
```
|
||||
|
||||
Rules of thumb:
|
||||
|
||||
- **Debian 13 or Ubuntu 24.04 images**, unprivileged (the default).
|
||||
Privileged only with a written reason (`nuc/jellyfin-client.md` is
|
||||
the one example: it needs raw device access).
|
||||
- **Configure through `incus exec` and write the commands down** in
|
||||
`doc/ks4/<service>.md` as you go. The doc *is* the rebuild
|
||||
procedure — that is the whole convention: re-running it recreates
|
||||
the service.
|
||||
- The container is on the NAT bridge `incusbr0` (192.168.1.0/24). It
|
||||
is **not** reachable from the internet by itself.
|
||||
- **Web services**: do not open ports. Add an nginx vhost in the
|
||||
`gateway` container that proxies to `http://192.168.1.30:PORT`;
|
||||
gateway is the single HTTP/S entry point and handles TLS.
|
||||
Non-HTTP services (mail, wireguard) use an incus `proxy` device —
|
||||
copy an existing one as a model.
|
||||
|
||||
## 2. No Docker inside the container
|
||||
|
||||
Docker-in-incus works (`login`/`outline` predate this rule) but costs
|
||||
a second layer of networking, storage and updates, and hides the
|
||||
service's data behind docker volumes.
|
||||
|
||||
Since incus can run **OCI images natively**, an upstream
|
||||
`docker-compose.yml` becomes incus instances:
|
||||
|
||||
```sh
|
||||
incus remote add docker https://docker.io --protocol=oci # once per host
|
||||
incus launch docker:library/redis myservice-redis
|
||||
```
|
||||
|
||||
and multi-container stacks are described with
|
||||
[`incus-compose`](https://github.com/lxc/incus-compose) (upstream,
|
||||
compose-file syntax → incus instances). Each piece is then a normal
|
||||
instance: same network, same snapshots, same backups, `incus exec` to
|
||||
debug, no nested runtime.
|
||||
|
||||
If you truly need Docker (upstream ships only a compose stack you
|
||||
don't want to translate), say so in the service doc and note where
|
||||
the volumes live — you will need them in step 4.
|
||||
|
||||
## 3. Snapshots (do nothing, but know why)
|
||||
|
||||
The `default` profile gives every instance `snapshots.schedule =
|
||||
0 3 * * *`, `snapshots.expiry = 7d`. Leave it. Those snapshots are
|
||||
what make the nightly replication incremental — an instance without
|
||||
them forces a full re-send of its whole disk every night.
|
||||
|
||||
## 4. Wire it into the backups
|
||||
|
||||
**Automatic, nothing to do:**
|
||||
|
||||
- the 01:00 replica leg copies *every* instance to the `backup` pool
|
||||
and off-site — new instances included;
|
||||
- **databases are auto-discovered**: any running container with
|
||||
MariaDB/MySQL or PostgreSQL gets every non-system database dumped
|
||||
nightly, plus users/grants.
|
||||
|
||||
**Manual, one line:** data that lives in the filesystem (uploads,
|
||||
repositories, mail spools…) must be listed in
|
||||
[`restic-paths`](https://git.lutran.fr/julien/scripts/src/branch/main/restic-paths), one absolute host path per line:
|
||||
|
||||
```
|
||||
/var/lib/incus/storage-pools/data/containers/myservice/rootfs/var/lib/myservice
|
||||
```
|
||||
|
||||
(The container's `/x` is `…/containers/<name>/rootfs/x` on the host.)
|
||||
No comments in that file — every line is read as a path. Regenerable
|
||||
caches can be skipped by adding a pattern to
|
||||
[`restic-exclude`](https://git.lutran.fr/julien/scripts/src/branch/main/restic-exclude). Both files live in the `scripts` git
|
||||
repo: edit there, commit, `git pull` on ks4.
|
||||
|
||||
For the database auto-discovery to work, keep the defaults:
|
||||
|
||||
- MariaDB/MySQL: client **and** dump binaries installed, root access
|
||||
over the local unix socket. **Do not leave a `database = …` line in
|
||||
`/root/.my.cnf`** — it breaks `SHOW DATABASES` and the container is
|
||||
then skipped (this happened once and went unnoticed for months).
|
||||
- PostgreSQL: reachable as the `postgres` system user (peer auth).
|
||||
- PostgreSQL in Docker: the container's image name must contain
|
||||
"postgres" (official images do).
|
||||
|
||||
A database you do *not* want dumped (huge, static, re-importable)
|
||||
goes into `/root/scripts/db-exclude` as `<instance>/<db>` — every
|
||||
skip is logged, so the list cannot rot silently.
|
||||
|
||||
## 5. Verify, once
|
||||
|
||||
```sh
|
||||
cd /root/scripts && git pull
|
||||
/root/scripts/restic-backup.sh -s dumps # ~2 min: see your DB in the log
|
||||
ls /backup/dumps/mariadb/myservice/ # dump present?
|
||||
/root/scripts/restic-backup.sh # full run, then:
|
||||
. /root/.restic-env
|
||||
restic -r s3:s3.sbg.io.cloud.ovh.net/restic-data ls latest | grep myservice | head
|
||||
```
|
||||
|
||||
Seeing your paths and dumps in that listing means the service is
|
||||
protected. Do this the day you create the service, not the day you
|
||||
need it.
|
||||
|
||||
## 6. Pitfalls (learned the hard way)
|
||||
|
||||
- **`/tmp` on ks4 is a 16 GiB tmpfs — it is RAM.** Never restore, dump
|
||||
or stage data there; use `/backup/…`.
|
||||
- **Avoid millions of tiny files.** A service that keeps every version
|
||||
of everything (seafile did, 24.9 M objects) makes every backup and
|
||||
every filesystem walk take days. If the upstream software has a
|
||||
garbage-collection or retention setting, turn it on *and schedule
|
||||
it* the day you deploy — see [ks4/seafile-gc.md](ks4/seafile-gc.md).
|
||||
- **Keep data under one predictable path** per service; scattered data
|
||||
means several `restic-paths` lines and things get forgotten.
|
||||
- **Static IP + a note in the service doc**; two containers fighting
|
||||
over one IP is a confusing outage.
|
||||
- Cron on ks4 does not have `/usr/local/bin` in its `PATH` — use
|
||||
absolute paths in anything you schedule.
|
||||
|
||||
## 7. Checklist
|
||||
|
||||
- [ ] container created from a Debian/Ubuntu image, static IP noted
|
||||
- [ ] no Docker (or a written reason + volume paths)
|
||||
- [ ] install/config steps written in `doc/ks4/<service>.md`
|
||||
- [ ] exposed through gateway (HTTP) or a proxy device (other)
|
||||
- [ ] snapshots left at the profile default
|
||||
- [ ] data paths added to `restic-paths` (committed to git)
|
||||
- [ ] `restic-backup.sh` run once; dumps and paths verified in the repo
|
||||
- [ ] service added to the table in [ks4/install.md](ks4/install.md)
|
||||
@@ -0,0 +1,45 @@
|
||||
# Homelab: nuc
|
||||
|
||||
Incus host on the LAN — **on-demand**: since 2026-08-30 it only needs to
|
||||
run when watching Jellyfin or using the Spotify Connect kiosk. Everything
|
||||
always-on (blocky/DNS, privoxy, transmission-bt) moved to
|
||||
[`nas`](../nas/README.md), which is why nuc can now be powered off.
|
||||
|
||||
⚠️ Powering nuc off has backup consequences — its 03:30 replication to
|
||||
nas only runs while it is up. See [nas-install.md](../nas/nas-install.md) §9.
|
||||
|
||||
- Debian 13, Intel Alder Lake-N (iGPU `i915`, shared by both Jellyfin containers) —
|
||||
bare-metal reinstall: [nuc-install.md](nuc-install.md)
|
||||
- Instances are bridged onto the LAN (192.168.0.0/24)
|
||||
- Storage: ZFS pool `data` on the SSD (all instance disks). The USB
|
||||
4 TB and its pool **left nuc on 2026-08-30** — the enclosure's JMicron
|
||||
bridge was suspending the pool ([usb4t-dropouts.md](usb4t-dropouts.md));
|
||||
the disk now sits on direct SATA in [`nas`](../nas/nas-install.md) as
|
||||
pool `tank`.
|
||||
- Media library: `/srv/media` is now an **NFSv4 mount from nas**
|
||||
(`192.168.0.4:/media`, read-only). `jellyfin-server` stays here for
|
||||
the iGPU and reads it with `shift=false`
|
||||
([jellyfin-server.md](jellyfin-server.md)).
|
||||
- Backups: nuc and nas **cross-replicate** — nuc pushes all its
|
||||
instances to `nas:nucbackup` at 03:30, nas pushes its own to
|
||||
`nuc:nasbackup` at 04:00, so neither host's instances depend on that
|
||||
host surviving ([nas/nas-install.md](../nas/nas-install.md) §9).
|
||||
The ks4 pull leg moved to nas as well
|
||||
([ks2/nas-seed.md](../ks2/nas-seed.md)), so nuc no longer runs a
|
||||
WireGuard tunnel.
|
||||
|
||||
- ⚠️ **Never reboot nuc with the display powered on** — the i915 probe
|
||||
dies and takes every container with it (including LAN DNS). Symptoms
|
||||
and fix: [jellyfin-client.md](jellyfin-client.md#-never-boot-nuc-with-the-display-active-2026-08-30)
|
||||
|
||||
## Instances
|
||||
|
||||
| Name | IP | Doc | Features |
|
||||
|---|---|---|---|
|
||||
| [jellyfin-server](jellyfin-server.md) | 192.168.0.5 | ✅ | unprivileged, autostart; iGPU render node (`gpu` device, render gid) for QSV/VAAPI transcoding; `/srv/media` disk device (NFS from nas, `shift=false`, read-only); proxy device → host :8096 |
|
||||
| [jellyfin-client](jellyfin-client.md) | 192.168.0.6 | ✅ | **privileged**, autostart; full iGPU (`gpu` device, gid 44) → HDMI kiosk (cage + Jellyfin Media Player); custom `raw.lxc` (bind `/dev/snd`, `/dev/input`, host `/run/udev`); Pioneer USB audio as ALSA default; FR keymap; go-librespot Spotify Connect ("Pioneer A-70") |
|
||||
| homeassistant | (stopped) | — | **virtual machine**, 50 GiB root disk on pool `data` |
|
||||
|
||||
Moved to [`nas`](../nas/README.md) on 2026-08-30: `transmission-bt`
|
||||
(next to the media dataset it writes to), plus `blocky` and `privoxy`
|
||||
so LAN DNS and the proxy survive nuc being shut down.
|
||||
+162
-1
@@ -116,6 +116,15 @@ EOF'
|
||||
# --- Kiosk user + seat management ----------------------------------------------
|
||||
incus exec "$CNAME" -- bash -c 'id kiosk >/dev/null 2>&1 || useradd -m -G video,render,input,audio kiosk'
|
||||
|
||||
# ⚠️ Group names are NOT enough. The host (Debian) and the container
|
||||
# (Ubuntu) allocate dynamic system gids independently, so the container's
|
||||
# `input` group does not necessarily have the same gid as the group that
|
||||
# owns /dev/input/* on the host. Bind the kiosk user to the *numeric*
|
||||
# host gid, whatever it is called inside:
|
||||
HOST_INPUT_GID="$(stat -c %g /dev/input/event0)"
|
||||
incus exec "$CNAME" -- usermod -aG "$HOST_INPUT_GID" kiosk
|
||||
incus exec "$CNAME" -- id kiosk # must list $HOST_INPUT_GID
|
||||
|
||||
# In a container seatd must NOT bind the seat to a VT (there is no usable VT;
|
||||
# it would try to open the host's active tty and hang the compositor forever).
|
||||
incus exec "$CNAME" -- mkdir -p /etc/systemd/system/seatd.service.d
|
||||
@@ -245,10 +254,17 @@ it works regardless of which USB port the dongle lands on.
|
||||
`/etc/udev/rules.d/99-jellyfin-kiosk-recover.rules`:
|
||||
|
||||
```
|
||||
# Logitech Unifying receiver (re)plugged -> recover the jellyfin-client kiosk
|
||||
# A Logitech receiver was (re)plugged -> recover the jellyfin-client kiosk.
|
||||
# c52b = Unifying receiver (K400); c539 = Lightspeed receiver (G603).
|
||||
ACTION=="add", SUBSYSTEM=="usb", ENV{DEVTYPE}=="usb_device", ATTR{idVendor}=="046d", ATTR{idProduct}=="c52b", RUN+="/usr/bin/systemctl --no-block start jellyfin-kiosk-recover.service"
|
||||
ACTION=="add", SUBSYSTEM=="usb", ENV{DEVTYPE}=="usb_device", ATTR{idVendor}=="046d", ATTR{idProduct}=="c539", RUN+="/usr/bin/systemctl --no-block start jellyfin-kiosk-recover.service"
|
||||
```
|
||||
|
||||
⚠️ The match is per product ID, so **a receiver not listed here will not
|
||||
auto-recover** — the kiosk must be restarted by hand after plugging it in
|
||||
(`incus exec jellyfin-client -- systemctl restart jellyfin-kiosk`). Add
|
||||
the new id here when introducing different input hardware.
|
||||
|
||||
`/etc/systemd/system/jellyfin-kiosk-recover.service` — oneshot, so a burst of
|
||||
udev events during one plug merges into a single restart (natural debounce):
|
||||
|
||||
@@ -285,6 +301,134 @@ udevadm control --reload-rules && systemctl daemon-reload
|
||||
journalctl -t jellyfin-kiosk-recover -f
|
||||
```
|
||||
|
||||
## ⚠️ Never boot nuc with the display active (2026-08-30)
|
||||
|
||||
**Symptom:** after a reboot, *no* Incus container starts. `incus list`
|
||||
answers but `incus start <anything>` hangs forever, `systemctl status
|
||||
incus` sits in `activating (start-post)`, and LAN DNS is down because
|
||||
blocky never came up. Nothing in the incus logs explains it.
|
||||
|
||||
**Cause — nothing to do with incus.** If the TV/projector is connected
|
||||
**and powered on** when nuc boots, firmware hands i915 an already-lit
|
||||
pipe. The driver's state readback then trips a series of warnings and
|
||||
the probe never completes:
|
||||
|
||||
```
|
||||
drm_WARN_ON(!pll_active) intel_ddi.c:4019 intel_ddi_get_clock
|
||||
drm_WARN_ON(p0 == 0 || p1 == 0 || p2 == 0) intel_dpll_mgr.c:2878
|
||||
drm_WARN_ON(pixel_rate == 0) skl_watermark.c:1729
|
||||
```
|
||||
(all inside `intel_modeset_setup_hw_state` ← `intel_display_driver_probe_nogem`)
|
||||
|
||||
The cascade:
|
||||
|
||||
1. i915 probe dies → **`/dev/dri` never appears** (no GPU at all)
|
||||
2. `snd_hda_intel` waits forever for i915's audio component →
|
||||
permanent **deferred probe** holding the PCI device lock on `0000:00:1f.3`
|
||||
3. incusd reads that device's `sriov_numvfs` while enumerating
|
||||
resources → blocks in **D state** → the daemon never signals ready,
|
||||
so nothing autostarts and every `incus start` hangs
|
||||
|
||||
**Reproduced identically on 6.12.107 and 6.12.105** — it is the display
|
||||
path, not a kernel regression. Do not waste time pinning kernels.
|
||||
|
||||
**Diagnosis, in order:**
|
||||
|
||||
```sh
|
||||
ls /dev/dri/ # empty = i915 probe failed
|
||||
cat /sys/kernel/debug/devices_deferred # snd_hda_intel entry = the deadlock
|
||||
ps -eLo pid,tid,stat,wchan:26,comm | awk '$3 ~ /D/' # incusd in sriov_numvfs_show
|
||||
dmesg -T | grep -E 'drm_WARN_ON|deferred probe pending'
|
||||
```
|
||||
|
||||
**Fix:** disconnect HDMI (or power the display fully off — not standby),
|
||||
reboot, then **hotplug the cable back in**. Connecting after boot goes
|
||||
through normal connector detection instead of firmware state readback
|
||||
and works fine.
|
||||
|
||||
In-place recovery is *not* possible: `modprobe -r i915` fails (module in
|
||||
use by the wedged probe) and `modprobe i915` times out. A reboot is the
|
||||
only way out.
|
||||
|
||||
**After hotplugging, restart the kiosk** — `cage` started with zero
|
||||
outputs and will not pick the display up on its own:
|
||||
|
||||
```sh
|
||||
incus exec jellyfin-client -- systemctl restart jellyfin-kiosk
|
||||
cat /sys/class/drm/card0-HDMI-A-2/status # expect: connected
|
||||
```
|
||||
|
||||
Note the HDA controller this wedges is only used for **HDMI audio**,
|
||||
which this setup does not use — audio goes to the Pioneer USB DAC via
|
||||
`/etc/asound.conf`. It is pure collateral damage, but it takes the whole
|
||||
host down with it.
|
||||
|
||||
## Input gid mismatch — latent, fix it anyway (2026-08-30)
|
||||
|
||||
> ⚠️ **This was not the cause of the 2026-08-30 outage.** That turned out
|
||||
> to be a flat/switched-off K400 — a G603 on the same port and the same
|
||||
> `event0` worked immediately. The mismatch below is real and worth
|
||||
> correcting, but with `LIBSEAT_BACKEND=seatd` it is **seatd (running as
|
||||
> root) that opens input devices** and passes the fd to cage, so the
|
||||
> kiosk user's group membership is not on the critical path for input.
|
||||
> Fix it for the direct-open fallback path, not as a debugging lead.
|
||||
>
|
||||
> **Before suspecting software, prove the hardware emits anything:**
|
||||
> ```sh
|
||||
> timeout 60 cat /dev/input/eventN | wc -c # press keys; 0 bytes = nothing reached the kernel
|
||||
> ```
|
||||
> That one check would have saved an hour.
|
||||
>
|
||||
> **Dead K400 batteries are invisible from the host.** This K400 exposes
|
||||
> no `hidpp_battery_*` node under `/sys/class/power_supply/`, so charge
|
||||
> cannot be read. Worse, the receiver still lists the keyboard as a paired
|
||||
> peer (`0003:046D:4024.*` under the `C52B` receiver) whether or not it is
|
||||
> awake, and `/dev/input/event0` plus a `Logitech K400` entry in
|
||||
> `/proc/bus/input/devices` are present either way — so every software
|
||||
> check looks perfectly healthy. **Zero bytes from the raw capture is the
|
||||
> only signal.** Swapping in a different receiver on the same port is the
|
||||
> quickest A/B confirmation.
|
||||
|
||||
**Symptom (if it ever does bite):** kiosk renders but input does nothing,
|
||||
with no errors — `WLR_LIBINPUT_NO_DEVICES=1` keeps cage alive rather than
|
||||
failing loudly.
|
||||
|
||||
**Cause:** `/dev/input/*` is `crw-rw---- root:<host input gid>`. The
|
||||
install script adds `kiosk` to the group *named* `input` inside the
|
||||
container, but Debian (host) and Ubuntu (container) allocate dynamic
|
||||
system gids independently:
|
||||
|
||||
```
|
||||
host /dev/input/event0 gid 996 -> "input"
|
||||
container "input" group gid 995 <- kiosk was here
|
||||
container gid 996 -> "systemd-timesync"
|
||||
```
|
||||
|
||||
So the kiosk user was in the wrong group and could not open any input
|
||||
device. The udev side was fine — check it first to rule it out:
|
||||
`cat /run/udev/data/c13:64` should show `E:ID_INPUT=1` etc.
|
||||
|
||||
**Diagnose:**
|
||||
|
||||
```sh
|
||||
stat -c '%n %a %u:%g' /dev/input/event0 # host gid
|
||||
incus exec jellyfin-client -- id kiosk # does it include that gid?
|
||||
# open() test — do NOT use `head`/`cat`, reading an event device blocks
|
||||
# with no pending events and looks like a permission failure:
|
||||
incus exec jellyfin-client -- su -s /bin/bash kiosk -c 'exec 3< /dev/input/event0 && echo OPEN_OK'
|
||||
```
|
||||
|
||||
**Fix** (persists in the container's `/etc/group`):
|
||||
|
||||
```sh
|
||||
HOST_INPUT_GID="$(stat -c %g /dev/input/event0)"
|
||||
incus exec jellyfin-client -- usermod -aG "$HOST_INPUT_GID" kiosk
|
||||
incus exec jellyfin-client -- systemctl restart jellyfin-kiosk
|
||||
```
|
||||
|
||||
Re-check this after any host reinstall — the host's `input` gid is
|
||||
dynamically allocated and can come back different.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
```sh
|
||||
@@ -303,6 +447,23 @@ incus exec jellyfin-client -- udevadm info /dev/input/event0 # udev db visibl
|
||||
visible (#2); check `run-udev.mount` is active.
|
||||
- **Video OK, no sound; JMP log shows `AO: [null]`** → ALSA default broken
|
||||
(#3); check `/etc/asound.conf` and the card name in `aplay -l`.
|
||||
**Most common cause: the Pioneer DAC is simply switched off.**
|
||||
`/etc/asound.conf` pins the default to it *by card name* (`Device`), so
|
||||
with the amp off the name does not exist and `default` fails to open —
|
||||
mpv then falls back to null silently: picture, no sound, no error.
|
||||
One-line check before blaming anything else:
|
||||
```sh
|
||||
incus exec jellyfin-client -- su -s /bin/bash kiosk -c 'aplay -D default -d 1 /usr/share/sounds/alsa/Front_Center.wav'
|
||||
```
|
||||
`audio open error: No such device` = amp is off. Power it on; no
|
||||
restart needed, JMP opens the device per playback.
|
||||
- **Display hotplugged after boot → cage exits once, then recovers by
|
||||
itself.** If the kiosk started with no outputs, plugging the HDMI in
|
||||
makes cage fail (`Failed with result 'exit-code'`); the unit's
|
||||
`Restart=on-failure` / `RestartSec=5` restarts it ~5 s later, this time
|
||||
with the display present. **Do not restart it by hand** — check
|
||||
`systemctl show jellyfin-kiosk -p ActiveEnterTimestamp` first and only
|
||||
intervene if the timestamp predates the hotplug. Verified 2026-08-31.
|
||||
- **Keyboard plugged in after boot isn't seen** — the host udev db is live
|
||||
through the bind, but udev hotplug *events* don't cross the container's
|
||||
network namespace, so cage only enumerates at startup. Re-plugging the
|
||||
|
||||
+81
-2
@@ -5,8 +5,10 @@ Jellyfin media **server** in an unprivileged Incus container on `nuc`.
|
||||
- Image: `images:ubuntu/24.04`, Jellyfin from the official repo (repo.jellyfin.org)
|
||||
- IP: `192.168.0.5` (LAN bridge) — web UI/API on `http://192.168.0.5:8096`
|
||||
- iGPU render node (`/dev/dri/renderD128`) passed for QSV/VAAPI hardware transcoding
|
||||
- Media library: host `/srv/media` (ZFS dataset `usb4t/media`, USB 4 TB)
|
||||
mounted at `/media` with `shift=true` (needs ZFS ≥ 2.2 for idmapped mounts)
|
||||
- Media library: host `/srv/media` — since 2026-08-30 an **NFSv4 mount
|
||||
from `nas`** (`192.168.0.4:/media`, dataset `tank/media`), mounted at
|
||||
`/media` in the container with **`shift=false`** and `readonly=true`.
|
||||
See [nas/nas-install.md](../nas/nas-install.md) §8.
|
||||
- Port 8096 additionally proxied to the host address (`web` proxy device)
|
||||
|
||||
## Install script
|
||||
@@ -77,6 +79,83 @@ incus restart "$CNAME"
|
||||
echo "Done. Open http://<host-ip>:8096 to run the setup wizard."
|
||||
```
|
||||
|
||||
## Media over NFS (2026-08-30)
|
||||
|
||||
The library moved to `nas` when the 4 TB left nuc's USB enclosure. The
|
||||
container keeps the same path, so everything below still applies — only
|
||||
the mount underneath `/srv/media` changed.
|
||||
|
||||
```sh
|
||||
# nuc host: /etc/fstab
|
||||
192.168.0.4:/media /srv/media nfs4 ro,_netdev,soft,timeo=100,retrans=3 0 0
|
||||
```
|
||||
|
||||
`shift=true` **cannot** be used: idmapped mounts are not supported on
|
||||
NFS (nor CIFS). Per the troubleshooting note below, dropping the shift is
|
||||
enough for a read-only library — the export uses `all_squash` so files
|
||||
carry synthetic world-readable ownership:
|
||||
|
||||
```sh
|
||||
incus stop jellyfin-server # shift cannot be hot-applied
|
||||
incus config device set jellyfin-server media shift=false
|
||||
incus config device set jellyfin-server media readonly=true
|
||||
incus start jellyfin-server
|
||||
```
|
||||
|
||||
⚠️ Two traps:
|
||||
|
||||
- **Boot ordering.** Add `remote-fs.target` to nuc's
|
||||
`/etc/systemd/system/incus.service.d/after-zfs.conf`, or the container
|
||||
starts against an empty mountpoint and Jellyfin shows an empty library
|
||||
(and may prune the library metadata).
|
||||
- **`soft` is deliberate.** A hung nas should fail Jellyfin's reads, not
|
||||
wedge nuc's processes in uninterruptible sleep the way the suspended
|
||||
`usb4t` pool did ([usb4t-dropouts.md](usb4t-dropouts.md)).
|
||||
|
||||
`transmission-bt` is no longer on nuc — it moved to nas and writes to
|
||||
the dataset locally ([nas/transmission-bt.md](../nas/transmission-bt.md)),
|
||||
so nuc's mount is read-only and there is exactly one writer.
|
||||
|
||||
## ⚠️ Real-time monitoring does not work over NFS (2026-08-31)
|
||||
|
||||
Libraries have `EnableRealtimeMonitor=true` and Jellyfin reports
|
||||
`SupportsLibraryMonitor: true`, but **new files never appear on their
|
||||
own**. Jellyfin watches with inotify, which only reports changes made
|
||||
through the local mount; transmission writes them on **nas**, so nuc's
|
||||
NFS client sees nothing. Jellyfin looks healthy and silently misses
|
||||
everything until a scan.
|
||||
|
||||
This is a regression from the 2026-08-30 storage move — before it,
|
||||
transmission and jellyfin-server shared one local dataset on nuc and
|
||||
inotify fired normally.
|
||||
|
||||
**Fix in place:** transmission calls Jellyfin's `/Library/Refresh` when a
|
||||
download completes — see
|
||||
[nas/transmission-bt.md](../nas/transmission-bt.md). Downloads appear
|
||||
within seconds; if nuc is powered off the request fails harmlessly and
|
||||
the scheduled scan catches up.
|
||||
|
||||
Manual scan (UI): Dashboard → Scheduled Tasks → **Scan Media Library**.
|
||||
By API:
|
||||
|
||||
```sh
|
||||
curl -X POST -H "X-Emby-Token: <key>" http://192.168.0.5:8096/Library/Refresh # expect 204
|
||||
```
|
||||
|
||||
Diagnosing "my download is not in Jellyfin", in order — the first three
|
||||
were all fine when this was hit, which is what made it confusing:
|
||||
|
||||
```sh
|
||||
ls /export/media/downloads/ # on nas: file there?
|
||||
incus exec jellyfin-server -- ls /media/downloads/ # visible through NFS?
|
||||
incus exec jellyfin-server -- find /var/lib/jellyfin/root -name '*.mblink' -exec cat {} + # in a library path?
|
||||
incus exec jellyfin-server -- cat /var/lib/jellyfin/data/ScheduledTasks/*.js | grep -o '"Name":"Scan Media Library".*' # when did it last scan?
|
||||
```
|
||||
|
||||
⚠️ **nuc's mount is read-only.** Reorganising finished downloads into
|
||||
`/media/movies` or `/media/tv-shows` can no longer be done from nuc — do
|
||||
it on nas under `/export/media/`.
|
||||
|
||||
## First-run configuration
|
||||
|
||||
1. Run the setup wizard; add libraries pointing at `/media/...`.
|
||||
|
||||
+63
-13
@@ -12,14 +12,14 @@ How to rebuild the Incus host from scratch if `/dev/sda` (512 GB SSD,
|
||||
`usb4t/media` → `/srv/media` (media library)
|
||||
- USB: Pioneer USB audio (`08e4:0176`), Logitech Unifying receiver (K400),
|
||||
CSCTEK USB Audio and HID
|
||||
- NIC: `enp1s0` (static `192.168.0.3/24`, gw `192.168.0.2`)
|
||||
- NIC: `enp1s0` (static `192.168.0.3/24`, gw `192.168.0.1`)
|
||||
|
||||
## ⚠️ What dies with sda
|
||||
|
||||
The ZFS pool `data` lives on `sda5` → **all instance root disks are lost**
|
||||
with the OS. The USB pool `usb4t` (backups + media) survives — all
|
||||
instances are replicated onto it with
|
||||
[`scripts/incus-copy.sh`](../scripts/incus-copy.sh) (deployed at
|
||||
[`scripts/incus-copy.sh`](https://git.lutran.fr/julien/scripts/src/branch/main/incus-copy.sh) (deployed at
|
||||
`/root/scripts/` on nuc; set up 2026-08-09):
|
||||
|
||||
```sh
|
||||
@@ -31,7 +31,7 @@ incus project create backup -c features.images=false -c features.profiles=false
|
||||
/root/scripts/incus-copy.sh -p backup -s nucbackup
|
||||
```
|
||||
|
||||
Runs nightly via `/etc/cron.d/incus-copy` at **03:30** (30 min after
|
||||
Runs nightly from root's crontab at **03:30** (30 min after
|
||||
the profile-scheduled 03:00 instance snapshots, so VM refreshes stay
|
||||
incremental), logging to `/var/log/incus-copy.log` (logrotate:
|
||||
`/etc/logrotate.d/incus-copy`). Note the script's `flock` is global:
|
||||
@@ -87,7 +87,7 @@ allow-hotplug enp1s0
|
||||
iface enp1s0 inet static
|
||||
address 192.168.0.3
|
||||
netmask 255.255.255.0
|
||||
gateway 192.168.0.2
|
||||
gateway 192.168.0.1
|
||||
dns-nameservers 1.1.1.1 9.9.9.9
|
||||
```
|
||||
|
||||
@@ -165,13 +165,56 @@ hosts reach them normally. (So test a container's LAN service from inside
|
||||
the container or from an external LAN host — never by pinging its IP from
|
||||
the nuc or a sibling container; that always fails by design.)
|
||||
|
||||
LAN gateway note: the router/gateway is **`192.168.0.2`** (migrated from
|
||||
`192.168.0.1`, 2026-08 — `.1` is gone). DHCP-configured instances pick the
|
||||
new gateway up automatically; **statically-configured ones must be updated
|
||||
by hand.** Current static holdouts: privoxy
|
||||
(`/etc/systemd/network/eth0.network`, `Gateway=`) and transmission-bt
|
||||
(netplan `routes: via:` + the WG kill-switch `/32`). Symptom of a missed
|
||||
one: the service is up and its port answers, but nothing it fetches works.
|
||||
LAN gateway note: the router/gateway is **`192.168.0.1`** — the FTTH box,
|
||||
since 2026-09 (it was `.2`, the Archer C7, from 2026-08; and `.1` before
|
||||
that). **Every host and instance is statically configured, so each one
|
||||
must be updated by hand.** Symptom of a missed one: the service is up and
|
||||
its port answers, but nothing it fetches works.
|
||||
|
||||
Every LAN host and instance is now **statically configured** (verified
|
||||
2026-09-16) — nothing on this LAN depends on a DHCP reservation any more.
|
||||
On a gateway change, update all of these by hand:
|
||||
|
||||
| Where | File | Address |
|
||||
|---|---|---|
|
||||
| nas host | `/etc/network/interfaces`, `gateway` | `.4` |
|
||||
| nuc host | `/etc/network/interfaces`, `gateway` | `.3` |
|
||||
| blocky | `/etc/systemd/network/eth0.network`, `Gateway=` | `.254` |
|
||||
| privoxy | `/etc/systemd/network/eth0.network`, `Gateway=` | `.11` |
|
||||
| transmission-bt | netplan `routes: via:` (WG kill-switch `/32`) | `.7` |
|
||||
| jellyfin-server | netplan `routes: - to: default / via:` | `.5` |
|
||||
| jellyfin-client | `/etc/systemd/network/10-eth0.network`, `Gateway=` | `.6` |
|
||||
|
||||
`homeassistant` (a HAOS **VM**, NetworkManager, normally stopped) is
|
||||
deliberately left on DHCP — it never had a reservation and nothing
|
||||
addresses it by IP.
|
||||
|
||||
⚠️ **Why everything is static now: DHCP reservations did not survive the
|
||||
FTTH migration.** They lived in the Archer C7's `dhcp.@host[-1]` list
|
||||
(the `add_host` block in
|
||||
[../archer-c7/upgrade-openwrt-25.12.md](../archer-c7/upgrade-openwrt-25.12.md)),
|
||||
and the FTTH box did not inherit them. blocky held `192.168.0.254` that
|
||||
way; renewing its lease handed it a pool address and took LAN DNS down
|
||||
with it. Static config removes the dependency entirely.
|
||||
|
||||
Two **non-container** hosts also lost their reservations and are still
|
||||
dynamic — harmless, nothing addresses them by IP, but the old fixed
|
||||
addresses are gone: `LAPTOP719974` (was `.20`) and `patate` (was `.21`).
|
||||
|
||||
⚠️ **`jellyfin-client` cannot use netplan at all.** It is a privileged
|
||||
kiosk whose `raw.lxc` bind-mounts the host's `/run/udev` read-only, so
|
||||
`netplan generate` dies with `cannot create directory /run/udev/rules.d`
|
||||
— which means netplan changes there **silently fail to regenerate at
|
||||
boot**. It is configured with plain systemd-networkd
|
||||
(`/etc/systemd/network/10-eth0.network`); its old netplan yaml is parked
|
||||
at `/root/10-lxc.yaml.netplan-disabled-ftth`. Always verify a network
|
||||
change in that container with `incus restart jellyfin-client`, not just
|
||||
`netplan apply`.
|
||||
|
||||
**Fallback hardware:** the Archer C7 and the LTE box are kept on the
|
||||
shelf. Their addressing does not clash with the current LAN — **the
|
||||
gateway is the only thing that differs**, so failing back means walking
|
||||
the table above and setting `.2` (C7) instead of `.1`.
|
||||
|
||||
Let `julien` run harmless incus commands (list/info/config/show…)
|
||||
without a password — mutating ones (`exec`, `start/stop`, `delete`)
|
||||
@@ -245,8 +288,8 @@ once `usb4t` is imported.)
|
||||
| When | What | Where |
|
||||
|-------|------|-------|
|
||||
| 03:00 | instance snapshots (`snapshots.schedule` on the default profile, expiry 7d) | incus |
|
||||
| 03:30 | replicate all instances to the USB pool (`incus-copy.sh -p backup -s nucbackup`) | `/etc/cron.d/incus-copy` → `/var/log/incus-copy.log` |
|
||||
| 05:00 | apt dist-upgrade all running containers (`incus-container-upgrade.sh`; VMs and non-apt containers skipped; jellyfin pinned to the 10.11 series in-container) | `/etc/cron.d/incus-container-upgrade` → `/var/log/incus-container-upgrade.log` |
|
||||
| 03:30 | replicate all instances to the USB pool (`incus-copy.sh -p backup -s nucbackup`) | root crontab → `/var/log/incus-copy.log` |
|
||||
| 05:00 | apt dist-upgrade all running containers (`incus-container-upgrade.sh`; VMs and non-apt containers skipped; jellyfin pinned to the 10.11 series in-container) | root crontab → `/var/log/incus-container-upgrade.log` |
|
||||
|
||||
The ordering is deliberate: snapshot → backup → upgrade, so a broken
|
||||
upgrade is always one snapshot-restore away and the replicas predate it.
|
||||
@@ -265,6 +308,13 @@ Both logs rotate monthly (`/etc/logrotate.d/incus-*`).
|
||||
the install scripts in the per-container docs include it)
|
||||
- [ ] Host boots to `multi-user.target`, nothing grabs the GPU
|
||||
(required by the jellyfin-client kiosk)
|
||||
- [ ] ⚠️ **Reboot with the TV/projector disconnected or powered off.**
|
||||
Booting with the display active kills the i915 probe, which wedges
|
||||
`snd_hda_intel` in a deferred probe, which blocks incusd in
|
||||
`sriov_numvfs_show` — **no container starts at all, including
|
||||
blocky/DNS**. Hotplug the cable back after boot, then
|
||||
`incus exec jellyfin-client -- systemctl restart jellyfin-kiosk`.
|
||||
Full diagnosis: [jellyfin-client.md](jellyfin-client.md)
|
||||
- [ ] Jellyfin web at `http://192.168.0.5:8096`, kiosk UI on HDMI,
|
||||
sound on the Pioneer, "Pioneer A-70" visible in Spotify Connect
|
||||
- [ ] LAN DNS: clients use blocky at `192.168.0.254` (host itself uses
|
||||
|
||||
@@ -0,0 +1,156 @@
|
||||
# usb4t: USB dropouts suspend the pool (2026-08) — RESOLVED
|
||||
|
||||
> **Outcome (2026-08-30): the disk moved off USB entirely.** It now
|
||||
> runs on **direct SATA** in the new host `nas`
|
||||
> ([nas/README.md](../nas/README.md)), as pool `tank`, and the ks4
|
||||
> pull leg plus its WireGuard tunnel moved with it
|
||||
> ([ks2/nas-seed.md](../ks2/nas-seed.md)). Everything below is the
|
||||
> investigation that led there — worth keeping for the diagnosis
|
||||
> method and for the alerting gap it exposed, which applies to any
|
||||
> host.
|
||||
|
||||
Symptom seen first in the nightly backup log
|
||||
(`/var/log/incus-copy.log`, job in root's crontab): every
|
||||
instance fails with
|
||||
|
||||
```
|
||||
Error: Refresh instance: Create instance volume from copy failed:
|
||||
Volume exists in database but not on storage
|
||||
```
|
||||
|
||||
That message is misleading — incus is fine. The `usb4t` pool underneath
|
||||
was **SUSPENDED**, so the replica volumes incus has in its database had
|
||||
no storage behind them.
|
||||
|
||||
## Diagnosis (2026-08-28)
|
||||
|
||||
The disk is healthy, the USB link is not:
|
||||
|
||||
- `smartctl -a -d sat /dev/sdc`: **PASSED**, 0 reallocated, 0 pending,
|
||||
0 offline-uncorrectable, **0 UDMA CRC errors**, 463 power-on hours.
|
||||
- `dmesg`: `usb 4-3: USB disconnect` followed by immediate
|
||||
re-enumeration — the enclosure drops off the bus and comes back as a
|
||||
new device. Bridge is **JMicron 152d:0578** (the kernel already
|
||||
disables UAS for it and applies quirks).
|
||||
- **61 USB disconnects in 30 days**; pool suspensions on Aug 17
|
||||
(resilver), 19, 21, 27 (×2) and 28.
|
||||
- While suspended, anything touching the pool hangs — the journal
|
||||
shows tasks blocked for 1000+ seconds.
|
||||
|
||||
Blast radius each time: nuc's own replicas (`usb4t/backup/nuc`) stop
|
||||
being refreshed, `/srv/media` disappears from the Jellyfin containers,
|
||||
and — once seeded — the ks4 pull leg (`usb4t/backup/ks4`) would stop
|
||||
too. **The pool holding the off-site copy of ks4 must not be the least
|
||||
reliable device in the setup.**
|
||||
|
||||
## Recovery
|
||||
|
||||
```sh
|
||||
zpool clear usb4t # device is back; ZFS resumes IO
|
||||
zpool status -v usb4t # lists files damaged by the interrupted writes
|
||||
zpool scrub usb4t # validate the whole pool
|
||||
```
|
||||
|
||||
If `clear` hangs: `zpool export usb4t && zpool import usb4t`, reboot as
|
||||
a last resort. Datasets remount by themselves once the pool resumes.
|
||||
|
||||
Damage from the 2026-08-28 incident, as first reported: two
|
||||
pool-metadata objects (`<metadata>:<0x0>`, `<metadata>:<0x3d>`) and
|
||||
one media file. **The scrub cleared all of it** — 1.08 T verified in
|
||||
2 h 29 m, `0 B repaired, 0 errors`, and the error list is now
|
||||
`No known data errors`. Those entries were artefacts of the
|
||||
interrupted writes, not real corruption; nothing had to be restored.
|
||||
(One CKSUM count remains on the vdev — a single checksum ZFS fixed
|
||||
from its own redundant metadata copy.)
|
||||
|
||||
The scrub also served as a 2.5-hour soak test of the new cable:
|
||||
**zero USB disconnects** during it.
|
||||
|
||||
## Fixes, in order
|
||||
|
||||
1. ~~**Cable + port**~~ — replaced 2026-08-28 with a short quality
|
||||
cable straight into a rear USB-3 port, no hub. It survived a 2.5 h
|
||||
scrub, then **dropped again on 2026-08-29 16:22** (pool suspended,
|
||||
cleared 08-30 00:32). **Cable ruled out.**
|
||||
2. ~~**Disable USB autosuspend**~~ — **done, and ruled out**: the
|
||||
kernel cmdline carries `usb-storage.quirks=152d:0578:u` (UAS
|
||||
disabled for this bridge) and the device sits at
|
||||
`power/control=on`. It still dropped on 2026-08-29, so neither
|
||||
power management nor UAS explains this.
|
||||
3. **Replace the enclosure — now the prime candidate.** With the cable
|
||||
ruled out and the disk SMART-clean (0 reallocated/pending/CRC), the
|
||||
JMicron 152d:0578 bridge is what remains. An ASMedia-based
|
||||
enclosure (or a direct SATA connection) is the durable fix.
|
||||
⚠️ Until this is settled, this pool is not a trustworthy home for
|
||||
the ks4 off-site replica leg — treat it as a gate before seeding
|
||||
over FTTH.
|
||||
|
||||
## Alerting (why nobody noticed for 11 days)
|
||||
|
||||
`zfs-zed` was installed, enabled and running — but the host had **no
|
||||
MTA**, so its notifications went nowhere for 11 days. Fixed
|
||||
2026-08-28; the working chain is:
|
||||
|
||||
```
|
||||
zed → mail (bsd-mailx) → /usr/sbin/sendmail = msmtp → mail.lutran.fr:587 (auth, STARTTLS)
|
||||
```
|
||||
|
||||
- `apt install msmtp msmtp-mta bsd-mailx`
|
||||
- **`/etc/msmtprc` (mode 600) holds the SMTP account** — host
|
||||
`mail.lutran.fr`, port **587 (submission)**, user `zed@lutran.fr`,
|
||||
STARTTLS, `aliases /etc/aliases`.
|
||||
- `/etc/zfs/zed.d/zed.rc` (mode 600): `ZED_EMAIL_ADDR="julien@lutran.fr"`,
|
||||
`ZED_EMAIL_PROG="mail"`, `ZED_EMAIL_OPTS="-s '@SUBJECT@' @ADDRESS@"`,
|
||||
`ZED_NOTIFY_VERBOSE=1` (so scrub/resilver results are mailed too, not
|
||||
only failures), `ZED_NOTIFY_INTERVAL_SECS=3600`. Original kept as
|
||||
`zed.rc.orig`. **No credentials in this file** — they live only in
|
||||
`/etc/msmtprc`.
|
||||
|
||||
**Why nuc differs from ks2/ks4** (which mail via s-nail straight to
|
||||
port 25, and work fine): it is not the zed config, it is the source
|
||||
address. ks4 *is* the mail server's host (internal client,
|
||||
`mynetworks`); ks2 is a datacenter IP with a PTR. nuc sends from a
|
||||
**dynamic home IP with no PTR and no SPF alignment**, so the inbound
|
||||
MX path treats it like any stranger. Hence submission + auth here,
|
||||
deliberately different from the other two hosts.
|
||||
|
||||
Two dead ends worth remembering:
|
||||
|
||||
- **Port 25 is the wrong port for alerts.** Authentication succeeds,
|
||||
then rspamd rejects the message: `554 5.7.1 Spam message rejected` —
|
||||
a short mail from a dynamic home IP with no SPF alignment scores
|
||||
badly on the inbound MX path. Submission (587) does not apply those
|
||||
rules.
|
||||
- **The home IP has no PTR**, so postfix's reverse lookup delays the
|
||||
587 greeting by several seconds. Probes with short timeouts look
|
||||
like a firewall block (`lost connection after CONNECT, commands=0/0`
|
||||
in the server log) — give SMTP tests ≥30 s before concluding
|
||||
anything.
|
||||
|
||||
### ZED does not report everything — hence the watchdog
|
||||
|
||||
The 2026-08-29 suspension produced **no email**, although the mail path
|
||||
demonstrably worked (the scrub-finish notification arrived the evening
|
||||
before). Two reasons:
|
||||
|
||||
- a suspended pool keeps its vdev marked **ONLINE**, so
|
||||
`statechange-notify.sh` never fires;
|
||||
- the `data`-class events ZED *did* raise were muted because
|
||||
`ZED_NOTIFY_DATA` was unset (now `=1`).
|
||||
|
||||
Fixed 2026-08-30 by adding
|
||||
[`zpool-health.sh`](https://git.lutran.fr/julien/scripts/src/branch/main/zpool-health.sh),
|
||||
run every 15 min from root's crontab. It mails only on
|
||||
`healthy <-> problem` **transitions**, so it is silent in normal
|
||||
operation and cannot spam; `-t` sends a test. Worth deploying on ks4
|
||||
too — its `data` pool is single-disk and has the same blind spot.
|
||||
|
||||
Verify:
|
||||
|
||||
```sh
|
||||
/root/scripts/zpool-health.sh -m julien@lutran.fr -t
|
||||
tail -2 /var/log/msmtp.log # expect smtpstatus=250
|
||||
```
|
||||
|
||||
End-to-end proof: with `ZED_NOTIFY_VERBOSE=1`, the next scrub
|
||||
completion arrives as a mail by itself.
|
||||
Reference in New Issue
Block a user