From c8d759ce9ee0d4919c2e02370b5d91608297cacb Mon Sep 17 00:00:00 2001 From: Julien Lutran Date: Wed, 16 Sep 2026 13:37:56 +0200 Subject: [PATCH] doc: build the ks4 -> nas backup pull leg MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit FTTH is up, so the leg nas-seed.md had been holding since 2026-08-30 is now built: wg-ks4 on nas (10.8.0.22), nas peered on ks4's wireguard container, incus remote over the tunnel, 05:00 cron, and the first pass seeding under a systemd-run unit. Three corrections the runbook needed, all found by running it: - nas had no wireguard-tools at all. transmission-bt carries its own tunnel inside the container, so the host never needed them. - Every ks4 instance has an instance-level eth0 pinned to incusbr0 with a static 192.168.1.x, so each copy failed in under a second with "Cannot use manually specified ipv4.address when using unmanaged parent bridge". nas now runs a managed incusbr0 on 192.168.1.254/24 — deliberately not .1, which must keep resolving over wg-ks4. - The seed command was missing -p backup, which the doc's own verification step already assumed. Also records the measured rate: ~125 Mbit/s, ks4's OVH uplink rather than the home downlink, so ~31 h for the first pass — during which the shared incus-copy lock suppresses the 04:00 nasbackup job. Co-Authored-By: Claude Opus 5 (1M context) --- ks2/nas-seed.md | 80 +++++++++++++++++++++++++++++++++++++++------- nas/README.md | 16 +++++++++- nas/nas-install.md | 10 ++++-- 3 files changed, 90 insertions(+), 16 deletions(-) diff --git a/ks2/nas-seed.md b/ks2/nas-seed.md index 7f1f0c4..34dc96c 100644 --- a/ks2/nas-seed.md +++ b/ks2/nas-seed.md @@ -1,6 +1,9 @@ # ks4 pull leg — seed after FTTH -Status: **prepared, waiting on the FTTH link.** +Status: **built and seeding — 2026-09-16.** FTTH is up (gateway +`192.168.0.1`, see [nuc/nuc-install.md](../nuc/nuc-install.md)); the +tunnel, the incus remote and the cron are in place and the first pass is +running. Remaining: let the seed finish, then the test-restore below. ⚠️ **Changed 2026-08-30: this leg lands on `nas`, not on nuc.** It was originally designed for nuc's USB pool `usb4t`, but that pool proved to @@ -33,20 +36,28 @@ Consequences versus the original plan: ## Prerequisites -- [ ] `tank` running on SATA for ≥ 7 days with zero pool suspensions — - the gate that replaces "fix the USB enclosure" -- [ ] FTTH up (the first pass moves ~1.75 TiB) +- [x] `tank` running on SATA for ≥ 7 days with zero pool suspensions — + the gate that replaces "fix the USB enclosure". + Verified 2026-09-16: `ONLINE`, scrub clean 2026-09-14, uptime 2 w 2 d, + zero suspensions in the journal. +- [x] FTTH up (the first pass moves ~1.75 TiB) ## Setup (root on nas) ```sh -# 1. peer nas on ks4's wireguard container -# (run on ks4) — from /etc/wireguard/wg-ks4.key on nas +# 0. the nas host has no wireguard-tools — transmission-bt carries its own +# tunnel *inside* the container, so the host never needed them +apt-get install -y wireguard-tools + +# 1. key on nas, then peer it on ks4's wireguard container +umask 077; wg genkey > /etc/wireguard/wg-ks4.key +wg pubkey < /etc/wireguard/wg-ks4.key # -> +# (run on ks4) incus exec wireguard -- wg set wg0 peer allowed-ips 10.8.0.22/32 incus exec wireguard -- wg-quick save wg0 # 2. tunnel on nas: /etc/wireguard/wg-ks4.conf, modelled on nuc's -# Address = 10.8.0.22/32, peer pubkey TVs6d7…, +# Address = 10.8.0.22/24, peer pubkey TVs6d7…, # Endpoint = 193.70.35.17:51845, # AllowedIPs = 10.8.0.0/24, 192.168.1.1/32, keepalive 25 systemctl enable --now wg-quick@wg-ks4 @@ -56,14 +67,56 @@ incus remote add ks4 https://192.168.1.1:8443 --accept-certificate --token '…' incus list ks4: | head # sanity: remote reachable ``` -## Seed +⚠️ **nas needs a managed `incusbr0` of its own, or every copy fails +instantly.** All 18 ks4 instances carry an *instance-level* `eth0` +(`nictype: bridged`, `parent: incusbr0`, `ipv4.address: 192.168.1.x`). +nas has no such bridge, so instance creation dies with: + +``` +Device validation failed for "eth0": Cannot use manually specified +ipv4.address when using unmanaged parent bridge +``` + +Create a managed network of the same name — but **give it `.254`, never +`.1`**: `192.168.1.1` must keep resolving over `wg-ks4` to ks4's incus +API, and a local address always beats a route. ```sh -# full pull of every ks4 instance into pool ks4backup (tmux — first pass -# moves ~1.75 TiB through the WG tunnel) -/root/scripts/incus-copy.sh -r ks4 -s ks4backup 2>&1 | tee -a /var/log/incus-copy-ks4.log +incus network create incusbr0 \ + ipv4.address=192.168.1.254/24 ipv4.nat=false ipv6.address=none +ip route get 192.168.1.1 # must still say: dev wg-ks4 ``` +The bridge stays inert — replicas are never started here. + +## Seed + +⚠️ **`-p backup` is required**, exactly as for the nuc leg +([nas/nas-install.md](../nas/nas-install.md) §9a) — and as the +verification command below already assumed. Without it the 18 replicas +land in `default` alongside nas's live instances. + +Run it detached rather than in a shell that can drop — the first pass is +long (measured **~125 Mbit/s**, which is ks4's OVH *uplink*, not the home +downlink: **~31 h** for the full set): + +```sh +systemd-run --unit=ks4-seed --collect \ + /bin/bash -c '/root/scripts/incus-copy.sh -r ks4 -s ks4backup -p backup \ + >> /var/log/incus-copy-ks4.log 2>&1' + +systemctl is-active ks4-seed # progress: +tail -f /var/log/incus-copy-ks4.log +``` + +⚠️ **The seed suppresses the 04:00 `nasbackup` job while it runs.** +`incus-copy.sh` takes a single `/run/lock/incus-copy.lock` for every +shape, so any run starting while the seed holds it aborts with +`another incus-copy run holds …`. Over a ~31 h seed that skips one or two +nights of the nas-local copy — accepted; those replicas are small, +same-host, and rebuildable. The 03:30 nuc push is unaffected (it runs on +nuc, with nuc's own lock). + Notes: - First pass is a full send per instance; later refreshes are @@ -79,9 +132,12 @@ Add to nas's root crontab, offset from the 03:30 nuc→nas push, the 04:00 nas→nuc push and ks4's own 01:00/05:00 jobs: ```cron -0 5 * * * /root/scripts/incus-copy.sh -r ks4 -s ks4backup >> /var/log/incus-copy-ks4.log 2>&1 +0 5 * * * /root/scripts/incus-copy.sh -r ks4 -s ks4backup -p backup >> /var/log/incus-copy-ks4.log 2>&1 ``` +`/etc/logrotate.d/incus-copy` already lists `incus-copy-ks4.log`, so +nothing to add there. + ## Verification (release gate for ks2) ```sh diff --git a/nas/README.md b/nas/README.md index 82f0ad0..53fe20e 100644 --- a/nas/README.md +++ b/nas/README.md @@ -36,12 +36,26 @@ Storage + backup host on the LAN, added 2026-08. | privoxy | 192.168.0.11 | — | unprivileged, autostart; filtering HTTP proxy, listens on **:3128** (not privoxy's default 8118); static config in `/etc/systemd/network/eth0.network` (`Gateway=192.168.0.1`), `DNS=192.168.0.254`. Moved from nuc 2026-08-30 | | [transmission-bt](transmission-bt.md) | 192.168.0.7 | ✅ | unprivileged, autostart; always-on WireGuard full tunnel → ks4 (egress = 193.70.35.17, kill switch: no default route in `main`, wg-quick's `fwmark`/`suppress_prefixlength` rules send traffic to table 51820 — **`netplan apply` wipes those rules, so always `systemctl restart wg-quick@wg0` after it**); **IPv6 disabled** (`/etc/sysctl.d/99-no-ipv6.conf`) since the tunnel is `AllowedIPs = 0.0.0.0/0` only and the FTTH box's native IPv6 RA bypassed the kill switch entirely. Extending the tunnel to `::/0` is **not currently possible**: ks4 has a global v6 address and a default v6 route but **no working v6 egress** (verified 2026-09-16 — both ICMP and TCP to the v6 internet fail while v4 is fine), so it cannot act as a v6 exit. Fix OVH v6 on ks4 first if v6 peers are ever wanted; `/export/media` disk device (`shift=true`), downloads to `/media/downloads`; web UI :9091 (LAN only). Moved from nuc 2026-08-30 | +## Host tunnel + +`wg-ks4` — `10.8.0.22/24`, peer = the `wireguard` container on ks4, +endpoint `193.70.35.17:51845`, `AllowedIPs = 10.8.0.0/24, 192.168.1.1/32`. +It exists only to reach ks4's incus API at `192.168.1.1:8443` for the +05:00 pull. Key at `/etc/wireguard/wg-ks4.key`, unit +`wg-quick@wg-ks4` (enabled). + +⚠️ nas also runs a **managed `incusbr0` on `192.168.1.254/24`** — no +uplink, nothing attached, inert. It exists purely so the ks4 replicas' +instance-level `eth0` (`parent: incusbr0`, static `192.168.1.x`) passes +validation on arrival. It must never take `192.168.1.1`: that address has +to keep resolving over `wg-ks4`, and a local address beats a route. + ## Backup pools hosted here | incus pool | dataset | receives | |---|---|---| | `nucbackup` | `tank/backup/nuc` | nuc's instances (pushed nightly, 03:30) | -| `ks4backup` | `tank/backup/ks4` | ks4's instances (pulled over WG, 05:00 — after FTTH) | +| `ks4backup` | `tank/backup/ks4` | ks4's instances (pulled over `wg-ks4`, 05:00; built 2026-09-16, see [ks2/nas-seed.md](../ks2/nas-seed.md)) | | `nasbackup` | `tank/backup/nas` | **nas's own** instances (local copy, 04:00) | nas's own instances are replicated **locally** rather than to nuc: nuc is diff --git a/nas/nas-install.md b/nas/nas-install.md index c81f319..3a64eb2 100644 --- a/nas/nas-install.md +++ b/nas/nas-install.md @@ -710,11 +710,15 @@ is now here and carries its own). remove `/etc/wireguard/wg-ks4.conf`, and drop the `10.8.0.20/32` peer on ks4. -Seed manually in tmux — the first pass is **~1.75 TiB** over the home -downlink and only makes sense once FTTH is up: +Built 2026-09-16 — full runbook, corrections and gotchas: +[ks2/nas-seed.md](../ks2/nas-seed.md). Two things that block the copy if +missed: nas needs `wireguard-tools` installed and a **managed `incusbr0` +on `192.168.1.254/24`** (never `.1`), and the seed needs **`-p backup`**. ```sh -/root/scripts/incus-copy.sh -r ks4 -s ks4backup 2>&1 | tee -a /var/log/incus-copy-ks4.log +systemd-run --unit=ks4-seed --collect \ + /bin/bash -c '/root/scripts/incus-copy.sh -r ks4 -s ks4backup -p backup \ + >> /var/log/incus-copy-ks4.log 2>&1' ``` Then test-restore one instance before ticking the gate in