doc: build the ks4 -> nas backup pull leg

FTTH is up, so the leg nas-seed.md had been holding since 2026-08-30 is
now built: wg-ks4 on nas (10.8.0.22), nas peered on ks4's wireguard
container, incus remote over the tunnel, 05:00 cron, and the first pass
seeding under a systemd-run unit.

Three corrections the runbook needed, all found by running it:

- nas had no wireguard-tools at all. transmission-bt carries its own
  tunnel inside the container, so the host never needed them.
- Every ks4 instance has an instance-level eth0 pinned to incusbr0 with
  a static 192.168.1.x, so each copy failed in under a second with
  "Cannot use manually specified ipv4.address when using unmanaged
  parent bridge". nas now runs a managed incusbr0 on 192.168.1.254/24 —
  deliberately not .1, which must keep resolving over wg-ks4.
- The seed command was missing -p backup, which the doc's own
  verification step already assumed.

Also records the measured rate: ~125 Mbit/s, ks4's OVH uplink rather
than the home downlink, so ~31 h for the first pass — during which the
shared incus-copy lock suppresses the 04:00 nasbackup job.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Julien Lutran
2026-09-16 13:37:56 +02:00
co-authored by Claude Opus 5
parent 7e6b8f2848
commit c8d759ce9e
3 changed files with 90 additions and 16 deletions
+68 -12
View File
@@ -1,6 +1,9 @@
# ks4 pull leg — seed after FTTH
Status: **prepared, waiting on the FTTH link.**
Status: **built and seeding — 2026-09-16.** FTTH is up (gateway
`192.168.0.1`, see [nuc/nuc-install.md](../nuc/nuc-install.md)); the
tunnel, the incus remote and the cron are in place and the first pass is
running. Remaining: let the seed finish, then the test-restore below.
⚠️ **Changed 2026-08-30: this leg lands on `nas`, not on nuc.** It was
originally designed for nuc's USB pool `usb4t`, but that pool proved to
@@ -33,20 +36,28 @@ Consequences versus the original plan:
## Prerequisites
- [ ] `tank` running on SATA for ≥ 7 days with zero pool suspensions —
the gate that replaces "fix the USB enclosure"
- [ ] FTTH up (the first pass moves ~1.75 TiB)
- [x] `tank` running on SATA for ≥ 7 days with zero pool suspensions —
the gate that replaces "fix the USB enclosure".
Verified 2026-09-16: `ONLINE`, scrub clean 2026-09-14, uptime 2 w 2 d,
zero suspensions in the journal.
- [x] FTTH up (the first pass moves ~1.75 TiB)
## Setup (root on nas)
```sh
# 1. peer nas on ks4's wireguard container
# (run on ks4) — <nas-pubkey> from /etc/wireguard/wg-ks4.key on nas
# 0. the nas host has no wireguard-tools — transmission-bt carries its own
# tunnel *inside* the container, so the host never needed them
apt-get install -y wireguard-tools
# 1. key on nas, then peer it on ks4's wireguard container
umask 077; wg genkey > /etc/wireguard/wg-ks4.key
wg pubkey < /etc/wireguard/wg-ks4.key # -> <nas-pubkey>
# (run on ks4)
incus exec wireguard -- wg set wg0 peer <nas-pubkey> allowed-ips 10.8.0.22/32
incus exec wireguard -- wg-quick save wg0
# 2. tunnel on nas: /etc/wireguard/wg-ks4.conf, modelled on nuc's
# Address = 10.8.0.22/32, peer pubkey TVs6d7…,
# Address = 10.8.0.22/24, peer pubkey TVs6d7…,
# Endpoint = 193.70.35.17:51845,
# AllowedIPs = 10.8.0.0/24, 192.168.1.1/32, keepalive 25
systemctl enable --now wg-quick@wg-ks4
@@ -56,14 +67,56 @@ incus remote add ks4 https://192.168.1.1:8443 --accept-certificate --token '…'
incus list ks4: | head # sanity: remote reachable
```
## Seed
⚠️ **nas needs a managed `incusbr0` of its own, or every copy fails
instantly.** All 18 ks4 instances carry an *instance-level* `eth0`
(`nictype: bridged`, `parent: incusbr0`, `ipv4.address: 192.168.1.x`).
nas has no such bridge, so instance creation dies with:
```
Device validation failed for "eth0": Cannot use manually specified
ipv4.address when using unmanaged parent bridge
```
Create a managed network of the same name — but **give it `.254`, never
`.1`**: `192.168.1.1` must keep resolving over `wg-ks4` to ks4's incus
API, and a local address always beats a route.
```sh
# full pull of every ks4 instance into pool ks4backup (tmux — first pass
# moves ~1.75 TiB through the WG tunnel)
/root/scripts/incus-copy.sh -r ks4 -s ks4backup 2>&1 | tee -a /var/log/incus-copy-ks4.log
incus network create incusbr0 \
ipv4.address=192.168.1.254/24 ipv4.nat=false ipv6.address=none
ip route get 192.168.1.1 # must still say: dev wg-ks4
```
The bridge stays inert — replicas are never started here.
## Seed
⚠️ **`-p backup` is required**, exactly as for the nuc leg
([nas/nas-install.md](../nas/nas-install.md) §9a) — and as the
verification command below already assumed. Without it the 18 replicas
land in `default` alongside nas's live instances.
Run it detached rather than in a shell that can drop — the first pass is
long (measured **~125 Mbit/s**, which is ks4's OVH *uplink*, not the home
downlink: **~31 h** for the full set):
```sh
systemd-run --unit=ks4-seed --collect \
/bin/bash -c '/root/scripts/incus-copy.sh -r ks4 -s ks4backup -p backup \
>> /var/log/incus-copy-ks4.log 2>&1'
systemctl is-active ks4-seed # progress:
tail -f /var/log/incus-copy-ks4.log
```
⚠️ **The seed suppresses the 04:00 `nasbackup` job while it runs.**
`incus-copy.sh` takes a single `/run/lock/incus-copy.lock` for every
shape, so any run starting while the seed holds it aborts with
`another incus-copy run holds …`. Over a ~31 h seed that skips one or two
nights of the nas-local copy — accepted; those replicas are small,
same-host, and rebuildable. The 03:30 nuc push is unaffected (it runs on
nuc, with nuc's own lock).
Notes:
- First pass is a full send per instance; later refreshes are
@@ -79,9 +132,12 @@ Add to nas's root crontab, offset from the 03:30 nuc→nas push, the
04:00 nas→nuc push and ks4's own 01:00/05:00 jobs:
```cron
0 5 * * * /root/scripts/incus-copy.sh -r ks4 -s ks4backup >> /var/log/incus-copy-ks4.log 2>&1
0 5 * * * /root/scripts/incus-copy.sh -r ks4 -s ks4backup -p backup >> /var/log/incus-copy-ks4.log 2>&1
```
`/etc/logrotate.d/incus-copy` already lists `incus-copy-ks4.log`, so
nothing to add there.
## Verification (release gate for ks2)
```sh
+15 -1
View File
@@ -36,12 +36,26 @@ Storage + backup host on the LAN, added 2026-08.
| privoxy | 192.168.0.11 | — | unprivileged, autostart; filtering HTTP proxy, listens on **:3128** (not privoxy's default 8118); static config in `/etc/systemd/network/eth0.network` (`Gateway=192.168.0.1`), `DNS=192.168.0.254`. Moved from nuc 2026-08-30 |
| [transmission-bt](transmission-bt.md) | 192.168.0.7 | ✅ | unprivileged, autostart; always-on WireGuard full tunnel → ks4 (egress = 193.70.35.17, kill switch: no default route in `main`, wg-quick's `fwmark`/`suppress_prefixlength` rules send traffic to table 51820 — **`netplan apply` wipes those rules, so always `systemctl restart wg-quick@wg0` after it**); **IPv6 disabled** (`/etc/sysctl.d/99-no-ipv6.conf`) since the tunnel is `AllowedIPs = 0.0.0.0/0` only and the FTTH box's native IPv6 RA bypassed the kill switch entirely. Extending the tunnel to `::/0` is **not currently possible**: ks4 has a global v6 address and a default v6 route but **no working v6 egress** (verified 2026-09-16 — both ICMP and TCP to the v6 internet fail while v4 is fine), so it cannot act as a v6 exit. Fix OVH v6 on ks4 first if v6 peers are ever wanted; `/export/media` disk device (`shift=true`), downloads to `/media/downloads`; web UI :9091 (LAN only). Moved from nuc 2026-08-30 |
## Host tunnel
`wg-ks4``10.8.0.22/24`, peer = the `wireguard` container on ks4,
endpoint `193.70.35.17:51845`, `AllowedIPs = 10.8.0.0/24, 192.168.1.1/32`.
It exists only to reach ks4's incus API at `192.168.1.1:8443` for the
05:00 pull. Key at `/etc/wireguard/wg-ks4.key`, unit
`wg-quick@wg-ks4` (enabled).
⚠️ nas also runs a **managed `incusbr0` on `192.168.1.254/24`** — no
uplink, nothing attached, inert. It exists purely so the ks4 replicas'
instance-level `eth0` (`parent: incusbr0`, static `192.168.1.x`) passes
validation on arrival. It must never take `192.168.1.1`: that address has
to keep resolving over `wg-ks4`, and a local address beats a route.
## Backup pools hosted here
| incus pool | dataset | receives |
|---|---|---|
| `nucbackup` | `tank/backup/nuc` | nuc's instances (pushed nightly, 03:30) |
| `ks4backup` | `tank/backup/ks4` | ks4's instances (pulled over WG, 05:00 — after FTTH) |
| `ks4backup` | `tank/backup/ks4` | ks4's instances (pulled over `wg-ks4`, 05:00; built 2026-09-16, see [ks2/nas-seed.md](../ks2/nas-seed.md)) |
| `nasbackup` | `tank/backup/nas` | **nas's own** instances (local copy, 04:00) |
nas's own instances are replicated **locally** rather than to nuc: nuc is
+7 -3
View File
@@ -710,11 +710,15 @@ is now here and carries its own).
remove `/etc/wireguard/wg-ks4.conf`, and drop the `10.8.0.20/32` peer
on ks4.
Seed manually in tmux — the first pass is **~1.75 TiB** over the home
downlink and only makes sense once FTTH is up:
Built 2026-09-16 — full runbook, corrections and gotchas:
[ks2/nas-seed.md](../ks2/nas-seed.md). Two things that block the copy if
missed: nas needs `wireguard-tools` installed and a **managed `incusbr0`
on `192.168.1.254/24`** (never `.1`), and the seed needs **`-p backup`**.
```sh
/root/scripts/incus-copy.sh -r ks4 -s ks4backup 2>&1 | tee -a /var/log/incus-copy-ks4.log
systemd-run --unit=ks4-seed --collect \
/bin/bash -c '/root/scripts/incus-copy.sh -r ks4 -s ks4backup -p backup \
>> /var/log/incus-copy-ks4.log 2>&1'
```
Then test-restore one instance before ticking the gate in