doc: build the ks4 -> nas backup pull leg
FTTH is up, so the leg nas-seed.md had been holding since 2026-08-30 is now built: wg-ks4 on nas (10.8.0.22), nas peered on ks4's wireguard container, incus remote over the tunnel, 05:00 cron, and the first pass seeding under a systemd-run unit. Three corrections the runbook needed, all found by running it: - nas had no wireguard-tools at all. transmission-bt carries its own tunnel inside the container, so the host never needed them. - Every ks4 instance has an instance-level eth0 pinned to incusbr0 with a static 192.168.1.x, so each copy failed in under a second with "Cannot use manually specified ipv4.address when using unmanaged parent bridge". nas now runs a managed incusbr0 on 192.168.1.254/24 — deliberately not .1, which must keep resolving over wg-ks4. - The seed command was missing -p backup, which the doc's own verification step already assumed. Also records the measured rate: ~125 Mbit/s, ks4's OVH uplink rather than the home downlink, so ~31 h for the first pass — during which the shared incus-copy lock suppresses the 04:00 nasbackup job. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5
parent
7e6b8f2848
commit
c8d759ce9e
+68
-12
@@ -1,6 +1,9 @@
|
|||||||
# ks4 pull leg — seed after FTTH
|
# ks4 pull leg — seed after FTTH
|
||||||
|
|
||||||
Status: **prepared, waiting on the FTTH link.**
|
Status: **built and seeding — 2026-09-16.** FTTH is up (gateway
|
||||||
|
`192.168.0.1`, see [nuc/nuc-install.md](../nuc/nuc-install.md)); the
|
||||||
|
tunnel, the incus remote and the cron are in place and the first pass is
|
||||||
|
running. Remaining: let the seed finish, then the test-restore below.
|
||||||
|
|
||||||
⚠️ **Changed 2026-08-30: this leg lands on `nas`, not on nuc.** It was
|
⚠️ **Changed 2026-08-30: this leg lands on `nas`, not on nuc.** It was
|
||||||
originally designed for nuc's USB pool `usb4t`, but that pool proved to
|
originally designed for nuc's USB pool `usb4t`, but that pool proved to
|
||||||
@@ -33,20 +36,28 @@ Consequences versus the original plan:
|
|||||||
|
|
||||||
## Prerequisites
|
## Prerequisites
|
||||||
|
|
||||||
- [ ] `tank` running on SATA for ≥ 7 days with zero pool suspensions —
|
- [x] `tank` running on SATA for ≥ 7 days with zero pool suspensions —
|
||||||
the gate that replaces "fix the USB enclosure"
|
the gate that replaces "fix the USB enclosure".
|
||||||
- [ ] FTTH up (the first pass moves ~1.75 TiB)
|
Verified 2026-09-16: `ONLINE`, scrub clean 2026-09-14, uptime 2 w 2 d,
|
||||||
|
zero suspensions in the journal.
|
||||||
|
- [x] FTTH up (the first pass moves ~1.75 TiB)
|
||||||
|
|
||||||
## Setup (root on nas)
|
## Setup (root on nas)
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
# 1. peer nas on ks4's wireguard container
|
# 0. the nas host has no wireguard-tools — transmission-bt carries its own
|
||||||
# (run on ks4) — <nas-pubkey> from /etc/wireguard/wg-ks4.key on nas
|
# tunnel *inside* the container, so the host never needed them
|
||||||
|
apt-get install -y wireguard-tools
|
||||||
|
|
||||||
|
# 1. key on nas, then peer it on ks4's wireguard container
|
||||||
|
umask 077; wg genkey > /etc/wireguard/wg-ks4.key
|
||||||
|
wg pubkey < /etc/wireguard/wg-ks4.key # -> <nas-pubkey>
|
||||||
|
# (run on ks4)
|
||||||
incus exec wireguard -- wg set wg0 peer <nas-pubkey> allowed-ips 10.8.0.22/32
|
incus exec wireguard -- wg set wg0 peer <nas-pubkey> allowed-ips 10.8.0.22/32
|
||||||
incus exec wireguard -- wg-quick save wg0
|
incus exec wireguard -- wg-quick save wg0
|
||||||
|
|
||||||
# 2. tunnel on nas: /etc/wireguard/wg-ks4.conf, modelled on nuc's
|
# 2. tunnel on nas: /etc/wireguard/wg-ks4.conf, modelled on nuc's
|
||||||
# Address = 10.8.0.22/32, peer pubkey TVs6d7…,
|
# Address = 10.8.0.22/24, peer pubkey TVs6d7…,
|
||||||
# Endpoint = 193.70.35.17:51845,
|
# Endpoint = 193.70.35.17:51845,
|
||||||
# AllowedIPs = 10.8.0.0/24, 192.168.1.1/32, keepalive 25
|
# AllowedIPs = 10.8.0.0/24, 192.168.1.1/32, keepalive 25
|
||||||
systemctl enable --now wg-quick@wg-ks4
|
systemctl enable --now wg-quick@wg-ks4
|
||||||
@@ -56,14 +67,56 @@ incus remote add ks4 https://192.168.1.1:8443 --accept-certificate --token '…'
|
|||||||
incus list ks4: | head # sanity: remote reachable
|
incus list ks4: | head # sanity: remote reachable
|
||||||
```
|
```
|
||||||
|
|
||||||
## Seed
|
⚠️ **nas needs a managed `incusbr0` of its own, or every copy fails
|
||||||
|
instantly.** All 18 ks4 instances carry an *instance-level* `eth0`
|
||||||
|
(`nictype: bridged`, `parent: incusbr0`, `ipv4.address: 192.168.1.x`).
|
||||||
|
nas has no such bridge, so instance creation dies with:
|
||||||
|
|
||||||
|
```
|
||||||
|
Device validation failed for "eth0": Cannot use manually specified
|
||||||
|
ipv4.address when using unmanaged parent bridge
|
||||||
|
```
|
||||||
|
|
||||||
|
Create a managed network of the same name — but **give it `.254`, never
|
||||||
|
`.1`**: `192.168.1.1` must keep resolving over `wg-ks4` to ks4's incus
|
||||||
|
API, and a local address always beats a route.
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
# full pull of every ks4 instance into pool ks4backup (tmux — first pass
|
incus network create incusbr0 \
|
||||||
# moves ~1.75 TiB through the WG tunnel)
|
ipv4.address=192.168.1.254/24 ipv4.nat=false ipv6.address=none
|
||||||
/root/scripts/incus-copy.sh -r ks4 -s ks4backup 2>&1 | tee -a /var/log/incus-copy-ks4.log
|
ip route get 192.168.1.1 # must still say: dev wg-ks4
|
||||||
```
|
```
|
||||||
|
|
||||||
|
The bridge stays inert — replicas are never started here.
|
||||||
|
|
||||||
|
## Seed
|
||||||
|
|
||||||
|
⚠️ **`-p backup` is required**, exactly as for the nuc leg
|
||||||
|
([nas/nas-install.md](../nas/nas-install.md) §9a) — and as the
|
||||||
|
verification command below already assumed. Without it the 18 replicas
|
||||||
|
land in `default` alongside nas's live instances.
|
||||||
|
|
||||||
|
Run it detached rather than in a shell that can drop — the first pass is
|
||||||
|
long (measured **~125 Mbit/s**, which is ks4's OVH *uplink*, not the home
|
||||||
|
downlink: **~31 h** for the full set):
|
||||||
|
|
||||||
|
```sh
|
||||||
|
systemd-run --unit=ks4-seed --collect \
|
||||||
|
/bin/bash -c '/root/scripts/incus-copy.sh -r ks4 -s ks4backup -p backup \
|
||||||
|
>> /var/log/incus-copy-ks4.log 2>&1'
|
||||||
|
|
||||||
|
systemctl is-active ks4-seed # progress:
|
||||||
|
tail -f /var/log/incus-copy-ks4.log
|
||||||
|
```
|
||||||
|
|
||||||
|
⚠️ **The seed suppresses the 04:00 `nasbackup` job while it runs.**
|
||||||
|
`incus-copy.sh` takes a single `/run/lock/incus-copy.lock` for every
|
||||||
|
shape, so any run starting while the seed holds it aborts with
|
||||||
|
`another incus-copy run holds …`. Over a ~31 h seed that skips one or two
|
||||||
|
nights of the nas-local copy — accepted; those replicas are small,
|
||||||
|
same-host, and rebuildable. The 03:30 nuc push is unaffected (it runs on
|
||||||
|
nuc, with nuc's own lock).
|
||||||
|
|
||||||
Notes:
|
Notes:
|
||||||
|
|
||||||
- First pass is a full send per instance; later refreshes are
|
- First pass is a full send per instance; later refreshes are
|
||||||
@@ -79,9 +132,12 @@ Add to nas's root crontab, offset from the 03:30 nuc→nas push, the
|
|||||||
04:00 nas→nuc push and ks4's own 01:00/05:00 jobs:
|
04:00 nas→nuc push and ks4's own 01:00/05:00 jobs:
|
||||||
|
|
||||||
```cron
|
```cron
|
||||||
0 5 * * * /root/scripts/incus-copy.sh -r ks4 -s ks4backup >> /var/log/incus-copy-ks4.log 2>&1
|
0 5 * * * /root/scripts/incus-copy.sh -r ks4 -s ks4backup -p backup >> /var/log/incus-copy-ks4.log 2>&1
|
||||||
```
|
```
|
||||||
|
|
||||||
|
`/etc/logrotate.d/incus-copy` already lists `incus-copy-ks4.log`, so
|
||||||
|
nothing to add there.
|
||||||
|
|
||||||
## Verification (release gate for ks2)
|
## Verification (release gate for ks2)
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
|
|||||||
+15
-1
@@ -36,12 +36,26 @@ Storage + backup host on the LAN, added 2026-08.
|
|||||||
| privoxy | 192.168.0.11 | — | unprivileged, autostart; filtering HTTP proxy, listens on **:3128** (not privoxy's default 8118); static config in `/etc/systemd/network/eth0.network` (`Gateway=192.168.0.1`), `DNS=192.168.0.254`. Moved from nuc 2026-08-30 |
|
| privoxy | 192.168.0.11 | — | unprivileged, autostart; filtering HTTP proxy, listens on **:3128** (not privoxy's default 8118); static config in `/etc/systemd/network/eth0.network` (`Gateway=192.168.0.1`), `DNS=192.168.0.254`. Moved from nuc 2026-08-30 |
|
||||||
| [transmission-bt](transmission-bt.md) | 192.168.0.7 | ✅ | unprivileged, autostart; always-on WireGuard full tunnel → ks4 (egress = 193.70.35.17, kill switch: no default route in `main`, wg-quick's `fwmark`/`suppress_prefixlength` rules send traffic to table 51820 — **`netplan apply` wipes those rules, so always `systemctl restart wg-quick@wg0` after it**); **IPv6 disabled** (`/etc/sysctl.d/99-no-ipv6.conf`) since the tunnel is `AllowedIPs = 0.0.0.0/0` only and the FTTH box's native IPv6 RA bypassed the kill switch entirely. Extending the tunnel to `::/0` is **not currently possible**: ks4 has a global v6 address and a default v6 route but **no working v6 egress** (verified 2026-09-16 — both ICMP and TCP to the v6 internet fail while v4 is fine), so it cannot act as a v6 exit. Fix OVH v6 on ks4 first if v6 peers are ever wanted; `/export/media` disk device (`shift=true`), downloads to `/media/downloads`; web UI :9091 (LAN only). Moved from nuc 2026-08-30 |
|
| [transmission-bt](transmission-bt.md) | 192.168.0.7 | ✅ | unprivileged, autostart; always-on WireGuard full tunnel → ks4 (egress = 193.70.35.17, kill switch: no default route in `main`, wg-quick's `fwmark`/`suppress_prefixlength` rules send traffic to table 51820 — **`netplan apply` wipes those rules, so always `systemctl restart wg-quick@wg0` after it**); **IPv6 disabled** (`/etc/sysctl.d/99-no-ipv6.conf`) since the tunnel is `AllowedIPs = 0.0.0.0/0` only and the FTTH box's native IPv6 RA bypassed the kill switch entirely. Extending the tunnel to `::/0` is **not currently possible**: ks4 has a global v6 address and a default v6 route but **no working v6 egress** (verified 2026-09-16 — both ICMP and TCP to the v6 internet fail while v4 is fine), so it cannot act as a v6 exit. Fix OVH v6 on ks4 first if v6 peers are ever wanted; `/export/media` disk device (`shift=true`), downloads to `/media/downloads`; web UI :9091 (LAN only). Moved from nuc 2026-08-30 |
|
||||||
|
|
||||||
|
## Host tunnel
|
||||||
|
|
||||||
|
`wg-ks4` — `10.8.0.22/24`, peer = the `wireguard` container on ks4,
|
||||||
|
endpoint `193.70.35.17:51845`, `AllowedIPs = 10.8.0.0/24, 192.168.1.1/32`.
|
||||||
|
It exists only to reach ks4's incus API at `192.168.1.1:8443` for the
|
||||||
|
05:00 pull. Key at `/etc/wireguard/wg-ks4.key`, unit
|
||||||
|
`wg-quick@wg-ks4` (enabled).
|
||||||
|
|
||||||
|
⚠️ nas also runs a **managed `incusbr0` on `192.168.1.254/24`** — no
|
||||||
|
uplink, nothing attached, inert. It exists purely so the ks4 replicas'
|
||||||
|
instance-level `eth0` (`parent: incusbr0`, static `192.168.1.x`) passes
|
||||||
|
validation on arrival. It must never take `192.168.1.1`: that address has
|
||||||
|
to keep resolving over `wg-ks4`, and a local address beats a route.
|
||||||
|
|
||||||
## Backup pools hosted here
|
## Backup pools hosted here
|
||||||
|
|
||||||
| incus pool | dataset | receives |
|
| incus pool | dataset | receives |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| `nucbackup` | `tank/backup/nuc` | nuc's instances (pushed nightly, 03:30) |
|
| `nucbackup` | `tank/backup/nuc` | nuc's instances (pushed nightly, 03:30) |
|
||||||
| `ks4backup` | `tank/backup/ks4` | ks4's instances (pulled over WG, 05:00 — after FTTH) |
|
| `ks4backup` | `tank/backup/ks4` | ks4's instances (pulled over `wg-ks4`, 05:00; built 2026-09-16, see [ks2/nas-seed.md](../ks2/nas-seed.md)) |
|
||||||
| `nasbackup` | `tank/backup/nas` | **nas's own** instances (local copy, 04:00) |
|
| `nasbackup` | `tank/backup/nas` | **nas's own** instances (local copy, 04:00) |
|
||||||
|
|
||||||
nas's own instances are replicated **locally** rather than to nuc: nuc is
|
nas's own instances are replicated **locally** rather than to nuc: nuc is
|
||||||
|
|||||||
+7
-3
@@ -710,11 +710,15 @@ is now here and carries its own).
|
|||||||
remove `/etc/wireguard/wg-ks4.conf`, and drop the `10.8.0.20/32` peer
|
remove `/etc/wireguard/wg-ks4.conf`, and drop the `10.8.0.20/32` peer
|
||||||
on ks4.
|
on ks4.
|
||||||
|
|
||||||
Seed manually in tmux — the first pass is **~1.75 TiB** over the home
|
Built 2026-09-16 — full runbook, corrections and gotchas:
|
||||||
downlink and only makes sense once FTTH is up:
|
[ks2/nas-seed.md](../ks2/nas-seed.md). Two things that block the copy if
|
||||||
|
missed: nas needs `wireguard-tools` installed and a **managed `incusbr0`
|
||||||
|
on `192.168.1.254/24`** (never `.1`), and the seed needs **`-p backup`**.
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
/root/scripts/incus-copy.sh -r ks4 -s ks4backup 2>&1 | tee -a /var/log/incus-copy-ks4.log
|
systemd-run --unit=ks4-seed --collect \
|
||||||
|
/bin/bash -c '/root/scripts/incus-copy.sh -r ks4 -s ks4backup -p backup \
|
||||||
|
>> /var/log/incus-copy-ks4.log 2>&1'
|
||||||
```
|
```
|
||||||
|
|
||||||
Then test-restore one instance before ticking the gate in
|
Then test-restore one instance before ticking the gate in
|
||||||
|
|||||||
Reference in New Issue
Block a user