ks2: ks4 pull leg and its WireGuard tunnel move from nuc to nas

nuc-seed.md -> nas-seed.md. The leg was designed around nuc's USB pool,
which is exactly the device it must not depend on. Target pool ks4backup
now lives on tank; nas becomes WG peer 10.8.0.22 and nuc's tunnel retires
once seeded — nuc no longer needs one at all, since transmission-bt (the
only other user) moved to nas with its own in-container tunnel.

ks4 needs no change: traffic arrives masqueraded as the wireguard
container whichever peer sent it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Julien Lutran
2026-08-30 23:53:30 +02:00
co-authored by Claude Opus 5
parent 58dcaa5d41
commit 66c7bca28f
3 changed files with 75 additions and 27 deletions
+2 -2
View File
@@ -8,8 +8,8 @@ few days of margin.
- [ ] restic S3 leg running nightly for ≥ a week, `done (rc=0)`, - [ ] restic S3 leg running nightly for ≥ a week, `done (rc=0)`,
restore test passed ([restic-backup.md](../ks4/restic-backup.md) §7) restore test passed ([restic-backup.md](../ks4/restic-backup.md) §7)
- [ ] nuc pull leg seeded and one instance test-restored - [ ] **nas** pull leg seeded and one instance test-restored
([nuc-seed.md](nuc-seed.md)) ([nas-seed.md](nas-seed.md))
- [ ] local leg (sdb5) cron green in `/var/log/incus-copy.log` - [ ] local leg (sdb5) cron green in `/var/log/incus-copy.log`
## 1. Cut the last flows to ks2 (root on ks4) ## 1. Cut the last flows to ks2 (root on ks4)
+65 -18
View File
@@ -1,19 +1,57 @@
# nuc pull leg — seed after FTTH # ks4 pull leg — seed after FTTH
Status: **prepared, waiting on the FTTH link.** Everything is already Status: **prepared, waiting on the FTTH link.**
configured on nuc (see the main [README](../README.md)): incus remote
`ks4` over the WireGuard tunnel (`wg-ks4`, 10.8.0.20 → 10.8.0.1),
target pool `ks4backup` on the USB ZFS pool (`usb4t/backup/ks4`) —
only the seed itself waited on bandwidth.
## Seed (root on nuc) ⚠️ **Changed 2026-08-30: this leg lands on `nas`, not on nuc.** It was
originally designed for nuc's USB pool `usb4t`, but that pool proved to
be the least reliable device in the setup
([nuc/usb4t-dropouts.md](../nuc/usb4t-dropouts.md)) — which is exactly
what an off-site copy of ks4 must not be. The 4 TB disk moved to direct
SATA on the new host `nas` (`192.168.0.4`,
[nas/nas-install.md](../nas/nas-install.md)), and the target pool
`ks4backup` moved with it.
Consequences versus the original plan:
- Target pool `ks4backup` is now backed by `tank/backup/ks4` on nas.
- The **WireGuard tunnel moves too**: nas becomes peer `10.8.0.22`;
nuc's `wg-ks4` (`10.8.0.20`) is retired once this works. nuc no longer
needs a tunnel at all — `transmission-bt`, its only other user, now
runs on nas and carries its own in-container tunnel (`10.8.0.21`,
unchanged, ks4 needs no edit for it).
- ks4's ufw rule is unchanged: traffic arrives masqueraded as the
`wireguard` container (`192.168.1.18`) whichever peer sent it.
## Prerequisites
- [ ] `tank` running on SATA for ≥ 7 days with zero pool suspensions —
the gate that replaces "fix the USB enclosure"
- [ ] FTTH up (the first pass moves ~1.75 TiB)
## Setup (root on nas)
```sh ```sh
# sanity: remote reachable through the tunnel # 1. peer nas on ks4's wireguard container
incus list ks4: | head # (run on ks4) — <nas-pubkey> from /etc/wireguard/wg-ks4.key on nas
incus exec wireguard -- wg set wg0 peer <nas-pubkey> allowed-ips 10.8.0.22/32
incus exec wireguard -- wg-quick save wg0
# full pull of every ks4 instance into pool ks4backup (screen/tmux — # 2. tunnel on nas: /etc/wireguard/wg-ks4.conf, modelled on nuc's
# first pass moves ~1.7 T through the WG tunnel) # Address = 10.8.0.22/32, peer pubkey TVs6d7…,
# Endpoint = 193.70.35.17:51845,
# AllowedIPs = 10.8.0.0/24, 192.168.1.1/32, keepalive 25
systemctl enable --now wg-quick@wg-ks4
# 3. incus remote over the tunnel
incus remote add ks4 https://192.168.1.1:8443 --accept-certificate --token '…'
incus list ks4: | head # sanity: remote reachable
```
## Seed
```sh
# full pull of every ks4 instance into pool ks4backup (tmux — first pass
# moves ~1.75 TiB through the WG tunnel)
/root/scripts/incus-copy.sh -r ks4 -s ks4backup 2>&1 | tee -a /var/log/incus-copy-ks4.log /root/scripts/incus-copy.sh -r ks4 -s ks4backup 2>&1 | tee -a /var/log/incus-copy-ks4.log
``` ```
@@ -23,13 +61,13 @@ Notes:
ZFS-incremental **as long as they run at least every ZFS-incremental **as long as they run at least every
`snapshots.expiry` (7 d on ks4)** — same caveat as `snapshots.expiry` (7 d on ks4)** — same caveat as
[ks4's local leg](../ks4/local-backup-cron.md). [ks4's local leg](../ks4/local-backup-cron.md).
- Replicas arrive stopped with `boot.autostart=false` (script does - Replicas arrive stopped with `boot.autostart=false` (the script does
this) — they must never come up on the LAN with ks4's proxy devices. this) — they must never come up on the LAN with ks4's proxy devices.
## Cron (after the seed) ## Cron (after the seed)
Add to nuc's root crontab, offset from the 03:30 local nucbackup copy Add to nas's root crontab, offset from the 03:30 nuc→nas push, the
and ks4's own 01:00/05:00 jobs: 04:00 nas→nuc push and ks4's own 01:00/05:00 jobs:
```cron ```cron
0 5 * * * /root/scripts/incus-copy.sh -r ks4 -s ks4backup >> /var/log/incus-copy-ks4.log 2>&1 0 5 * * * /root/scripts/incus-copy.sh -r ks4 -s ks4backup >> /var/log/incus-copy-ks4.log 2>&1
@@ -38,12 +76,21 @@ and ks4's own 01:00/05:00 jobs:
## Verification (release gate for ks2) ## Verification (release gate for ks2)
```sh ```sh
incus list --project backup 2>/dev/null || incus list | grep -c . # all ks4 instances present incus list --project backup -c ns -f csv # all ks4 instances present
# test-restore one instance: copy a replica to the default pool, # test-restore one instance: copy a replica to the local pool,
# start it isolated, check the service answers, then delete it # start it isolated, check the service answers, then delete it
incus copy solar solar-restoretest -s default incus copy solar solar-restoretest -s incus
incus start solar-restoretest && incus exec solar-restoretest -- systemctl is-system-running incus start solar-restoretest && incus exec solar-restoretest -- systemctl is-system-running
incus delete -f solar-restoretest incus delete -f solar-restoretest
``` ```
Once verified, tick the nuc gate in the [ks2 plan](plan.md). Once verified, tick the nas gate in the [ks2 plan](plan.md) and retire
nuc's tunnel:
```sh
# on nuc
systemctl disable --now wg-quick@wg-ks4 && rm /etc/wireguard/wg-ks4.conf
# on ks4
incus exec wireguard -- wg set wg0 peer <nuc-pubkey> remove
incus exec wireguard -- wg-quick save wg0
```
+8 -7
View File
@@ -17,10 +17,11 @@ What remains on the box is **cold history**: instance replicas on pool
`/backup/ns3061243` on pool `backup` (last refreshed 2026-08-28), `/backup/ns3061243` on pool `backup` (last refreshed 2026-08-28),
snapshotted daily there (`zfs-auto-snapshot.sh`, 2-month expiry). snapshotted daily there (`zfs-auto-snapshot.sh`, 2-month expiry).
⚠️ While the nuc leg waits for FTTH, instances have no *fresh* ⚠️ While the nas leg waits for FTTH, instances have no *fresh*
off-site copy — the ks2 push is to be re-enabled as soon as the off-site copy — the ks2 push is to be re-enabled as soon as the
initial restic sync finishes (decided 2026-08-28), and retired again initial restic sync finishes (decided 2026-08-28), and retired again
when nuc takes over. when nas takes over. (The leg moved from nuc to the new host `nas`
on 2026-08-30 — see [nas-seed.md](nas-seed.md).)
## Inventory findings (2026-08-22) ## Inventory findings (2026-08-22)
@@ -51,8 +52,8 @@ when nuc takes over.
| local, 2nd disk | `incus-copy.sh -p backup -s backup` → sdb5 pool | **live** (01:00) | | local, 2nd disk | `incus-copy.sh -p backup -s backup` → sdb5 pool | **live** (01:00) |
| off-site, S3 (data+DB) | **restic** → bucket `restic-data` ([restic-backup.md](../ks4/restic-backup.md)) | **live** (05:00) | | off-site, S3 (data+DB) | **restic** → bucket `restic-data` ([restic-backup.md](../ks4/restic-backup.md)) | **live** (05:00) |
| off-site, S3 (instances) | restic over `incus file mount` of the `backup`-project replicas | **shelved** 2026-08-28 (replication covers instances) | | off-site, S3 (instances) | restic over `incus file mount` of the `backup`-project replicas | **shelved** 2026-08-28 (replication covers instances) |
| off-site, nuc | nuc pulls `ks4:*` → pool `ks4backup` over WG ([nuc-seed.md](nuc-seed.md)) | waiting FTTH (< Sep 30) | | off-site, nas | **nas** pulls `ks4:*` → pool `ks4backup` over WG ([nas-seed.md](nas-seed.md)) | waiting FTTH (< Sep 30); moved off nuc 2026-08-30 |
| off-site, ks2 (interim) | `incus-copy.sh -d ks2 -m push` at 02:00 until the nuc leg seeds | to re-enable once the restic seed finishes | | off-site, ks2 (interim) | `incus-copy.sh -d ks2 -m push` at 02:00 until the nas leg seeds | to re-enable once the restic seed finishes |
## Actions (backup work documented in [`../ks4/`](../ks4/), ks2-only tasks here) ## Actions (backup work documented in [`../ks4/`](../ks4/), ks2-only tasks here)
@@ -65,9 +66,9 @@ when nuc takes over.
([restic-backup.md](../ks4/restic-backup.md), live 2026-08-28; the ([restic-backup.md](../ks4/restic-backup.md), live 2026-08-28; the
predecessor's doc is kept as reference) predecessor's doc is kept as reference)
4. ~~instance leg to S3~~**shelved 2026-08-28**: instances are 4. ~~instance leg to S3~~**shelved 2026-08-28**: instances are
protected by replication (sdb + nuc/ks2), their data and configs by protected by replication (sdb + nas/ks2), their data and configs by
`restic-data` `restic-data`
5. [nuc-seed.md](nuc-seed.md) — **prepared**; after FTTH: seed nuc 5. [nas-seed.md](nas-seed.md) — **prepared**; after FTTH: seed the nas
pull leg, verify all instances, test-restore one pull leg, verify all instances, test-restore one
6. [decommission.md](decommission.md) — **prepared**; cut flows, 6. [decommission.md](decommission.md) — **prepared**; cut flows,
final diff of `/backup/ns3061243`, wipe pools, terminate at OVH final diff of `/backup/ns3061243`, wipe pools, terminate at OVH
@@ -77,7 +78,7 @@ when nuc takes over.
- [x] biwiki + spot consciously abandoned (2026-08-22) - [x] biwiki + spot consciously abandoned (2026-08-22)
- [x] local leg cron running since 2026-08-22, **18/18 instances** - [x] local leg cron running since 2026-08-22, **18/18 instances**
replicated (verified 2026-08-28) replicated (verified 2026-08-28)
- [ ] nuc leg fully seeded **and** one instance test-restored - [ ] **nas** leg fully seeded **and** one instance test-restored
- [x] restic S3 backups live (05:00) **and** restore drill passed - [x] restic S3 backups live (05:00) **and** restore drill passed
2026-08-28: tree restored byte-identical to live, dump restored 2026-08-28: tree restored byte-identical to live, dump restored
and loaded into a scratch MariaDB (12/12 tables) and loaded into a scratch MariaDB (12/12 tables)