ks2: ks4 pull leg and its WireGuard tunnel move from nuc to nas
nuc-seed.md -> nas-seed.md. The leg was designed around nuc's USB pool, which is exactly the device it must not depend on. Target pool ks4backup now lives on tank; nas becomes WG peer 10.8.0.22 and nuc's tunnel retires once seeded — nuc no longer needs one at all, since transmission-bt (the only other user) moved to nas with its own in-container tunnel. ks4 needs no change: traffic arrives masqueraded as the wireguard container whichever peer sent it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5
parent
58dcaa5d41
commit
66c7bca28f
+2
-2
@@ -8,8 +8,8 @@ few days of margin.
|
|||||||
|
|
||||||
- [ ] restic S3 leg running nightly for ≥ a week, `done (rc=0)`,
|
- [ ] restic S3 leg running nightly for ≥ a week, `done (rc=0)`,
|
||||||
restore test passed ([restic-backup.md](../ks4/restic-backup.md) §7)
|
restore test passed ([restic-backup.md](../ks4/restic-backup.md) §7)
|
||||||
- [ ] nuc pull leg seeded and one instance test-restored
|
- [ ] **nas** pull leg seeded and one instance test-restored
|
||||||
([nuc-seed.md](nuc-seed.md))
|
([nas-seed.md](nas-seed.md))
|
||||||
- [ ] local leg (sdb5) cron green in `/var/log/incus-copy.log`
|
- [ ] local leg (sdb5) cron green in `/var/log/incus-copy.log`
|
||||||
|
|
||||||
## 1. Cut the last flows to ks2 (root on ks4)
|
## 1. Cut the last flows to ks2 (root on ks4)
|
||||||
|
|||||||
+65
-18
@@ -1,19 +1,57 @@
|
|||||||
# nuc pull leg — seed after FTTH
|
# ks4 pull leg — seed after FTTH
|
||||||
|
|
||||||
Status: **prepared, waiting on the FTTH link.** Everything is already
|
Status: **prepared, waiting on the FTTH link.**
|
||||||
configured on nuc (see the main [README](../README.md)): incus remote
|
|
||||||
`ks4` over the WireGuard tunnel (`wg-ks4`, 10.8.0.20 → 10.8.0.1),
|
|
||||||
target pool `ks4backup` on the USB ZFS pool (`usb4t/backup/ks4`) —
|
|
||||||
only the seed itself waited on bandwidth.
|
|
||||||
|
|
||||||
## Seed (root on nuc)
|
⚠️ **Changed 2026-08-30: this leg lands on `nas`, not on nuc.** It was
|
||||||
|
originally designed for nuc's USB pool `usb4t`, but that pool proved to
|
||||||
|
be the least reliable device in the setup
|
||||||
|
([nuc/usb4t-dropouts.md](../nuc/usb4t-dropouts.md)) — which is exactly
|
||||||
|
what an off-site copy of ks4 must not be. The 4 TB disk moved to direct
|
||||||
|
SATA on the new host `nas` (`192.168.0.4`,
|
||||||
|
[nas/nas-install.md](../nas/nas-install.md)), and the target pool
|
||||||
|
`ks4backup` moved with it.
|
||||||
|
|
||||||
|
Consequences versus the original plan:
|
||||||
|
|
||||||
|
- Target pool `ks4backup` is now backed by `tank/backup/ks4` on nas.
|
||||||
|
- The **WireGuard tunnel moves too**: nas becomes peer `10.8.0.22`;
|
||||||
|
nuc's `wg-ks4` (`10.8.0.20`) is retired once this works. nuc no longer
|
||||||
|
needs a tunnel at all — `transmission-bt`, its only other user, now
|
||||||
|
runs on nas and carries its own in-container tunnel (`10.8.0.21`,
|
||||||
|
unchanged, ks4 needs no edit for it).
|
||||||
|
- ks4's ufw rule is unchanged: traffic arrives masqueraded as the
|
||||||
|
`wireguard` container (`192.168.1.18`) whichever peer sent it.
|
||||||
|
|
||||||
|
## Prerequisites
|
||||||
|
|
||||||
|
- [ ] `tank` running on SATA for ≥ 7 days with zero pool suspensions —
|
||||||
|
the gate that replaces "fix the USB enclosure"
|
||||||
|
- [ ] FTTH up (the first pass moves ~1.75 TiB)
|
||||||
|
|
||||||
|
## Setup (root on nas)
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
# sanity: remote reachable through the tunnel
|
# 1. peer nas on ks4's wireguard container
|
||||||
incus list ks4: | head
|
# (run on ks4) — <nas-pubkey> from /etc/wireguard/wg-ks4.key on nas
|
||||||
|
incus exec wireguard -- wg set wg0 peer <nas-pubkey> allowed-ips 10.8.0.22/32
|
||||||
|
incus exec wireguard -- wg-quick save wg0
|
||||||
|
|
||||||
# full pull of every ks4 instance into pool ks4backup (screen/tmux —
|
# 2. tunnel on nas: /etc/wireguard/wg-ks4.conf, modelled on nuc's
|
||||||
# first pass moves ~1.7 T through the WG tunnel)
|
# Address = 10.8.0.22/32, peer pubkey TVs6d7…,
|
||||||
|
# Endpoint = 193.70.35.17:51845,
|
||||||
|
# AllowedIPs = 10.8.0.0/24, 192.168.1.1/32, keepalive 25
|
||||||
|
systemctl enable --now wg-quick@wg-ks4
|
||||||
|
|
||||||
|
# 3. incus remote over the tunnel
|
||||||
|
incus remote add ks4 https://192.168.1.1:8443 --accept-certificate --token '…'
|
||||||
|
incus list ks4: | head # sanity: remote reachable
|
||||||
|
```
|
||||||
|
|
||||||
|
## Seed
|
||||||
|
|
||||||
|
```sh
|
||||||
|
# full pull of every ks4 instance into pool ks4backup (tmux — first pass
|
||||||
|
# moves ~1.75 TiB through the WG tunnel)
|
||||||
/root/scripts/incus-copy.sh -r ks4 -s ks4backup 2>&1 | tee -a /var/log/incus-copy-ks4.log
|
/root/scripts/incus-copy.sh -r ks4 -s ks4backup 2>&1 | tee -a /var/log/incus-copy-ks4.log
|
||||||
```
|
```
|
||||||
|
|
||||||
@@ -23,13 +61,13 @@ Notes:
|
|||||||
ZFS-incremental **as long as they run at least every
|
ZFS-incremental **as long as they run at least every
|
||||||
`snapshots.expiry` (7 d on ks4)** — same caveat as
|
`snapshots.expiry` (7 d on ks4)** — same caveat as
|
||||||
[ks4's local leg](../ks4/local-backup-cron.md).
|
[ks4's local leg](../ks4/local-backup-cron.md).
|
||||||
- Replicas arrive stopped with `boot.autostart=false` (script does
|
- Replicas arrive stopped with `boot.autostart=false` (the script does
|
||||||
this) — they must never come up on the LAN with ks4's proxy devices.
|
this) — they must never come up on the LAN with ks4's proxy devices.
|
||||||
|
|
||||||
## Cron (after the seed)
|
## Cron (after the seed)
|
||||||
|
|
||||||
Add to nuc's root crontab, offset from the 03:30 local nucbackup copy
|
Add to nas's root crontab, offset from the 03:30 nuc→nas push, the
|
||||||
and ks4's own 01:00/05:00 jobs:
|
04:00 nas→nuc push and ks4's own 01:00/05:00 jobs:
|
||||||
|
|
||||||
```cron
|
```cron
|
||||||
0 5 * * * /root/scripts/incus-copy.sh -r ks4 -s ks4backup >> /var/log/incus-copy-ks4.log 2>&1
|
0 5 * * * /root/scripts/incus-copy.sh -r ks4 -s ks4backup >> /var/log/incus-copy-ks4.log 2>&1
|
||||||
@@ -38,12 +76,21 @@ and ks4's own 01:00/05:00 jobs:
|
|||||||
## Verification (release gate for ks2)
|
## Verification (release gate for ks2)
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
incus list --project backup 2>/dev/null || incus list | grep -c . # all ks4 instances present
|
incus list --project backup -c ns -f csv # all ks4 instances present
|
||||||
# test-restore one instance: copy a replica to the default pool,
|
# test-restore one instance: copy a replica to the local pool,
|
||||||
# start it isolated, check the service answers, then delete it
|
# start it isolated, check the service answers, then delete it
|
||||||
incus copy solar solar-restoretest -s default
|
incus copy solar solar-restoretest -s incus
|
||||||
incus start solar-restoretest && incus exec solar-restoretest -- systemctl is-system-running
|
incus start solar-restoretest && incus exec solar-restoretest -- systemctl is-system-running
|
||||||
incus delete -f solar-restoretest
|
incus delete -f solar-restoretest
|
||||||
```
|
```
|
||||||
|
|
||||||
Once verified, tick the nuc gate in the [ks2 plan](plan.md).
|
Once verified, tick the nas gate in the [ks2 plan](plan.md) and retire
|
||||||
|
nuc's tunnel:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
# on nuc
|
||||||
|
systemctl disable --now wg-quick@wg-ks4 && rm /etc/wireguard/wg-ks4.conf
|
||||||
|
# on ks4
|
||||||
|
incus exec wireguard -- wg set wg0 peer <nuc-pubkey> remove
|
||||||
|
incus exec wireguard -- wg-quick save wg0
|
||||||
|
```
|
||||||
|
|||||||
+8
-7
@@ -17,10 +17,11 @@ What remains on the box is **cold history**: instance replicas on pool
|
|||||||
`/backup/ns3061243` on pool `backup` (last refreshed 2026-08-28),
|
`/backup/ns3061243` on pool `backup` (last refreshed 2026-08-28),
|
||||||
snapshotted daily there (`zfs-auto-snapshot.sh`, 2-month expiry).
|
snapshotted daily there (`zfs-auto-snapshot.sh`, 2-month expiry).
|
||||||
|
|
||||||
⚠️ While the nuc leg waits for FTTH, instances have no *fresh*
|
⚠️ While the nas leg waits for FTTH, instances have no *fresh*
|
||||||
off-site copy — the ks2 push is to be re-enabled as soon as the
|
off-site copy — the ks2 push is to be re-enabled as soon as the
|
||||||
initial restic sync finishes (decided 2026-08-28), and retired again
|
initial restic sync finishes (decided 2026-08-28), and retired again
|
||||||
when nuc takes over.
|
when nas takes over. (The leg moved from nuc to the new host `nas`
|
||||||
|
on 2026-08-30 — see [nas-seed.md](nas-seed.md).)
|
||||||
|
|
||||||
## Inventory findings (2026-08-22)
|
## Inventory findings (2026-08-22)
|
||||||
|
|
||||||
@@ -51,8 +52,8 @@ when nuc takes over.
|
|||||||
| local, 2nd disk | `incus-copy.sh -p backup -s backup` → sdb5 pool | **live** (01:00) |
|
| local, 2nd disk | `incus-copy.sh -p backup -s backup` → sdb5 pool | **live** (01:00) |
|
||||||
| off-site, S3 (data+DB) | **restic** → bucket `restic-data` ([restic-backup.md](../ks4/restic-backup.md)) | **live** (05:00) |
|
| off-site, S3 (data+DB) | **restic** → bucket `restic-data` ([restic-backup.md](../ks4/restic-backup.md)) | **live** (05:00) |
|
||||||
| off-site, S3 (instances) | restic over `incus file mount` of the `backup`-project replicas | **shelved** 2026-08-28 (replication covers instances) |
|
| off-site, S3 (instances) | restic over `incus file mount` of the `backup`-project replicas | **shelved** 2026-08-28 (replication covers instances) |
|
||||||
| off-site, nuc | nuc pulls `ks4:*` → pool `ks4backup` over WG ([nuc-seed.md](nuc-seed.md)) | waiting FTTH (< Sep 30) |
|
| off-site, nas | **nas** pulls `ks4:*` → pool `ks4backup` over WG ([nas-seed.md](nas-seed.md)) | waiting FTTH (< Sep 30); moved off nuc 2026-08-30 |
|
||||||
| off-site, ks2 (interim) | `incus-copy.sh -d ks2 -m push` at 02:00 until the nuc leg seeds | to re-enable once the restic seed finishes |
|
| off-site, ks2 (interim) | `incus-copy.sh -d ks2 -m push` at 02:00 until the nas leg seeds | to re-enable once the restic seed finishes |
|
||||||
|
|
||||||
## Actions (backup work documented in [`../ks4/`](../ks4/), ks2-only tasks here)
|
## Actions (backup work documented in [`../ks4/`](../ks4/), ks2-only tasks here)
|
||||||
|
|
||||||
@@ -65,9 +66,9 @@ when nuc takes over.
|
|||||||
([restic-backup.md](../ks4/restic-backup.md), live 2026-08-28; the
|
([restic-backup.md](../ks4/restic-backup.md), live 2026-08-28; the
|
||||||
predecessor's doc is kept as reference)
|
predecessor's doc is kept as reference)
|
||||||
4. ~~instance leg to S3~~ — **shelved 2026-08-28**: instances are
|
4. ~~instance leg to S3~~ — **shelved 2026-08-28**: instances are
|
||||||
protected by replication (sdb + nuc/ks2), their data and configs by
|
protected by replication (sdb + nas/ks2), their data and configs by
|
||||||
`restic-data`
|
`restic-data`
|
||||||
5. [nuc-seed.md](nuc-seed.md) — **prepared**; after FTTH: seed nuc
|
5. [nas-seed.md](nas-seed.md) — **prepared**; after FTTH: seed the nas
|
||||||
pull leg, verify all instances, test-restore one
|
pull leg, verify all instances, test-restore one
|
||||||
6. [decommission.md](decommission.md) — **prepared**; cut flows,
|
6. [decommission.md](decommission.md) — **prepared**; cut flows,
|
||||||
final diff of `/backup/ns3061243`, wipe pools, terminate at OVH
|
final diff of `/backup/ns3061243`, wipe pools, terminate at OVH
|
||||||
@@ -77,7 +78,7 @@ when nuc takes over.
|
|||||||
- [x] biwiki + spot consciously abandoned (2026-08-22)
|
- [x] biwiki + spot consciously abandoned (2026-08-22)
|
||||||
- [x] local leg cron running since 2026-08-22, **18/18 instances**
|
- [x] local leg cron running since 2026-08-22, **18/18 instances**
|
||||||
replicated (verified 2026-08-28)
|
replicated (verified 2026-08-28)
|
||||||
- [ ] nuc leg fully seeded **and** one instance test-restored
|
- [ ] **nas** leg fully seeded **and** one instance test-restored
|
||||||
- [x] restic S3 backups live (05:00) **and** restore drill passed
|
- [x] restic S3 backups live (05:00) **and** restore drill passed
|
||||||
2026-08-28: tree restored byte-identical to live, dump restored
|
2026-08-28: tree restored byte-identical to live, dump restored
|
||||||
and loaded into a scratch MariaDB (12/12 tables)
|
and loaded into a scratch MariaDB (12/12 tables)
|
||||||
|
|||||||
Reference in New Issue
Block a user