Julien LutranandClaude Opus 5
3c6b762038
ef88ef9 updated the headings but left the body describing the old setup,
so the two docs contradicted each other on the same procedure.
The one that would actually have failed: `incus storage create ks4backup
zfs source=usb4t/backup/ks4` — that pool no longer exists, the disk moved
to nas and the pool was renamed to `tank` on import. Also corrects the
WireGuard peer (nas is 10.8.0.22/32, not nuc's 10.8.0.20/32, which is
retired once nas is seeded), the host for the cron and the restore test,
and drops the "replicas live only on the USB drive" framing — direct SATA
was the entire point of the rebuild.
Adds a pointer to ks2/nas-seed.md as the authoritative seed procedure.
Historical notes (the 2026-08-09 verification, the homeassistant VM
measurement) are left as-is: they are dated observations, not steps.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
165 lines
7.5 KiB
Markdown
165 lines
7.5 KiB
Markdown
# incus-copy — ks4 instance replication
|
|
|
|
Incremental replication of all ks4 Incus instances via
|
|
[`scripts/incus-copy.sh`](https://git.lutran.fr/julien/scripts/src/branch/main/incus-copy.sh)
|
|
(`incus copy --refresh`: ZFS-incremental, produces ready-to-start
|
|
replicas instead of tarballs). Two legs replace the historical nightly
|
|
push to `ks2` (decommissioning):
|
|
|
|
1. **local** — replicas + dumps on a dedicated `backup` zpool on ks4's
|
|
second disk (`sdb5`), survives `sda` death
|
|
2. **off-site** — replicas pulled by **nas** into pool `ks4backup`
|
|
(dataset `tank/backup/ks4`), survives losing ks4 entirely
|
|
|
|
## The script
|
|
|
|
```
|
|
incus-copy.sh -d <dest-remote> [-m pull|push|relay] [-s <dest-pool>] # push local → remote
|
|
incus-copy.sh -r <src-remote> [-m pull|push|relay] [-s <dest-pool>] # pull remote → local
|
|
incus-copy.sh -p <dest-project> -s <dest-pool> # local → local project
|
|
```
|
|
|
|
- Copies **all** instances, running and stopped — a stopped instance's
|
|
filesystem never changes, so its refresh transfers nothing after the
|
|
first copy.
|
|
- `flock` on `/run/lock/incus-copy.lock` — overlapping cron runs abort
|
|
instead of racing the same instance.
|
|
- After each successful copy it sets `boot.autostart=false` on the
|
|
**replica** — otherwise the copy inherits autostart and would start
|
|
itself on the backup host after a reboot (or, for the local-project
|
|
leg, fight the live instance for its static IP on the same bridge).
|
|
- Exits non-zero if any instance failed (cron/monitoring can alert);
|
|
per-instance failures go to stderr, progress to stdout.
|
|
- Snapshot history is the **source's** job — set on ks4, e.g.:
|
|
`incus profile set default snapshots.schedule="0 3 * * *" snapshots.expiry=7d`.
|
|
`--refresh-exclude-older` keeps expired snapshots from being re-sent.
|
|
- Consistency: copies of running instances are crash-consistent;
|
|
databases may need recovery on restore. Snapshot history mitigates,
|
|
and [`incus-backup.sh`](https://git.lutran.fr/julien/scripts/src/branch/main/incus-backup.sh) (proper
|
|
`mariadb-dump`s + selected paths, manifest-driven) covers the DBs.
|
|
- **VMs**: `--refresh` re-sends a VM's **full block volume** every run
|
|
unless source and replica share a common snapshot — containers diff
|
|
cheaply regardless (verified on nuc: homeassistant re-sent 3.3 GiB,
|
|
containers only KBs). Set `snapshots.schedule` on sources *before*
|
|
enabling the crons, especially for the WAN leg.
|
|
|
|
## Leg 1 — local backup pool on ks4 (sdb5)
|
|
|
|
One-time setup (root on ks4). `sdb1-4` mirror the OS (mdraid); `sdb5`
|
|
was unused:
|
|
|
|
```sh
|
|
# find sdb's serial: ls -l /dev/disk/by-id/ | grep sdb5
|
|
zpool create -m none backup /dev/disk/by-id/ata-HGST_HUS726T6TALE6L1_<sdb-serial>-part5
|
|
zfs create backup/incus # incus-managed (replicas)
|
|
zfs create -o mountpoint=/backup backup/dumps # incus-backup.sh output
|
|
incus storage create backup zfs source=backup/incus
|
|
# replicas live in their own project so names don't collide with the
|
|
# live instances; it shares the default project's profiles
|
|
incus project create backup -c features.images=false -c features.profiles=false
|
|
```
|
|
|
|
Cron (root on ks4) — replaces both ks2 jobs:
|
|
|
|
```cron
|
|
0 1 * * * /root/scripts/incus-copy.sh -p backup -s backup >> /var/log/incus-copy.log 2>&1
|
|
# retired 2026-08-28 — dumps + data trees are restic's job (05:00),
|
|
# see ../backup-strategy.md and restic-backup.md
|
|
#0 4 * * * /root/scripts/incus-backup.sh -f /root/scripts/incus-backup.db -s data >> /var/log/incus-backup.log 2>&1
|
|
```
|
|
|
|
(`incus-backup.sh` without `-d` now writes locally to
|
|
`/backup/$(hostname -s)/` — the `backup/dumps` dataset.)
|
|
|
|
⚠️ Replicas in the `backup` project must stay **stopped** — they keep
|
|
the live containers' static `192.168.1.x` addresses.
|
|
|
|
## Leg 2 — off-site pull from nas (was nuc until 2026-08-30)
|
|
|
|
Storage pool on **nas** (recreated 2026-08-30): `ks4backup`, backed by
|
|
the dataset `tank/backup/ks4` — the same 4 TB disk, now on **direct
|
|
SATA** instead of the USB enclosure whose bridge kept suspending the pool
|
|
([nuc/usb4t-dropouts.md](../nuc/usb4t-dropouts.md)). Full replica set is
|
|
~1.75 TiB.
|
|
|
|
```sh
|
|
incus storage create ks4backup zfs source=tank/backup/ks4
|
|
```
|
|
|
|
⚠️ Originally this lived on nuc as `usb4t/backup/ks4`. That pool no
|
|
longer exists — the disk moved to nas and the pool was renamed on import
|
|
([nas/nas-install.md](../nas/nas-install.md) §5b). Following the old
|
|
command fails with "no such pool".
|
|
|
|
`tank` is a single vdev, so if the disk fails only backups are lost —
|
|
nas's own instances live on the `incus` SSD mirror and are unaffected.
|
|
|
|
**Direction: nas pulls, through the WireGuard tunnel.** Verified
|
|
2026-08-09: ks4's API listens on wildcard `:8443` (so it answers on
|
|
`192.168.1.1`, the `incusbr0` host address) but is **firewalled from
|
|
the internet** — the VPN path keeps it that way, needs no inbound port
|
|
at home, and doesn't care that the home public IP is dynamic. The
|
|
`wireguard` container (`192.168.1.18`, `wg0` `10.8.0.1/24`) is exposed
|
|
via a proxy device on public UDP `51845`.
|
|
|
|
Setup (✅ **done 2026-08-09**, verified end-to-end with
|
|
`incus list ks4:` from nuc):
|
|
|
|
- **wireguard container** (ks4): forwards + masquerades wg0→eth0
|
|
(pre-existing). **nas** is the peer for this leg —
|
|
`wg set wg0 peer <nas-pubkey> allowed-ips 10.8.0.22/32` +
|
|
`wg-quick save wg0`. (`10.8.0.20/32` was nuc's peer for the same leg
|
|
and is retired once nas is seeded; `10.8.0.21` is transmission-bt's
|
|
own in-container tunnel and is unrelated.)
|
|
- **ufw** (ks4): `ufw allow in on incusbr0 from 192.168.1.18 to any
|
|
port 8443 proto tcp` — the API stays firewalled from the internet
|
|
and the connection arrives masqueraded as the WG container.
|
|
- **nas**: `/etc/wireguard/wg-ks4.conf` (`wg-quick@wg-ks4` enabled,
|
|
`Address = 10.8.0.22/32`;
|
|
peer = container pubkey `TVs6d7…`, endpoint `193.70.35.17:51845`,
|
|
`AllowedIPs = 10.8.0.0/24, 192.168.1.1/32`, keepalive 25s) and
|
|
`incus remote add ks4 https://192.168.1.1:8443 --accept-certificate
|
|
--token '…'` (fingerprint cross-checked against the token).
|
|
|
|
In pull mode all control + migration traffic flows over that single
|
|
API connection, so nothing else needs routing. Copying the `wireguard`
|
|
container over its own tunnel is fine (crash-consistent, tiny, no
|
|
interruption); if the tunnel is down the cron job fails loudly instead
|
|
of hanging.
|
|
|
|
Then cron (root on **nas**) — stagger after ks4's local leg and after
|
|
nas's own 04:00 local replication:
|
|
|
|
```cron
|
|
30 2 * * * /root/scripts/incus-copy.sh -r ks4 -s ks4backup >> /var/log/incus-copy.log 2>&1
|
|
```
|
|
|
|
The **initial seed is ~1.75 TiB** over the home downlink — run the
|
|
first `incus-copy.sh -r ks4 -s ks4backup` manually (tmux), enable the
|
|
cron once it completes.
|
|
|
|
## Cutover checklist (then kill ks2)
|
|
|
|
1. First full cycle of all three jobs clean (logs above).
|
|
2. Restore test: on nas, start a small replica (e.g. `freshrss`) with
|
|
its NIC detached, check app data, then stop it.
|
|
3. Remove both ks2 cron lines on ks4, `incus remote remove ks2`,
|
|
cancel the server (`164.132.173.57` = ks2, rsync target of the old
|
|
4 AM job).
|
|
|
|
## Restore
|
|
|
|
```sh
|
|
# from nas (off-site replica):
|
|
incus copy <instance> ks4:<instance> --mode push
|
|
# from the local backup project (sda replaced, pool data rebuilt):
|
|
incus copy <instance> <instance> --project backup --target-project default -s data
|
|
```
|
|
|
|
The step-by-step seed procedure, including the WireGuard move and the
|
|
gate it depends on, is [ks2/nas-seed.md](../ks2/nas-seed.md) — that is
|
|
the authoritative version for this leg.
|
|
|
|
Remember replicas have `boot.autostart=false`; re-enable after a real
|
|
failover, and re-check it after copying back to ks4.
|