Julien LutranandClaude Fable 5
ee9eaee889
README with instance tables and nuc<->ks4 network flow chart; per-container install/troubleshooting docs for nuc (jellyfin server/client, transmission-bt, bare-metal reinstall) and the ks4 two-leg backup scheme (incus-copy over wireguard). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
148 lines
6.5 KiB
Markdown
148 lines
6.5 KiB
Markdown
# incus-copy — ks4 instance replication
|
|
|
|
Incremental replication of all ks4 Incus instances via
|
|
[`scripts/incus-copy.sh`](../scripts/incus-copy.sh)
|
|
(`incus copy --refresh`: ZFS-incremental, produces ready-to-start
|
|
replicas instead of tarballs). Two legs replace the historical nightly
|
|
push to `ks2` (decommissioning):
|
|
|
|
1. **local** — replicas + dumps on a dedicated `backup` zpool on ks4's
|
|
second disk (`sdb5`), survives `sda` death
|
|
2. **off-site** — replicas pulled by **nuc** into pool `ks4backup`
|
|
(dataset `usb4t/backup/ks4`), survives losing ks4 entirely
|
|
|
|
## The script
|
|
|
|
```
|
|
incus-copy.sh -d <dest-remote> [-m pull|push|relay] [-s <dest-pool>] # push local → remote
|
|
incus-copy.sh -r <src-remote> [-m pull|push|relay] [-s <dest-pool>] # pull remote → local
|
|
incus-copy.sh -p <dest-project> -s <dest-pool> # local → local project
|
|
```
|
|
|
|
- Copies **all** instances, running and stopped — a stopped instance's
|
|
filesystem never changes, so its refresh transfers nothing after the
|
|
first copy.
|
|
- `flock` on `/run/lock/incus-copy.lock` — overlapping cron runs abort
|
|
instead of racing the same instance.
|
|
- After each successful copy it sets `boot.autostart=false` on the
|
|
**replica** — otherwise the copy inherits autostart and would start
|
|
itself on the backup host after a reboot (or, for the local-project
|
|
leg, fight the live instance for its static IP on the same bridge).
|
|
- Exits non-zero if any instance failed (cron/monitoring can alert);
|
|
per-instance failures go to stderr, progress to stdout.
|
|
- Snapshot history is the **source's** job — set on ks4, e.g.:
|
|
`incus profile set default snapshots.schedule="0 3 * * *" snapshots.expiry=7d`.
|
|
`--refresh-exclude-older` keeps expired snapshots from being re-sent.
|
|
- Consistency: copies of running instances are crash-consistent;
|
|
databases may need recovery on restore. Snapshot history mitigates,
|
|
and [`incus-backup.sh`](../scripts/incus-backup.sh) (proper
|
|
`mariadb-dump`s + selected paths, manifest-driven) covers the DBs.
|
|
- **VMs**: `--refresh` re-sends a VM's **full block volume** every run
|
|
unless source and replica share a common snapshot — containers diff
|
|
cheaply regardless (verified on nuc: homeassistant re-sent 3.3 GiB,
|
|
containers only KBs). Set `snapshots.schedule` on sources *before*
|
|
enabling the crons, especially for the WAN leg.
|
|
|
|
## Leg 1 — local backup pool on ks4 (sdb5)
|
|
|
|
One-time setup (root on ks4). `sdb1-4` mirror the OS (mdraid); `sdb5`
|
|
was unused:
|
|
|
|
```sh
|
|
# find sdb's serial: ls -l /dev/disk/by-id/ | grep sdb5
|
|
zpool create -m none backup /dev/disk/by-id/ata-HGST_HUS726T6TALE6L1_<sdb-serial>-part5
|
|
zfs create backup/incus # incus-managed (replicas)
|
|
zfs create -o mountpoint=/backup backup/dumps # incus-backup.sh output
|
|
incus storage create backup zfs source=backup/incus
|
|
# replicas live in their own project so names don't collide with the
|
|
# live instances; it shares the default project's profiles
|
|
incus project create backup -c features.images=false -c features.profiles=false
|
|
```
|
|
|
|
Cron (root on ks4) — replaces both ks2 jobs:
|
|
|
|
```cron
|
|
0 1 * * * /root/scripts/incus-copy.sh -p backup -s backup >> /var/log/incus-copy.log 2>&1
|
|
0 4 * * * /root/scripts/incus-backup.sh -f /root/scripts/incus-backup.db -s data >> /var/log/incus-backup.log 2>&1
|
|
```
|
|
|
|
(`incus-backup.sh` without `-d` now writes locally to
|
|
`/backup/$(hostname -s)/` — the `backup/dumps` dataset.)
|
|
|
|
⚠️ Replicas in the `backup` project must stay **stopped** — they keep
|
|
the live containers' static `192.168.1.x` addresses.
|
|
|
|
## Leg 2 — off-site pull from nuc
|
|
|
|
Storage pool on nuc (done 2026-08-09): `ks4backup`, backed by the
|
|
dataset `usb4t/backup/ks4` on the USB 4 TB pool (quota on `usb4t/backup`
|
|
removed 2026-08-09 — full replica set is ~1.75 TiB):
|
|
|
|
```sh
|
|
incus storage create ks4backup zfs source=usb4t/backup/ks4
|
|
```
|
|
|
|
Replicas live only on the USB drive — if it fails, only backups are
|
|
lost; nuc's own instances (pool `data` on the SSD) are unaffected.
|
|
|
|
**Direction: nuc pulls, through the WireGuard tunnel.** Verified
|
|
2026-08-09: ks4's API listens on wildcard `:8443` (so it answers on
|
|
`192.168.1.1`, the `incusbr0` host address) but is **firewalled from
|
|
the internet** — the VPN path keeps it that way, needs no inbound port
|
|
at home, and doesn't care that nuc's public IP is dynamic. The
|
|
`wireguard` container (`192.168.1.18`, `wg0` `10.8.0.1/24`) is exposed
|
|
via a proxy device on public UDP `51845`.
|
|
|
|
Setup (✅ **done 2026-08-09**, verified end-to-end with
|
|
`incus list ks4:` from nuc):
|
|
|
|
- **wireguard container** (ks4): forwards + masquerades wg0→eth0
|
|
(pre-existing); nuc added as peer `10.8.0.20/32`
|
|
(`wg set wg0 peer 31Tlgloc… allowed-ips 10.8.0.20/32` +
|
|
`wg-quick save wg0`).
|
|
- **ufw** (ks4): `ufw allow in on incusbr0 from 192.168.1.18 to any
|
|
port 8443 proto tcp` — the API stays firewalled from the internet
|
|
and the connection arrives masqueraded as the WG container.
|
|
- **nuc**: `/etc/wireguard/wg-ks4.conf` (`wg-quick@wg-ks4` enabled;
|
|
peer = container pubkey `TVs6d7…`, endpoint `193.70.35.17:51845`,
|
|
`AllowedIPs = 10.8.0.0/24, 192.168.1.1/32`, keepalive 25s) and
|
|
`incus remote add ks4 https://192.168.1.1:8443 --accept-certificate
|
|
--token '…'` (fingerprint cross-checked against the token).
|
|
|
|
In pull mode all control + migration traffic flows over that single
|
|
API connection, so nothing else needs routing. Copying the `wireguard`
|
|
container over its own tunnel is fine (crash-consistent, tiny, no
|
|
interruption); if the tunnel is down the cron job fails loudly instead
|
|
of hanging.
|
|
|
|
Then cron (root on nuc) — stagger after ks4's local leg:
|
|
|
|
```cron
|
|
30 2 * * * /root/scripts/incus-copy.sh -r ks4 -s ks4backup >> /var/log/incus-copy.log 2>&1
|
|
```
|
|
|
|
The **initial seed is ~1.75 TiB** over the home downlink — run the
|
|
first `incus-copy.sh -r ks4 -s ks4backup` manually (tmux), enable the
|
|
cron once it completes.
|
|
|
|
## Cutover checklist (then kill ks2)
|
|
|
|
1. First full cycle of all three jobs clean (logs above).
|
|
2. Restore test: on nuc, start a small replica (e.g. `freshrss`) with
|
|
its NIC detached, check app data, then stop it.
|
|
3. Remove both ks2 cron lines on ks4, `incus remote remove ks2`,
|
|
cancel the server (`164.132.173.57` = ks2, rsync target of the old
|
|
4 AM job).
|
|
|
|
## Restore
|
|
|
|
```sh
|
|
# from nuc (off-site replica):
|
|
incus copy <instance> ks4:<instance> --mode push
|
|
# from the local backup project (sda replaced, pool data rebuilt):
|
|
incus copy <instance> <instance> --project backup --target-project default -s data
|
|
```
|
|
|
|
Remember replicas have `boot.autostart=false`; re-enable after a real
|
|
failover, and re-check it after copying back to ks4.
|