# incus-copy — ks4 instance replication Incremental replication of all ks4 Incus instances via [`scripts/incus-copy.sh`](https://git.lutran.fr/julien/scripts/src/branch/main/incus-copy.sh) (`incus copy --refresh`: ZFS-incremental, produces ready-to-start replicas instead of tarballs). Two legs replace the historical nightly push to `ks2` (decommissioning): 1. **local** — replicas + dumps on a dedicated `backup` zpool on ks4's second disk (`sdb5`), survives `sda` death 2. **off-site** — replicas pulled by **nas** into pool `ks4backup` (dataset `tank/backup/ks4`), survives losing ks4 entirely ## The script ``` incus-copy.sh -d [-m pull|push|relay] [-s ] # push local → remote incus-copy.sh -r [-m pull|push|relay] [-s ] # pull remote → local incus-copy.sh -p -s # local → local project ``` - Copies **all** instances, running and stopped — a stopped instance's filesystem never changes, so its refresh transfers nothing after the first copy. - `flock` on `/run/lock/incus-copy.lock` — overlapping cron runs abort instead of racing the same instance. - After each successful copy it sets `boot.autostart=false` on the **replica** — otherwise the copy inherits autostart and would start itself on the backup host after a reboot (or, for the local-project leg, fight the live instance for its static IP on the same bridge). - Exits non-zero if any instance failed (cron/monitoring can alert); per-instance failures go to stderr, progress to stdout. - Snapshot history is the **source's** job — set on ks4, e.g.: `incus profile set default snapshots.schedule="0 3 * * *" snapshots.expiry=7d`. `--refresh-exclude-older` keeps expired snapshots from being re-sent. - Consistency: copies of running instances are crash-consistent; databases may need recovery on restore. Snapshot history mitigates, and [`incus-backup.sh`](https://git.lutran.fr/julien/scripts/src/branch/main/incus-backup.sh) (proper `mariadb-dump`s + selected paths, manifest-driven) covers the DBs. - **VMs**: `--refresh` re-sends a VM's **full block volume** every run unless source and replica share a common snapshot — containers diff cheaply regardless (verified on nuc: homeassistant re-sent 3.3 GiB, containers only KBs). Set `snapshots.schedule` on sources *before* enabling the crons, especially for the WAN leg. ## Leg 1 — local backup pool on ks4 (sdb5) One-time setup (root on ks4). `sdb1-4` mirror the OS (mdraid); `sdb5` was unused: ```sh # find sdb's serial: ls -l /dev/disk/by-id/ | grep sdb5 zpool create -m none backup /dev/disk/by-id/ata-HGST_HUS726T6TALE6L1_-part5 zfs create backup/incus # incus-managed (replicas) zfs create -o mountpoint=/backup backup/dumps # incus-backup.sh output incus storage create backup zfs source=backup/incus # replicas live in their own project so names don't collide with the # live instances; it shares the default project's profiles incus project create backup -c features.images=false -c features.profiles=false ``` Cron (root on ks4) — replaces both ks2 jobs: ```cron 0 1 * * * /root/scripts/incus-copy.sh -p backup -s backup >> /var/log/incus-copy.log 2>&1 # retired 2026-08-28 — dumps + data trees are restic's job (05:00), # see ../backup-strategy.md and restic-backup.md #0 4 * * * /root/scripts/incus-backup.sh -f /root/scripts/incus-backup.db -s data >> /var/log/incus-backup.log 2>&1 ``` (`incus-backup.sh` without `-d` now writes locally to `/backup/$(hostname -s)/` — the `backup/dumps` dataset.) ⚠️ Replicas in the `backup` project must stay **stopped** — they keep the live containers' static `192.168.1.x` addresses. ## Leg 2 — off-site pull from nas (was nuc until 2026-08-30) Storage pool on **nas** (recreated 2026-08-30): `ks4backup`, backed by the dataset `tank/backup/ks4` — the same 4 TB disk, now on **direct SATA** instead of the USB enclosure whose bridge kept suspending the pool ([nuc/usb4t-dropouts.md](../nuc/usb4t-dropouts.md)). Full replica set is ~1.75 TiB. ```sh incus storage create ks4backup zfs source=tank/backup/ks4 ``` ⚠️ Originally this lived on nuc as `usb4t/backup/ks4`. That pool no longer exists — the disk moved to nas and the pool was renamed on import ([nas/nas-install.md](../nas/nas-install.md) §5b). Following the old command fails with "no such pool". `tank` is a single vdev, so if the disk fails only backups are lost — nas's own instances live on the `incus` SSD mirror and are unaffected. **Direction: nas pulls, through the WireGuard tunnel.** Verified 2026-08-09: ks4's API listens on wildcard `:8443` (so it answers on `192.168.1.1`, the `incusbr0` host address) but is **firewalled from the internet** — the VPN path keeps it that way, needs no inbound port at home, and doesn't care that the home public IP is dynamic. The `wireguard` container (`192.168.1.18`, `wg0` `10.8.0.1/24`) is exposed via a proxy device on public UDP `51845`. Setup (✅ **done 2026-08-09**, verified end-to-end with `incus list ks4:` from nuc): - **wireguard container** (ks4): forwards + masquerades wg0→eth0 (pre-existing). **nas** is the peer for this leg — `wg set wg0 peer allowed-ips 10.8.0.22/32` + `wg-quick save wg0`. (`10.8.0.20/32` was nuc's peer for the same leg and is retired once nas is seeded; `10.8.0.21` is transmission-bt's own in-container tunnel and is unrelated.) - **ufw** (ks4): `ufw allow in on incusbr0 from 192.168.1.18 to any port 8443 proto tcp` — the API stays firewalled from the internet and the connection arrives masqueraded as the WG container. - **nas**: `/etc/wireguard/wg-ks4.conf` (`wg-quick@wg-ks4` enabled, `Address = 10.8.0.22/32`; peer = container pubkey `TVs6d7…`, endpoint `193.70.35.17:51845`, `AllowedIPs = 10.8.0.0/24, 192.168.1.1/32`, keepalive 25s) and `incus remote add ks4 https://192.168.1.1:8443 --accept-certificate --token '…'` (fingerprint cross-checked against the token). In pull mode all control + migration traffic flows over that single API connection, so nothing else needs routing. Copying the `wireguard` container over its own tunnel is fine (crash-consistent, tiny, no interruption); if the tunnel is down the cron job fails loudly instead of hanging. Then cron (root on **nas**) — stagger after ks4's local leg and after nas's own 04:00 local replication: ```cron 30 2 * * * /root/scripts/incus-copy.sh -r ks4 -s ks4backup >> /var/log/incus-copy.log 2>&1 ``` The **initial seed is ~1.75 TiB** over the home downlink — run the first `incus-copy.sh -r ks4 -s ks4backup` manually (tmux), enable the cron once it completes. ## Cutover checklist (then kill ks2) 1. First full cycle of all three jobs clean (logs above). 2. Restore test: on nas, start a small replica (e.g. `freshrss`) with its NIC detached, check app data, then stop it. 3. Remove both ks2 cron lines on ks4, `incus remote remove ks2`, cancel the server (`164.132.173.57` = ks2, rsync target of the old 4 AM job). ## Restore ```sh # from nas (off-site replica): incus copy ks4: --mode push # from the local backup project (sda replaced, pool data rebuilt): incus copy --project backup --target-project default -s data ``` The step-by-step seed procedure, including the WireGuard move and the gate it depends on, is [ks2/nas-seed.md](../ks2/nas-seed.md) — that is the authoritative version for this leg. Remember replicas have `boot.autostart=false`; re-enable after a real failover, and re-check it after copying back to ks4.