doc/ks4/incus-copy.md

incus-copy — ks4 instance replication

Incremental replication of all ks4 Incus instances via scripts/incus-copy.sh (incus copy --refresh: ZFS-incremental, produces ready-to-start replicas instead of tarballs). Two legs replace the historical nightly push to ks2 (decommissioning):

1. local — replicas + dumps on a dedicated backup zpool on ks4's second disk (sdb5), survives sda death
2. off-site — replicas pulled by nuc into pool ks4backup (dataset usb4t/backup/ks4), survives losing ks4 entirely

The script

incus-copy.sh -d <dest-remote> [-m pull|push|relay] [-s <dest-pool>]   # push local → remote
incus-copy.sh -r <src-remote>  [-m pull|push|relay] [-s <dest-pool>]   # pull remote → local
incus-copy.sh -p <dest-project> -s <dest-pool>                         # local → local project

• Copies all instances, running and stopped — a stopped instance's filesystem never changes, so its refresh transfers nothing after the first copy.
• flock on /run/lock/incus-copy.lock — overlapping cron runs abort instead of racing the same instance.
• After each successful copy it sets boot.autostart=false on the replica — otherwise the copy inherits autostart and would start itself on the backup host after a reboot (or, for the local-project leg, fight the live instance for its static IP on the same bridge).
• Exits non-zero if any instance failed (cron/monitoring can alert); per-instance failures go to stderr, progress to stdout.
• Snapshot history is the source's job — set on ks4, e.g.: incus profile set default snapshots.schedule="0 3 * * *" snapshots.expiry=7d. --refresh-exclude-older keeps expired snapshots from being re-sent.
• Consistency: copies of running instances are crash-consistent; databases may need recovery on restore. Snapshot history mitigates, and incus-backup.sh (proper mariadb-dumps + selected paths, manifest-driven) covers the DBs.
• VMs: --refresh re-sends a VM's full block volume every run unless source and replica share a common snapshot — containers diff cheaply regardless (verified on nuc: homeassistant re-sent 3.3 GiB, containers only KBs). Set snapshots.schedule on sources before enabling the crons, especially for the WAN leg.

Leg 1 — local backup pool on ks4 (sdb5)

One-time setup (root on ks4). sdb1-4 mirror the OS (mdraid); sdb5 was unused:

# find sdb's serial: ls -l /dev/disk/by-id/ | grep sdb5
zpool create -m none backup /dev/disk/by-id/ata-HGST_HUS726T6TALE6L1_<sdb-serial>-part5
zfs create backup/incus                        # incus-managed (replicas)
zfs create -o mountpoint=/backup backup/dumps  # incus-backup.sh output
incus storage create backup zfs source=backup/incus
# replicas live in their own project so names don't collide with the
# live instances; it shares the default project's profiles
incus project create backup -c features.images=false -c features.profiles=false

Cron (root on ks4) — replaces both ks2 jobs:

0 1 * * * /root/scripts/incus-copy.sh -p backup -s backup >> /var/log/incus-copy.log 2>&1
0 4 * * * /root/scripts/incus-backup.sh -f /root/scripts/incus-backup.db -s data >> /var/log/incus-backup.log 2>&1

(incus-backup.sh without -d now writes locally to /backup/$(hostname -s)/ — the backup/dumps dataset.)

⚠️ Replicas in the backup project must stay stopped — they keep the live containers' static 192.168.1.x addresses.

Leg 2 — off-site pull from nuc

Storage pool on nuc (done 2026-08-09): ks4backup, backed by the dataset usb4t/backup/ks4 on the USB 4 TB pool (quota on usb4t/backup removed 2026-08-09 — full replica set is ~1.75 TiB):

incus storage create ks4backup zfs source=usb4t/backup/ks4

Replicas live only on the USB drive — if it fails, only backups are lost; nuc's own instances (pool data on the SSD) are unaffected.

Direction: nuc pulls, through the WireGuard tunnel. Verified 2026-08-09: ks4's API listens on wildcard :8443 (so it answers on 192.168.1.1, the incusbr0 host address) but is firewalled from the internet — the VPN path keeps it that way, needs no inbound port at home, and doesn't care that nuc's public IP is dynamic. The wireguard container (192.168.1.18, wg0 10.8.0.1/24) is exposed via a proxy device on public UDP 51845.

Setup (✅ done 2026-08-09, verified end-to-end with incus list ks4: from nuc):

• wireguard container (ks4): forwards + masquerades wg0→eth0 (pre-existing); nuc added as peer 10.8.0.20/32 (wg set wg0 peer 31Tlgloc… allowed-ips 10.8.0.20/32 + wg-quick save wg0).
• ufw (ks4): ufw allow in on incusbr0 from 192.168.1.18 to any port 8443 proto tcp — the API stays firewalled from the internet and the connection arrives masqueraded as the WG container.
• nuc: /etc/wireguard/wg-ks4.conf (wg-quick@wg-ks4 enabled; peer = container pubkey TVs6d7…, endpoint 193.70.35.17:51845, AllowedIPs = 10.8.0.0/24, 192.168.1.1/32, keepalive 25s) and incus remote add ks4 https://192.168.1.1:8443 --accept-certificate --token '…' (fingerprint cross-checked against the token).

In pull mode all control + migration traffic flows over that single API connection, so nothing else needs routing. Copying the wireguard container over its own tunnel is fine (crash-consistent, tiny, no interruption); if the tunnel is down the cron job fails loudly instead of hanging.

Then cron (root on nuc) — stagger after ks4's local leg:

30 2 * * * /root/scripts/incus-copy.sh -r ks4 -s ks4backup >> /var/log/incus-copy.log 2>&1

The initial seed is ~1.75 TiB over the home downlink — run the first incus-copy.sh -r ks4 -s ks4backup manually (tmux), enable the cron once it completes.

Cutover checklist (then kill ks2)

1. First full cycle of all three jobs clean (logs above).
2. Restore test: on nuc, start a small replica (e.g. freshrss) with its NIC detached, check app data, then stop it.
3. Remove both ks2 cron lines on ks4, incus remote remove ks2, cancel the server (164.132.173.57 = ks2, rsync target of the old 4 AM job).

Restore

# from nuc (off-site replica):
incus copy <instance> ks4:<instance> --mode push
# from the local backup project (sda replaced, pool data rebuilt):
incus copy <instance> <instance> --project backup --target-project default -s data

Remember replicas have boot.autostart=false; re-enable after a real failover, and re-check it after copying back to ks4.