Initial import: nuc and ks4 infrastructure documentation
README with instance tables and nuc<->ks4 network flow chart; per-container install/troubleshooting docs for nuc (jellyfin server/client, transmission-bt, bare-metal reinstall) and the ks4 two-leg backup scheme (incus-copy over wireguard). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Julien Lutran
Claude Fable 5
@@ -0,0 +1,147 @@
|
||||
# incus-copy — ks4 instance replication
|
||||
|
||||
Incremental replication of all ks4 Incus instances via
|
||||
[`scripts/incus-copy.sh`](../scripts/incus-copy.sh)
|
||||
(`incus copy --refresh`: ZFS-incremental, produces ready-to-start
|
||||
replicas instead of tarballs). Two legs replace the historical nightly
|
||||
push to `ks2` (decommissioning):
|
||||
|
||||
1. **local** — replicas + dumps on a dedicated `backup` zpool on ks4's
|
||||
second disk (`sdb5`), survives `sda` death
|
||||
2. **off-site** — replicas pulled by **nuc** into pool `ks4backup`
|
||||
(dataset `usb4t/backup/ks4`), survives losing ks4 entirely
|
||||
|
||||
## The script
|
||||
|
||||
```
|
||||
incus-copy.sh -d <dest-remote> [-m pull|push|relay] [-s <dest-pool>] # push local → remote
|
||||
incus-copy.sh -r <src-remote> [-m pull|push|relay] [-s <dest-pool>] # pull remote → local
|
||||
incus-copy.sh -p <dest-project> -s <dest-pool> # local → local project
|
||||
```
|
||||
|
||||
- Copies **all** instances, running and stopped — a stopped instance's
|
||||
filesystem never changes, so its refresh transfers nothing after the
|
||||
first copy.
|
||||
- `flock` on `/run/lock/incus-copy.lock` — overlapping cron runs abort
|
||||
instead of racing the same instance.
|
||||
- After each successful copy it sets `boot.autostart=false` on the
|
||||
**replica** — otherwise the copy inherits autostart and would start
|
||||
itself on the backup host after a reboot (or, for the local-project
|
||||
leg, fight the live instance for its static IP on the same bridge).
|
||||
- Exits non-zero if any instance failed (cron/monitoring can alert);
|
||||
per-instance failures go to stderr, progress to stdout.
|
||||
- Snapshot history is the **source's** job — set on ks4, e.g.:
|
||||
`incus profile set default snapshots.schedule="0 3 * * *" snapshots.expiry=7d`.
|
||||
`--refresh-exclude-older` keeps expired snapshots from being re-sent.
|
||||
- Consistency: copies of running instances are crash-consistent;
|
||||
databases may need recovery on restore. Snapshot history mitigates,
|
||||
and [`incus-backup.sh`](../scripts/incus-backup.sh) (proper
|
||||
`mariadb-dump`s + selected paths, manifest-driven) covers the DBs.
|
||||
- **VMs**: `--refresh` re-sends a VM's **full block volume** every run
|
||||
unless source and replica share a common snapshot — containers diff
|
||||
cheaply regardless (verified on nuc: homeassistant re-sent 3.3 GiB,
|
||||
containers only KBs). Set `snapshots.schedule` on sources *before*
|
||||
enabling the crons, especially for the WAN leg.
|
||||
|
||||
## Leg 1 — local backup pool on ks4 (sdb5)
|
||||
|
||||
One-time setup (root on ks4). `sdb1-4` mirror the OS (mdraid); `sdb5`
|
||||
was unused:
|
||||
|
||||
```sh
|
||||
# find sdb's serial: ls -l /dev/disk/by-id/ | grep sdb5
|
||||
zpool create -m none backup /dev/disk/by-id/ata-HGST_HUS726T6TALE6L1_<sdb-serial>-part5
|
||||
zfs create backup/incus # incus-managed (replicas)
|
||||
zfs create -o mountpoint=/backup backup/dumps # incus-backup.sh output
|
||||
incus storage create backup zfs source=backup/incus
|
||||
# replicas live in their own project so names don't collide with the
|
||||
# live instances; it shares the default project's profiles
|
||||
incus project create backup -c features.images=false -c features.profiles=false
|
||||
```
|
||||
|
||||
Cron (root on ks4) — replaces both ks2 jobs:
|
||||
|
||||
```cron
|
||||
0 1 * * * /root/scripts/incus-copy.sh -p backup -s backup >> /var/log/incus-copy.log 2>&1
|
||||
0 4 * * * /root/scripts/incus-backup.sh -f /root/scripts/incus-backup.db -s data >> /var/log/incus-backup.log 2>&1
|
||||
```
|
||||
|
||||
(`incus-backup.sh` without `-d` now writes locally to
|
||||
`/backup/$(hostname -s)/` — the `backup/dumps` dataset.)
|
||||
|
||||
⚠️ Replicas in the `backup` project must stay **stopped** — they keep
|
||||
the live containers' static `192.168.1.x` addresses.
|
||||
|
||||
## Leg 2 — off-site pull from nuc
|
||||
|
||||
Storage pool on nuc (done 2026-08-09): `ks4backup`, backed by the
|
||||
dataset `usb4t/backup/ks4` on the USB 4 TB pool (quota on `usb4t/backup`
|
||||
removed 2026-08-09 — full replica set is ~1.75 TiB):
|
||||
|
||||
```sh
|
||||
incus storage create ks4backup zfs source=usb4t/backup/ks4
|
||||
```
|
||||
|
||||
Replicas live only on the USB drive — if it fails, only backups are
|
||||
lost; nuc's own instances (pool `data` on the SSD) are unaffected.
|
||||
|
||||
**Direction: nuc pulls, through the WireGuard tunnel.** Verified
|
||||
2026-08-09: ks4's API listens on wildcard `:8443` (so it answers on
|
||||
`192.168.1.1`, the `incusbr0` host address) but is **firewalled from
|
||||
the internet** — the VPN path keeps it that way, needs no inbound port
|
||||
at home, and doesn't care that nuc's public IP is dynamic. The
|
||||
`wireguard` container (`192.168.1.18`, `wg0` `10.8.0.1/24`) is exposed
|
||||
via a proxy device on public UDP `51845`.
|
||||
|
||||
Setup (✅ **done 2026-08-09**, verified end-to-end with
|
||||
`incus list ks4:` from nuc):
|
||||
|
||||
- **wireguard container** (ks4): forwards + masquerades wg0→eth0
|
||||
(pre-existing); nuc added as peer `10.8.0.20/32`
|
||||
(`wg set wg0 peer 31Tlgloc… allowed-ips 10.8.0.20/32` +
|
||||
`wg-quick save wg0`).
|
||||
- **ufw** (ks4): `ufw allow in on incusbr0 from 192.168.1.18 to any
|
||||
port 8443 proto tcp` — the API stays firewalled from the internet
|
||||
and the connection arrives masqueraded as the WG container.
|
||||
- **nuc**: `/etc/wireguard/wg-ks4.conf` (`wg-quick@wg-ks4` enabled;
|
||||
peer = container pubkey `TVs6d7…`, endpoint `193.70.35.17:51845`,
|
||||
`AllowedIPs = 10.8.0.0/24, 192.168.1.1/32`, keepalive 25s) and
|
||||
`incus remote add ks4 https://192.168.1.1:8443 --accept-certificate
|
||||
--token '…'` (fingerprint cross-checked against the token).
|
||||
|
||||
In pull mode all control + migration traffic flows over that single
|
||||
API connection, so nothing else needs routing. Copying the `wireguard`
|
||||
container over its own tunnel is fine (crash-consistent, tiny, no
|
||||
interruption); if the tunnel is down the cron job fails loudly instead
|
||||
of hanging.
|
||||
|
||||
Then cron (root on nuc) — stagger after ks4's local leg:
|
||||
|
||||
```cron
|
||||
30 2 * * * /root/scripts/incus-copy.sh -r ks4 -s ks4backup >> /var/log/incus-copy.log 2>&1
|
||||
```
|
||||
|
||||
The **initial seed is ~1.75 TiB** over the home downlink — run the
|
||||
first `incus-copy.sh -r ks4 -s ks4backup` manually (tmux), enable the
|
||||
cron once it completes.
|
||||
|
||||
## Cutover checklist (then kill ks2)
|
||||
|
||||
1. First full cycle of all three jobs clean (logs above).
|
||||
2. Restore test: on nuc, start a small replica (e.g. `freshrss`) with
|
||||
its NIC detached, check app data, then stop it.
|
||||
3. Remove both ks2 cron lines on ks4, `incus remote remove ks2`,
|
||||
cancel the server (`164.132.173.57` = ks2, rsync target of the old
|
||||
4 AM job).
|
||||
|
||||
## Restore
|
||||
|
||||
```sh
|
||||
# from nuc (off-site replica):
|
||||
incus copy <instance> ks4:<instance> --mode push
|
||||
# from the local backup project (sda replaced, pool data rebuilt):
|
||||
incus copy <instance> <instance> --project backup --target-project default -s data
|
||||
```
|
||||
|
||||
Remember replicas have `boot.autostart=false`; re-enable after a real
|
||||
failover, and re-check it after copying back to ks4.
|
||||
Reference in New Issue
Block a user