Initial import: nuc and ks4 infrastructure documentation

README with instance tables and nuc<->ks4 network flow chart; per-container install/troubleshooting docs for nuc (jellyfin server/client, transmission-bt, bare-metal reinstall) and the ks4 two-leg backup scheme (incus-copy over wireguard). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 22:50:07 +02:00
commit ee9eaee889
7 changed files with 1255 additions and 0 deletions
@@ -0,0 +1,147 @@
+# incus-copy — ks4 instance replication
+
+Incremental replication of all ks4 Incus instances via
+[`scripts/incus-copy.sh`](../scripts/incus-copy.sh)
+(`incus copy --refresh`: ZFS-incremental, produces ready-to-start
+replicas instead of tarballs). Two legs replace the historical nightly
+push to `ks2` (decommissioning):
+
+1. **local** — replicas + dumps on a dedicated `backup` zpool on ks4's
+   second disk (`sdb5`), survives `sda` death
+2. **off-site** — replicas pulled by **nuc** into pool `ks4backup`
+   (dataset `usb4t/backup/ks4`), survives losing ks4 entirely
+
+## The script
+
+```
+incus-copy.sh -d <dest-remote> [-m pull|push|relay] [-s <dest-pool>]   # push local → remote
+incus-copy.sh -r <src-remote>  [-m pull|push|relay] [-s <dest-pool>]   # pull remote → local
+incus-copy.sh -p <dest-project> -s <dest-pool>                         # local → local project
+```
+
+- Copies **all** instances, running and stopped — a stopped instance's
+  filesystem never changes, so its refresh transfers nothing after the
+  first copy.
+- `flock` on `/run/lock/incus-copy.lock` — overlapping cron runs abort
+  instead of racing the same instance.
+- After each successful copy it sets `boot.autostart=false` on the
+  **replica** — otherwise the copy inherits autostart and would start
+  itself on the backup host after a reboot (or, for the local-project
+  leg, fight the live instance for its static IP on the same bridge).
+- Exits non-zero if any instance failed (cron/monitoring can alert);
+  per-instance failures go to stderr, progress to stdout.
+- Snapshot history is the **source's** job — set on ks4, e.g.:
+  `incus profile set default snapshots.schedule="0 3 * * *" snapshots.expiry=7d`.
+  `--refresh-exclude-older` keeps expired snapshots from being re-sent.
+- Consistency: copies of running instances are crash-consistent;
+  databases may need recovery on restore. Snapshot history mitigates,
+  and [`incus-backup.sh`](../scripts/incus-backup.sh) (proper
+  `mariadb-dump`s + selected paths, manifest-driven) covers the DBs.
+- **VMs**: `--refresh` re-sends a VM's **full block volume** every run
+  unless source and replica share a common snapshot — containers diff
+  cheaply regardless (verified on nuc: homeassistant re-sent 3.3 GiB,
+  containers only KBs). Set `snapshots.schedule` on sources *before*
+  enabling the crons, especially for the WAN leg.
+
+## Leg 1 — local backup pool on ks4 (sdb5)
+
+One-time setup (root on ks4). `sdb1-4` mirror the OS (mdraid); `sdb5`
+was unused:
+
+```sh
+# find sdb's serial: ls -l /dev/disk/by-id/ | grep sdb5
+zpool create -m none backup /dev/disk/by-id/ata-HGST_HUS726T6TALE6L1_<sdb-serial>-part5
+zfs create backup/incus                        # incus-managed (replicas)
+zfs create -o mountpoint=/backup backup/dumps  # incus-backup.sh output
+incus storage create backup zfs source=backup/incus
+# replicas live in their own project so names don't collide with the
+# live instances; it shares the default project's profiles
+incus project create backup -c features.images=false -c features.profiles=false
+```
+
+Cron (root on ks4) — replaces both ks2 jobs:
+
+```cron
+0 1 * * * /root/scripts/incus-copy.sh -p backup -s backup >> /var/log/incus-copy.log 2>&1
+0 4 * * * /root/scripts/incus-backup.sh -f /root/scripts/incus-backup.db -s data >> /var/log/incus-backup.log 2>&1
+```
+
+(`incus-backup.sh` without `-d` now writes locally to
+`/backup/$(hostname -s)/` — the `backup/dumps` dataset.)
+
+⚠️ Replicas in the `backup` project must stay **stopped** — they keep
+the live containers' static `192.168.1.x` addresses.
+
+## Leg 2 — off-site pull from nuc
+
+Storage pool on nuc (done 2026-08-09): `ks4backup`, backed by the
+dataset `usb4t/backup/ks4` on the USB 4 TB pool (quota on `usb4t/backup`
+removed 2026-08-09 — full replica set is ~1.75 TiB):
+
+```sh
+incus storage create ks4backup zfs source=usb4t/backup/ks4
+```
+
+Replicas live only on the USB drive — if it fails, only backups are
+lost; nuc's own instances (pool `data` on the SSD) are unaffected.
+
+**Direction: nuc pulls, through the WireGuard tunnel.** Verified
+2026-08-09: ks4's API listens on wildcard `:8443` (so it answers on
+`192.168.1.1`, the `incusbr0` host address) but is **firewalled from
+the internet** — the VPN path keeps it that way, needs no inbound port
+at home, and doesn't care that nuc's public IP is dynamic. The
+`wireguard` container (`192.168.1.18`, `wg0` `10.8.0.1/24`) is exposed
+via a proxy device on public UDP `51845`.
+
+Setup (✅ **done 2026-08-09**, verified end-to-end with
+`incus list ks4:` from nuc):
+
+- **wireguard container** (ks4): forwards + masquerades wg0→eth0
+  (pre-existing); nuc added as peer `10.8.0.20/32`
+  (`wg set wg0 peer 31Tlgloc… allowed-ips 10.8.0.20/32` +
+  `wg-quick save wg0`).
+- **ufw** (ks4): `ufw allow in on incusbr0 from 192.168.1.18 to any
+  port 8443 proto tcp` — the API stays firewalled from the internet
+  and the connection arrives masqueraded as the WG container.
+- **nuc**: `/etc/wireguard/wg-ks4.conf` (`wg-quick@wg-ks4` enabled;
+  peer = container pubkey `TVs6d7…`, endpoint `193.70.35.17:51845`,
+  `AllowedIPs = 10.8.0.0/24, 192.168.1.1/32`, keepalive 25s) and
+  `incus remote add ks4 https://192.168.1.1:8443 --accept-certificate
+  --token '…'` (fingerprint cross-checked against the token).
+
+In pull mode all control + migration traffic flows over that single
+API connection, so nothing else needs routing. Copying the `wireguard`
+container over its own tunnel is fine (crash-consistent, tiny, no
+interruption); if the tunnel is down the cron job fails loudly instead
+of hanging.
+
+Then cron (root on nuc) — stagger after ks4's local leg:
+
+```cron
+30 2 * * * /root/scripts/incus-copy.sh -r ks4 -s ks4backup >> /var/log/incus-copy.log 2>&1
+```
+
+The **initial seed is ~1.75 TiB** over the home downlink — run the
+first `incus-copy.sh -r ks4 -s ks4backup` manually (tmux), enable the
+cron once it completes.
+
+## Cutover checklist (then kill ks2)
+
+1. First full cycle of all three jobs clean (logs above).
+2. Restore test: on nuc, start a small replica (e.g. `freshrss`) with
+   its NIC detached, check app data, then stop it.
+3. Remove both ks2 cron lines on ks4, `incus remote remove ks2`,
+   cancel the server (`164.132.173.57` = ks2, rsync target of the old
+   4 AM job).
+
+## Restore
+
+```sh
+# from nuc (off-site replica):
+incus copy <instance> ks4:<instance> --mode push
+# from the local backup project (sda replaced, pool data rebuilt):
+incus copy <instance> <instance> --project backup --target-project default -s data
+```
+
+Remember replicas have `boot.autostart=false`; re-enable after a real
+failover, and re-check it after copying back to ks4.
@@ -0,0 +1,232 @@
+# ks4 — prod server setup & rebuild procedure
+
+Prod Incus host at OVH (`ks4.lutran.fr` → `193.70.35.17`, SSH on port
+**2233**). This is a scan-derived snapshot of how the box is currently
+laid out plus the steps to rebuild the host layer. Per-instance rebuild
+docs are still TODO — see [Instances](#instances).
+
+> Scanned 2026-08-09 as `julien` (in groups `sudo`, `incus`) with
+> passwordless `sudo incus …` only. Anything needing root beyond incus
+> (`zpool status`, host firewall dumps, container internals) is marked
+> **verify** below.
+
+## Hardware / hosting
+
+- OVH dedicated server, hostname `ns3061243`, up ~90 days at scan time
+- CPU: Intel Xeon D-1521 @ 2.40 GHz (4c/8t)
+- RAM: 31 GiB (+1 GiB swap)
+- Disks: 2× 6 TB HGST `HUS726T6TALE6L1` (`sda`, `sdb`)
+  - `sda1/2/3` + `sdb1/2/3` → **mdraid RAID1** mirrors:
+    `md1` → `/boot/efi` (vfat), `md2` → `/boot` (ext4),
+    `md3` → `/` (ext4, 40 GB)
+  - `sda4` / `sdb4` → swap (512 MB each)
+  - `sda5` (5.4 TB) → ZFS pool `data` (all Incus instance disks),
+    single vdev `ata-HGST_…_V9K2UX7L-part5`, `ONLINE`, weekly scrub
+    (Sundays, last clean run 2026-08-09)
+  - `sdb5` (5.4 TB) → **unused** (no fstype)
+- Network: `eno3` up, `193.70.35.17/24`, gw `193.70.35.254`,
+  IPv6 `2001:41d0:303:711::1/128`. `eno4` down.
+- OS: Debian 13 (trixie), kernel `6.12.86+deb13-amd64`
+
+### ⚠️ Data pool is NOT mirrored
+
+Confirmed via `zpool status`: the OS (`md1/2/3`) is RAID1 across both
+disks, but the ZFS `data` pool is a **single vdev on `sda5`**. If `sda`
+dies, every instance disk is lost even though the OS survives on `sdb`.
+Decision (2026-08-09): instead of mirroring, `sdb5` hosts a dedicated
+`backup` zpool receiving nightly replicas + dumps, with a second
+off-site replica on nuc — see [incus-copy.md](incus-copy.md).
+
+(`zpool status` also notes some pool features are disabled — an
+optional `zpool upgrade data` would enable them, at the cost of
+compatibility with older ZFS.)
+
+## Software layer
+
+- **Incus** from the Zabbly stable repo (`pkgs.zabbly.com/incus/stable`,
+  suite `trixie`), packages `incus`, `incus-base`, `incus-client`,
+  `incus-ui-canonical`, version `1:7.2-debian13-…`.
+- ZFS `2.4.3-1~bpo13+1` (`zfs-dkms`, `zfsutils-linux`, `zfs-zed`).
+- Incus daemon: `core.https_address: :8443` (web UI reachable; TLS auth),
+  firewall driver **nftables**, storage driver **zfs**, not clustered.
+
+Repo + install (matches nuc):
+```sh
+mkdir -p /etc/apt/keyrings
+curl -fsSL https://pkgs.zabbly.com/key.asc -o /etc/apt/keyrings/zabbly.asc
+cat > /etc/apt/sources.list.d/zabbly-incus-stable.sources <<EOF
+Enabled: yes
+Types: deb
+URIs: https://pkgs.zabbly.com/incus/stable
+Suites: trixie
+Components: main
+Architectures: amd64
+Signed-By: /etc/apt/keyrings/zabbly.asc
+EOF
+apt update && apt install -y incus zfs-dkms zfsutils-linux zfs-zed
+```
+
+## Networking / public ingress
+
+Unlike nuc (macvlan → real LAN IPs), ks4 uses a **NAT bridge**: all
+instances sit on `incusbr0` (`192.168.1.1/24`) with static
+`192.168.1.x` addresses. Public traffic reaches them through Incus
+**proxy devices** on the host's public IP:
+
+| Host port(s) | → Instance | Proxy notes |
+|---|---|---|
+| 80, 443/tcp | `gateway` (192.168.1.2) | `proxy_protocol=true` → nginx reverse proxy fans out to the app containers |
+| 25, 587, 993/tcp | `mail` (192.168.1.3) | `proxy_protocol=true` (SMTP/submission/IMAPS) |
+| 2244/tcp | `git` (192.168.1.6) | → container `:22` (git-over-SSH) |
+| 51845/udp | `wireguard` (192.168.1.18) | → WireGuard; VPN subnet `10.8.0.0/24` (server `wg0` 10.8.0.1) |
+| 2233/tcp | host `sshd` | admin SSH (not a proxy device) |
+| 8443/tcp | host `incus` daemon | remote API + web UI |
+
+So **`gateway` is the single HTTP/S entry point** and terminates TLS for
+the web apps; the app containers are not directly exposed. The `gateway`
+and `mail` proxies use the PROXY protocol, so nginx/postfix inside must
+be configured to expect it.
+
+Other bridges present: `incusbr-1001` (`10.183.142.1/24`) and
+`incusbr-1002` (`10.10.112.1/24`), one per restricted user project
+(see below). The host also runs a DNS resolver (dnsmasq) on each bridge
+IP `:53`.
+
+## Incus projects
+
+| Project | Purpose |
+|---|---|
+| `default` | all the service containers below |
+| `user-1001` | restricted project for `julien` (own bridge + quota) |
+| `user-1002` | restricted project for `franzz` |
+| `incus-compose-images` | image cache for the `incus-compose` tool (see note) |
+
+> `~/incus-compose` on the host is a Go project (own GitLab CI) — a
+> docker-compose-style deployer for Incus that julien develops. The
+> `incus-compose-images` project is its image store. Not required to run
+> the existing containers.
+
+## Instances
+
+All in the `default` project, on the `default` profile
+(`eth0` → `incusbr0`, `root` → pool `data`), unprivileged
+(`security.nesting=true`), with daily snapshots
+(`snapshots.schedule=@daily`, expiry 1w / 4w manual, ~7 kept).
+
+**Running:**
+
+| Name | IP | Role (inferred) | Base image | Notes |
+|---|---|---|---|---|
+| gateway | .2 | **nginx reverse proxy / TLS** — HTTP(S) ingress | Ubuntu 18.04 | proxy 80/443, PROXY protocol |
+| mail | .3 | **mail server** (Postfix/Dovecot stack) | Debian 9.9 (Apache/MariaDB/PHP) | proxy 25/587/993; ~20 GB |
+| nextcloud | .4 | **Nextcloud** | Debian stretch | ~726 GB data |
+| freshrss | .5 | **FreshRSS** reader | — | dedicated `root` disk device |
+| git | .6 | **Git server** (Gitea/GitLab-class) | Ubuntu 18.04 | SSH proxy host:2244→:22 |
+| seafile | .7 | **Seafile** file sync | — | ~950 GB, dedicated `root` disk |
+| databap | .8 | app (role unconfirmed — **verify**) | Ubuntu 18.04 | |
+| spot | .9 | app (role unconfirmed — **verify**; joal/torrent?) | Debian 9.9 | ~25 GB |
+| bitwarden | .15 | **Bitwarden / Vaultwarden** | Ubuntu 18.04 | ~23 GB |
+| solar | .17 | app (solar/PV monitoring? — **verify**) | Debian 9.9 | |
+| wireguard | .18 | **WireGuard VPN** | Debian trixie (rebuilt 2025-11-28) | proxy udp:51845; wg0 10.8.0.1/24 |
+| login | .19 | **SSO / auth** (runs Docker inside) | Debian 9.9 | `docker0`+bridge; nesting, syscall intercepts |
+| outline | .20 | **Outline** wiki (runs Docker inside) | Debian 9.9 | `docker0`+bridge; nesting, syscall intercepts |
+
+**Stopped (legacy / occasional):** `catc`, `mythoughts`, `qcm`,
+`wedding`, `wow` — no snapshots, kept around; roles not documented.
+
+> Most base images are old (Ubuntu 18.04 / Debian 9 "stretch", both EOL).
+> Treat these as pets to migrate, not reproduce. Per-instance install
+> docs (one `.md` each, per repo convention) still need to be written —
+> that requires shelling into each container, which the current
+> incus-only sudo grant doesn't allow.
+
+## Backups
+
+> **Migration in progress** (2026-08-09): the `ks2` destination below is
+> being replaced by a local `backup` zpool on `sdb5` + off-site pull
+> from nuc — plan and setup in [incus-copy.md](incus-copy.md). The
+> cron blocks here describe what runs **today**; update on cutover.
+
+Two **root** cron jobs (this repo's [`scripts/`](../scripts/) are the
+copies that run here) — the primary safety net for the non-mirrored
+`data` pool:
+
+```cron
+0 1 * * * /root/scripts/incus-copy.sh   -d ks2 -m push
+0 4 * * * /root/scripts/incus-backup.sh -d 164.132.173.57 -u rsyncbackup \
+            -i /root/.ssh/id_rsyncbackup -p 2233 \
+            -f /root/scripts/incus-backup.db -s data
+```
+
+- **01:00 — `incus-copy.sh` → `ks2`**: differential `incus copy … \
+  --refresh --refresh-exclude-older --mode push` of every *running*
+  instance to remote incus host `ks2` (a near-live replica). `ks2` must
+  exist as an `incus remote`.
+- **04:00 — `incus-backup.sh` → `164.132.173.57`**: rsync-over-ssh (user
+  `rsyncbackup`, key `/root/.ssh/id_rsyncbackup`, port 2233) of the incus
+  local+global DB dumps, each container's `backup.yaml`, MariaDB dumps
+  (`incus exec … mariadb-dump`), and selected rootfs paths into
+  `/backup/ks4/` on the remote. **Which** containers/DBs/paths are
+  backed up is driven by the JSON manifest
+  `/root/scripts/incus-backup.db` — keep it current when adding a
+  service.
+
+Both scripts run as root (they use `incus exec` / `incus admin sql`),
+not through the `julien` incus-only sudo grant. Logs:
+`/var/log/incus-copy.log`, `/var/log/incus-backup.log`.
+
+Manual export of a single instance:
+```sh
+sudo incus export <name> /path/to/<name>.tar.gz   # --instance-only to skip snapshots
+```
+
+## Host rebuild outline
+
+OVH reinstalls provide Debian; the ks4-specific layer is:
+
+1. **OS**: Debian 13, mdraid RAID1 (`md1`/`md2`/`md3`) across both disks
+   for `/boot/efi`, `/boot`, `/` — OVH's guided partitioning, or restore
+   the layout above. Leave `sda5` (and ideally `sdb5`) for ZFS.
+2. **SSH**: move sshd to port **2233**; restore
+   `/root/.ssh/authorized_keys` (incl. `id_rsa_claude.pub` from this
+   repo) and the `julien` account (groups `sudo`, `incus`) +
+   `/etc/sudoers.d/incus`.
+3. **ZFS pool**:
+   ```sh
+   zpool create data /dev/disk/by-id/<sda5-id>
+   # recommended: mirror it (see warning above)
+   zpool attach data <sda5-id> /dev/disk/by-id/<sdb5-id>
+   ```
+4. **Incus**: install from Zabbly (above), then `incus admin init`
+   with pool `data` and bridge `incusbr0` (`192.168.1.1/24`, NAT):
+   ```sh
+   cat <<EOF | incus admin init --preseed
+   config:
+     core.https_address: :8443
+   storage_pools:
+   - name: data
+     driver: zfs
+     config: {source: data}
+   networks:
+   - name: incusbr0
+     type: bridge
+     config: {ipv4.address: 192.168.1.1/24, ipv4.nat: "true", ipv6.address: none}
+   profiles:
+   - name: default
+     devices:
+       eth0: {name: eth0, network: incusbr0, type: nic}
+       root: {path: /, pool: data, type: disk}
+   EOF
+   ```
+5. **Restore instances** from exports, re-pin each static IP
+   (`incus config device set <name> eth0 ipv4.address 192.168.1.x`),
+   and re-add the proxy devices from the
+   [ingress table](#networking--public-ingress), e.g.:
+   ```sh
+   incus config device add gateway https proxy \
+     listen=tcp:0.0.0.0:443 connect=tcp:127.0.0.1:443 proxy_protocol=true
+   ```
+6. **Re-enable** daily snapshots on each instance, restore root's
+   crontab + `/root/scripts/` + `/root/.ssh/id_rsyncbackup`, add the
+   `ks2` incus remote, and confirm both backup jobs run (see
+   [Backups](#backups)).