doc/ks4/install.md

ks4 — prod server setup & rebuild procedure

Prod Incus host at OVH (ks4.lutran.fr → 193.70.35.17, SSH on port 2233). This is a scan-derived snapshot of how the box is currently laid out plus the steps to rebuild the host layer. Per-instance rebuild docs are still TODO — see Instances.

Scanned 2026-08-09 as julien (in groups sudo, incus) with passwordless sudo incus … only. Anything needing root beyond incus (zpool status, host firewall dumps, container internals) is marked verify below.

Hardware / hosting

• OVH dedicated server, hostname ns3061243, up ~90 days at scan time
• CPU: Intel Xeon D-1521 @ 2.40 GHz (4c/8t)
• RAM: 31 GiB (+1 GiB swap)
• Disks: 2× 6 TB HGST HUS726T6TALE6L1 (sda, sdb)
  • sda1/2/3 + sdb1/2/3 → mdraid RAID1 mirrors: md1 → /boot/efi (vfat), md2 → /boot (ext4), md3 → / (ext4, 40 GB)
  • sda4 / sdb4 → swap (512 MB each)
  • sda5 (5.4 TB) → ZFS pool data (all Incus instance disks), single vdev ata-HGST_…_V9K2UX7L-part5, ONLINE, weekly scrub (Sundays, last clean run 2026-08-09)
  • sdb5 (5.4 TB) → unused (no fstype)
• Network: eno3 up, 193.70.35.17/24, gw 193.70.35.254, IPv6 2001:41d0:303:711::1/128. eno4 down.
• OS: Debian 13 (trixie), kernel 6.12.86+deb13-amd64

⚠️ Data pool is NOT mirrored

Confirmed via zpool status: the OS (md1/2/3) is RAID1 across both disks, but the ZFS data pool is a single vdev on sda5. If sda dies, every instance disk is lost even though the OS survives on sdb. Decision (2026-08-09): instead of mirroring, sdb5 hosts a dedicated backup zpool receiving nightly replicas + dumps, with a second off-site replica on nuc — see incus-copy.md.

(zpool status also notes some pool features are disabled — an optional zpool upgrade data would enable them, at the cost of compatibility with older ZFS.)

Software layer

• Incus from the Zabbly stable repo (pkgs.zabbly.com/incus/stable, suite trixie), packages incus, incus-base, incus-client, incus-ui-canonical, version 1:7.2-debian13-….
• ZFS 2.4.3-1~bpo13+1 (zfs-dkms, zfsutils-linux, zfs-zed).
• Incus daemon: core.https_address: :8443 (web UI reachable; TLS auth), firewall driver nftables, storage driver zfs, not clustered.

Repo + install (matches nuc):

mkdir -p /etc/apt/keyrings
curl -fsSL https://pkgs.zabbly.com/key.asc -o /etc/apt/keyrings/zabbly.asc
cat > /etc/apt/sources.list.d/zabbly-incus-stable.sources <<EOF
Enabled: yes
Types: deb
URIs: https://pkgs.zabbly.com/incus/stable
Suites: trixie
Components: main
Architectures: amd64
Signed-By: /etc/apt/keyrings/zabbly.asc
EOF
apt update && apt install -y incus zfs-dkms zfsutils-linux zfs-zed

Networking / public ingress

Unlike nuc (macvlan → real LAN IPs), ks4 uses a NAT bridge: all instances sit on incusbr0 (192.168.1.1/24) with static 192.168.1.x addresses. Public traffic reaches them through Incus proxy devices on the host's public IP:

Host port(s)	→ Instance	Proxy notes
80, 443/tcp	gateway (192.168.1.2)	proxy_protocol=true → nginx reverse proxy fans out to the app containers
25, 587, 993/tcp	mail (192.168.1.3)	proxy_protocol=true (SMTP/submission/IMAPS)
2244/tcp	git (192.168.1.6)	→ container :22 (git-over-SSH)
51845/udp	wireguard (192.168.1.18)	→ WireGuard; VPN subnet 10.8.0.0/24 (server wg0 10.8.0.1)
2233/tcp	host sshd	admin SSH (not a proxy device)
8443/tcp	host incus daemon	remote API + web UI

So gateway is the single HTTP/S entry point and terminates TLS for the web apps; the app containers are not directly exposed. The gateway and mail proxies use the PROXY protocol, so nginx/postfix inside must be configured to expect it.

Other bridges present: incusbr-1001 (10.183.142.1/24) and incusbr-1002 (10.10.112.1/24), one per restricted user project (see below). The host also runs a DNS resolver (dnsmasq) on each bridge IP :53.

Incus projects

Project	Purpose
default	all the service containers below
user-1001	restricted project for julien (own bridge + quota)
user-1002	restricted project for franzz
incus-compose-images	image cache for the incus-compose tool (see note)

~/incus-compose on the host is a Go project (own GitLab CI) — a docker-compose-style deployer for Incus that julien develops. The incus-compose-images project is its image store. Not required to run the existing containers.

Instances

All in the default project, on the default profile (eth0 → incusbr0, root → pool data), unprivileged (security.nesting=true), with daily snapshots (snapshots.schedule=@daily, expiry 1w / 4w manual, ~7 kept).

Running:

Name	IP	Role (inferred)	Base image	Notes
gateway	.2	nginx reverse proxy / TLS — HTTP(S) ingress	Ubuntu 18.04	proxy 80/443, PROXY protocol
mail	.3	mail server (Postfix/Dovecot stack)	Debian 9.9 (Apache/MariaDB/PHP)	proxy 25/587/993; ~20 GB
nextcloud	.4	Nextcloud	Debian stretch	~726 GB data
freshrss	.5	FreshRSS reader	—	dedicated root disk device
git	.6	Git server (Gitea/GitLab-class)	Ubuntu 18.04	SSH proxy host:2244→:22
seafile	.7	Seafile file sync	—	~950 GB, dedicated root disk
databap	.8	app (role unconfirmed — verify)	Ubuntu 18.04
spot	.9	app (role unconfirmed — verify; joal/torrent?)	Debian 9.9	~25 GB
bitwarden	.15	Bitwarden / Vaultwarden	Ubuntu 18.04	~23 GB
solar	.17	app (solar/PV monitoring? — verify)	Debian 9.9
wireguard	.18	WireGuard VPN	Debian trixie (rebuilt 2025-11-28)	proxy udp:51845; wg0 10.8.0.1/24
login	.19	SSO / auth (runs Docker inside)	Debian 9.9	docker0+bridge; nesting, syscall intercepts
outline	.20	Outline wiki (runs Docker inside)	Debian 9.9	docker0+bridge; nesting, syscall intercepts

Stopped (legacy / occasional): catc, mythoughts, qcm, wedding, wow — no snapshots, kept around; roles not documented.

Most base images are old (Ubuntu 18.04 / Debian 9 "stretch", both EOL). Treat these as pets to migrate, not reproduce. Per-instance install docs (one .md each, per repo convention) still need to be written — that requires shelling into each container, which the current incus-only sudo grant doesn't allow.

Backups

Full picture, schedule and restore procedures: ../backup-strategy.md. Drivers live in the scripts repo and are deployed to /root/scripts. Current root cron jobs — the safety net for the non-mirrored data pool:

0 1 * * * /root/scripts/incus-copy.sh -p backup -s backup >> /var/log/incus-copy.log 2>&1
0 5 * * * /root/scripts/restic-backup.sh >> /var/log/restic-backup.log 2>&1
0 14 * * 0 /root/scripts/restic-maintenance.sh >> /var/log/restic-maintenance.log 2>&1

• 01:00 incus-copy.sh → replicas of every instance in project backup on the sdb5 pool (incus-copy.md, local-backup-cron.md); the S3 instance leg chains here once seeded.
• 05:00 restic-backup.sh → auto-discovered DB dumps + the data trees listed in scripts/restic-paths, into the restic-data bucket (restic-backup.md).
• Sun 14:00 restic-maintenance.sh → prune + integrity checks.

A ks2 interim replica push (incus-copy.sh -d ks2 -m push, 02:00) runs until the nuc pull leg is seeded. Retired 2026-08: the incus-backup.sh rsync leg and the first S3 implementation (plakar-s3-data.md — reference only).

• 01:00 — incus-copy.sh → ks2: differential incus copy … \ --refresh --refresh-exclude-older --mode push of every running instance to remote incus host ks2 (a near-live replica). ks2 must exist as an incus remote.
• 04:00 — incus-backup.sh → 164.132.173.57: rsync-over-ssh (user rsyncbackup, key /root/.ssh/id_rsyncbackup, port 2233) of the incus local+global DB dumps, each container's backup.yaml, MariaDB dumps (incus exec … mariadb-dump), and selected rootfs paths into /backup/ks4/ on the remote. Which containers/DBs/paths are backed up is driven by the JSON manifest /root/scripts/incus-backup.db — keep it current when adding a service.

Both scripts run as root (they use incus exec / incus admin sql), not through the julien incus-only sudo grant. Logs: /var/log/incus-copy.log, /var/log/incus-backup.log.

Manual export of a single instance:

sudo incus export <name> /path/to/<name>.tar.gz   # --instance-only to skip snapshots

Host rebuild outline

OVH reinstalls provide Debian; the ks4-specific layer is:

1. OS: Debian 13, mdraid RAID1 (md1/md2/md3) across both disks for /boot/efi, /boot, / — OVH's guided partitioning, or restore the layout above. Leave sda5 (and ideally sdb5) for ZFS.
2. SSH: move sshd to port 2233; restore /root/.ssh/authorized_keys (incl. id_rsa_claude.pub from this repo) and the julien account (groups sudo, incus) + /etc/sudoers.d/incus.
3. ZFS pool:

   zpool create data /dev/disk/by-id/<sda5-id>
   # recommended: mirror it (see warning above)
   zpool attach data <sda5-id> /dev/disk/by-id/<sdb5-id>
   

4. Incus: install from Zabbly (above), then incus admin init with pool data and bridge incusbr0 (192.168.1.1/24, NAT):

   cat <<EOF | incus admin init --preseed
   config:
     core.https_address: :8443
   storage_pools:
   - name: data
     driver: zfs
     config: {source: data}
   networks:
   - name: incusbr0
     type: bridge
     config: {ipv4.address: 192.168.1.1/24, ipv4.nat: "true", ipv6.address: none}
   profiles:
   - name: default
     devices:
       eth0: {name: eth0, network: incusbr0, type: nic}
       root: {path: /, pool: data, type: disk}
   EOF
   

5. Restore instances from exports, re-pin each static IP (incus config device set <name> eth0 ipv4.address 192.168.1.x), and re-add the proxy devices from the ingress table, e.g.:

   incus config device add gateway https proxy \
     listen=tcp:0.0.0.0:443 connect=tcp:127.0.0.1:443 proxy_protocol=true
   

6. Re-enable daily snapshots on each instance, restore root's crontab + /root/scripts/ + /root/.ssh/id_rsyncbackup, add the ks2 incus remote, and confirm both backup jobs run (see Backups).