# ks4 — prod server setup & rebuild procedure

Prod Incus host at OVH (`ks4.lutran.fr` → `193.70.35.17`, SSH on port
**2233**). This is a scan-derived snapshot of how the box is currently
laid out plus the steps to rebuild the host layer. Per-instance rebuild
docs are still TODO — see [Instances](#instances).

> Scanned 2026-08-09 as `julien` (in groups `sudo`, `incus`) with
> passwordless `sudo incus …` only. Anything needing root beyond incus
> (`zpool status`, host firewall dumps, container internals) is marked
> **verify** below.

## Hardware / hosting

- OVH dedicated server, hostname `ns3061243`, up ~90 days at scan time
- CPU: Intel Xeon D-1521 @ 2.40 GHz (4c/8t)
- RAM: 31 GiB (+1 GiB swap)
- Disks: 2× 6 TB HGST `HUS726T6TALE6L1` (`sda`, `sdb`)
  - `sda1/2/3` + `sdb1/2/3` → **mdraid RAID1** mirrors:
    `md1` → `/boot/efi` (vfat), `md2` → `/boot` (ext4),
    `md3` → `/` (ext4, 40 GB)
  - `sda4` / `sdb4` → swap (512 MB each)
  - `sda5` (5.4 TB) → ZFS pool `data` (all Incus instance disks),
    single vdev `ata-HGST_…_V9K2UX7L-part5`, `ONLINE`, weekly scrub
    (Sundays, last clean run 2026-08-09)
  - `sdb5` (5.4 TB) → **unused** (no fstype)
- Network: `eno3` up, `193.70.35.17/24`, gw `193.70.35.254`,
  IPv6 `2001:41d0:303:711::1/128`. `eno4` down.
- OS: Debian 13 (trixie), kernel `6.12.86+deb13-amd64`

### ⚠️ Data pool is NOT mirrored

Confirmed via `zpool status`: the OS (`md1/2/3`) is RAID1 across both
disks, but the ZFS `data` pool is a **single vdev on `sda5`**. If `sda`
dies, every instance disk is lost even though the OS survives on `sdb`.
Decision (2026-08-09): instead of mirroring, `sdb5` hosts a dedicated
`backup` zpool receiving nightly replicas + dumps, with a second
off-site replica on nuc — see [incus-copy.md](incus-copy.md).

(`zpool status` also notes some pool features are disabled — an
optional `zpool upgrade data` would enable them, at the cost of
compatibility with older ZFS.)

## Software layer

- **Incus** from the Zabbly stable repo (`pkgs.zabbly.com/incus/stable`,
  suite `trixie`), packages `incus`, `incus-base`, `incus-client`,
  `incus-ui-canonical`, version `1:7.2-debian13-…`.
- ZFS `2.4.3-1~bpo13+1` (`zfs-dkms`, `zfsutils-linux`, `zfs-zed`).
- Incus daemon: `core.https_address: :8443` (web UI reachable; TLS auth),
  firewall driver **nftables**, storage driver **zfs**, not clustered.

Repo + install (matches nuc):
```sh
mkdir -p /etc/apt/keyrings
curl -fsSL https://pkgs.zabbly.com/key.asc -o /etc/apt/keyrings/zabbly.asc
cat > /etc/apt/sources.list.d/zabbly-incus-stable.sources <<EOF
Enabled: yes
Types: deb
URIs: https://pkgs.zabbly.com/incus/stable
Suites: trixie
Components: main
Architectures: amd64
Signed-By: /etc/apt/keyrings/zabbly.asc
EOF
apt update && apt install -y incus zfs-dkms zfsutils-linux zfs-zed
```

## Networking / public ingress

Unlike nuc (macvlan → real LAN IPs), ks4 uses a **NAT bridge**: all
instances sit on `incusbr0` (`192.168.1.1/24`) with static
`192.168.1.x` addresses. Public traffic reaches them through Incus
**proxy devices** on the host's public IP:

| Host port(s) | → Instance | Proxy notes |
|---|---|---|
| 80, 443/tcp | `gateway` (192.168.1.2) | `proxy_protocol=true` → nginx reverse proxy fans out to the app containers |
| 25, 587, 993/tcp | `mail` (192.168.1.3) | `proxy_protocol=true` (SMTP/submission/IMAPS) |
| 2244/tcp | `git` (192.168.1.6) | → container `:22` (git-over-SSH) |
| 51845/udp | `wireguard` (192.168.1.18) | → WireGuard; VPN subnet `10.8.0.0/24` (server `wg0` 10.8.0.1) |
| 2233/tcp | host `sshd` | admin SSH (not a proxy device) |
| 8443/tcp | host `incus` daemon | remote API + web UI |

So **`gateway` is the single HTTP/S entry point** and terminates TLS for
the web apps; the app containers are not directly exposed. The `gateway`
and `mail` proxies use the PROXY protocol, so nginx/postfix inside must
be configured to expect it.

Other bridges present: `incusbr-1001` (`10.183.142.1/24`) and
`incusbr-1002` (`10.10.112.1/24`), one per restricted user project
(see below). The host also runs a DNS resolver (dnsmasq) on each bridge
IP `:53`.

## Incus projects

| Project | Purpose |
|---|---|
| `default` | all the service containers below |
| `user-1001` | restricted project for `julien` (own bridge + quota) |
| `user-1002` | restricted project for `franzz` |
| `incus-compose-images` | image cache for the `incus-compose` tool (see note) |

> `~/incus-compose` on the host is a Go project (own GitLab CI) — a
> docker-compose-style deployer for Incus that julien develops. The
> `incus-compose-images` project is its image store. Not required to run
> the existing containers.

## Instances

All in the `default` project, on the `default` profile
(`eth0` → `incusbr0`, `root` → pool `data`), unprivileged
(`security.nesting=true`), with daily snapshots
(`snapshots.schedule=@daily`, expiry 1w / 4w manual, ~7 kept).

**Running:**

| Name | IP | Role (inferred) | Base image | Notes |
|---|---|---|---|---|
| gateway | .2 | **nginx reverse proxy / TLS** — HTTP(S) ingress | Ubuntu 18.04 | proxy 80/443, PROXY protocol |
| mail | .3 | **mail server** (Postfix/Dovecot stack) | Debian 9.9 (Apache/MariaDB/PHP) | proxy 25/587/993; ~20 GB |
| nextcloud | .4 | **Nextcloud** | Debian stretch | ~726 GB data |
| freshrss | .5 | **FreshRSS** reader | — | dedicated `root` disk device |
| git | .6 | **Git server** (Gitea/GitLab-class) | Ubuntu 18.04 | SSH proxy host:2244→:22 |
| seafile | .7 | **Seafile** file sync | — | ~950 GB, dedicated `root` disk |
| databap | .8 | app (role unconfirmed — **verify**) | Ubuntu 18.04 | |
| spot | .9 | app (role unconfirmed — **verify**; joal/torrent?) | Debian 9.9 | ~25 GB |
| bitwarden | .15 | **Bitwarden / Vaultwarden** | Ubuntu 18.04 | ~23 GB |
| solar | .17 | app (solar/PV monitoring? — **verify**) | Debian 9.9 | |
| wireguard | .18 | **WireGuard VPN** | Debian trixie (rebuilt 2025-11-28) | proxy udp:51845; wg0 10.8.0.1/24 |
| login | .19 | **SSO / auth** (runs Docker inside) | Debian 9.9 | `docker0`+bridge; nesting, syscall intercepts |
| outline | .20 | **Outline** wiki (runs Docker inside) | Debian 9.9 | `docker0`+bridge; nesting, syscall intercepts |

**Stopped (legacy / occasional):** `catc`, `mythoughts`, `qcm`,
`wedding`, `wow` — no snapshots, kept around; roles not documented.

> Most base images are old (Ubuntu 18.04 / Debian 9 "stretch", both EOL).
> Treat these as pets to migrate, not reproduce. Per-instance install
> docs (one `.md` each, per repo convention) still need to be written —
> that requires shelling into each container, which the current
> incus-only sudo grant doesn't allow.

## Backups

Full picture, schedule and restore procedures:
**[../backup-strategy.md](../backup-strategy.md)**. Drivers live in the
[scripts repo](https://git.lutran.fr/julien/scripts) and are deployed
to `/root/scripts`. Current **root** cron jobs — the safety net for the
non-mirrored `data` pool:

```cron
0 1 * * * /root/scripts/incus-copy.sh -p backup -s backup >> /var/log/incus-copy.log 2>&1
0 5 * * * /root/scripts/restic-backup.sh >> /var/log/restic-backup.log 2>&1
0 14 * * 0 /root/scripts/restic-maintenance.sh >> /var/log/restic-maintenance.log 2>&1
```

- **01:00 `incus-copy.sh`** → replicas of every instance in project
  `backup` on the sdb5 pool ([incus-copy.md](incus-copy.md),
  [local-backup-cron.md](local-backup-cron.md)); the S3 instance leg
  chains here once seeded.
- **05:00 `restic-backup.sh`** → auto-discovered DB dumps + the data
  trees listed in
  [`scripts/restic-paths`](https://git.lutran.fr/julien/scripts/src/branch/main/restic-paths),
  into the `restic-data` bucket ([restic-backup.md](restic-backup.md)).
- **Sun 14:00 `restic-maintenance.sh`** → prune + integrity checks.

A `ks2` interim replica push (`incus-copy.sh -d ks2 -m push`, 02:00)
runs until the nuc pull leg is seeded. Retired 2026-08: the
`incus-backup.sh` rsync leg and the first S3 implementation
([plakar-s3-data.md](plakar-s3-data.md) — reference only).

- **01:00 — `incus-copy.sh` → `ks2`**: differential `incus copy … \
  --refresh --refresh-exclude-older --mode push` of every *running*
  instance to remote incus host `ks2` (a near-live replica). `ks2` must
  exist as an `incus remote`.
- **04:00 — `incus-backup.sh` → `164.132.173.57`**: rsync-over-ssh (user
  `rsyncbackup`, key `/root/.ssh/id_rsyncbackup`, port 2233) of the incus
  local+global DB dumps, each container's `backup.yaml`, MariaDB dumps
  (`incus exec … mariadb-dump`), and selected rootfs paths into
  `/backup/ks4/` on the remote. **Which** containers/DBs/paths are
  backed up is driven by the JSON manifest
  `/root/scripts/incus-backup.db` — keep it current when adding a
  service.

Both scripts run as root (they use `incus exec` / `incus admin sql`),
not through the `julien` incus-only sudo grant. Logs:
`/var/log/incus-copy.log`, `/var/log/incus-backup.log`.

Manual export of a single instance:
```sh
sudo incus export <name> /path/to/<name>.tar.gz   # --instance-only to skip snapshots
```

## Host rebuild outline

OVH reinstalls provide Debian; the ks4-specific layer is:

1. **OS**: Debian 13, mdraid RAID1 (`md1`/`md2`/`md3`) across both disks
   for `/boot/efi`, `/boot`, `/` — OVH's guided partitioning, or restore
   the layout above. Leave `sda5` (and ideally `sdb5`) for ZFS.
2. **SSH**: move sshd to port **2233**; restore
   `/root/.ssh/authorized_keys` (incl. `id_rsa_claude.pub` from this
   repo) and the `julien` account (groups `sudo`, `incus`) +
   `/etc/sudoers.d/incus`.
3. **ZFS pool**:
   ```sh
   zpool create data /dev/disk/by-id/<sda5-id>
   # recommended: mirror it (see warning above)
   zpool attach data <sda5-id> /dev/disk/by-id/<sdb5-id>
   ```
4. **Incus**: install from Zabbly (above), then `incus admin init`
   with pool `data` and bridge `incusbr0` (`192.168.1.1/24`, NAT):
   ```sh
   cat <<EOF | incus admin init --preseed
   config:
     core.https_address: :8443
   storage_pools:
   - name: data
     driver: zfs
     config: {source: data}
   networks:
   - name: incusbr0
     type: bridge
     config: {ipv4.address: 192.168.1.1/24, ipv4.nat: "true", ipv6.address: none}
   profiles:
   - name: default
     devices:
       eth0: {name: eth0, network: incusbr0, type: nic}
       root: {path: /, pool: data, type: disk}
   EOF
   ```
5. **Restore instances** from exports, re-pin each static IP
   (`incus config device set <name> eth0 ipv4.address 192.168.1.x`),
   and re-add the proxy devices from the
   [ingress table](#networking--public-ingress), e.g.:
   ```sh
   incus config device add gateway https proxy \
     listen=tcp:0.0.0.0:443 connect=tcp:127.0.0.1:443 proxy_protocol=true
   ```
6. **Re-enable** daily snapshots on each instance, restore root's
   crontab + `/root/scripts/` + `/root/.ssh/id_rsyncbackup`, add the
   `ks2` incus remote, and confirm both backup jobs run (see
   [Backups](#backups)).