doc: add backup-strategy.md and new-container.md for non-specialist readers

backup-strategy.md: the two-tool architecture (incus copy + restic),
ASCII map of the legs (ks4 sdb / nuc pull / two S3 buckets), the
schedule table, a 30-second health check, and restore recipes for a
file, a database and a whole instance.

new-container.md: conventions for adding a service — one container
per service, no Docker (incus runs OCI natively, incus-compose for
stacks), gateway/proxy ingress, leave the snapshot schedule alone,
what the backups pick up automatically vs the one line to add to
restic-paths, DB-discovery requirements (incl. the .my.cnf trap), a
verification run, and the pitfalls (tmpfs /tmp, tiny-file swarms,
cron PATH).

README: start-here links, ks4 durability bullet and flow chart
updated to the restic era.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

README.md

@@ -2,6 +2,11 @@
 
 Personal infrastructure notes.
 
+**Start here**: [backup-strategy.md](backup-strategy.md) — what is
+backed up, by which tool, on what schedule, and how to restore.
+[new-container.md](new-container.md) — conventions for adding a
+service to ks4 (and getting it backed up automatically).
+
 - [`nuc/`](nuc/) — home lab on `nuc` (see below)
 - [`ks4/`](ks4/) — prod server `ks4` at OVH (see below)
 - [`ks2/`](ks2/) — legacy backup server being decommissioned
@@ -61,15 +66,14 @@ Incus host at OVH — public-facing self-hosted services.
   outline, login (SSO), wireguard, + more — full table in
   [ks4/install.md](ks4/install.md).
 - ⚠️ The ZFS `data` pool is single-disk (not mirrored); durability
-  rests on nightly root cron jobs — 01:00 `incus copy --refresh` of
+  rests on nightly cron jobs — 01:00 `incus copy --refresh` of all
-  all instances to the local `backup` pool on sdb5
+  instances to the local `backup` pool on sdb5, then the instance leg
+  to S3; 05:00 restic (DB dumps + data trees) to S3; nuc pulls the
+  replicas over WireGuard. Full picture and restore procedures:
+  **[backup-strategy.md](backup-strategy.md)**
   ([ks4/local-backup-cron.md](ks4/local-backup-cron.md),
-  [ks4/incus-copy.md](ks4/incus-copy.md)), 04:30 plakar backups
+  [ks4/incus-copy.md](ks4/incus-copy.md),
-  (DB dumps + data trees) to S3
+  [ks4/restic-backup.md](ks4/restic-backup.md)).
-  ([ks4/plakar-s3-data.md](ks4/plakar-s3-data.md)), plus during the
-  transition a 04:00 rsync stopgap to ks2
-  ([ks2 decommission plan](ks2/plan.md)); a pull leg to nuc pool
-  `ks4backup` awaits FTTH.
 
 ## Network flows (nuc <-> ks4)
 
@@ -90,10 +94,11 @@ Incus host at OVH — public-facing self-hosted services.
 |                                   |       |                                     |
 | 03:00 instance snapshots          |       | 03:00 instance snapshots            |
 | 03:30 incus-copy: all instances   |       | 01:00 incus-copy: all instances     |
-|   -> project backup, pool         |       |   -> project backup, zpool          |
+|   -> project backup, pool         |       |   -> project backup, zpool sdb5     |
-|   nucbackup (usb4t/backup/nuc)    |       |   backup (sdb5)          [planned]  |
+|   nucbackup (usb4t/backup/nuc)    |       |   then restic instance leg -> S3    |
-| 05:00 apt upgrade all containers  |       | 04:00 incus-backup.sh: DB dumps     |
+| 05:00 pull ks4:* -> ks4backup     |       | 05:00 restic: DB dumps + data       |
-|                                   |       |   -> /backup (sdb5)      [planned]  |
+|   [pending FTTH]                  |       |   trees -> S3 (restic-data)         |
+| 05:30 apt upgrade all containers  |       | Sun 14:00 restic maintenance        |
 +-----------------------------------+       +-------------------------------------+
    phones/laptops: WG peers 10.8.0.2-3 reach 192.168.1.x through the same endpoint
 ```
@@ -106,6 +111,10 @@ fixed endpoint; ks4's incus API is never exposed to the internet.
 - One markdown file per instance in `nuc/`, containing the full
   install script (idempotent-ish, run as root on the host), first-run
   configuration, and troubleshooting notes.
-- Containers are built from `images:ubuntu/24.04` and configured
+- Containers are built from Debian/Ubuntu images and configured
   exclusively through `incus exec`/`incus config` so the doc is the
   single source of truth — rebuilding = re-running the script.
+- Adding a service to ks4: follow
+  [new-container.md](new-container.md) (no Docker in containers —
+  incus runs OCI images natively and `incus-compose` handles stacks;
+  data paths go into `scripts/restic-paths`).

backup-strategy.md

@@ -0,0 +1,98 @@
+# ks4 backup strategy (2026-08)
+
+Audience: anyone with root on ks4 who needs to know **what is protected,
+where, by which tool, and how to check it**. Details live in the linked
+docs; this page is the map.
+
+## Two tools, no more
+
+| tool | job | why this one |
+|---|---|---|
+| **`incus copy --refresh`** | replicate whole instances (ready-to-start copies) | ZFS-incremental, native to the platform, restores are `incus start` |
+| **restic** | off-site backups to S3 (files, database dumps, instance trees) | dedup + encryption, local metadata cache → a night costs only the churn ([restic-backup.md](ks4/restic-backup.md)) |
+
+plakar was evaluated first and replaced (its nightly cost scaled with
+tree size — [PlakarKorp/plakar#2338](https://github.com/PlakarKorp/plakar/issues/2338));
+the old backup server **ks2 is retired** ([ks2/plan.md](ks2/plan.md)).
+
+## The map
+
+```
+ ks4 (OVH)                                            off-site
+ ┌─────────────────────────────────────────────┐
+ │  live instances        pool `data` (sda)    │
+ │  nextcloud, seafile, mail, git, ...         │
+ │        │                                    │
+ │        │ 01:00  incus copy --refresh        │      nuc (home LAN, via WireGuard)
+ │        ▼                                    │      ┌──────────────────────────┐
+ │  replicas (stopped)    pool `backup` (sdb)  │─────▶│ 05:00 incus copy (pull)  │
+ │  project `backup`                           │      │ pool `ks4backup` (USB)   │
+ │        │                                    │      └──────────────────────────┘
+ │        │ 01:00+ restic (via incus file mount)
+ │        ▼                                    │      OVH Object Storage (S3, sbg)
+ │  ─────────────────────────────────────────  │      ┌──────────────────────────┐
+ │  05:00 restic-backup.sh                     │─────▶│ bucket restic-data       │
+ │    • DB dumps (MariaDB/PostgreSQL, auto)    │      │  dumps + all data trees  │
+ │    • data trees from /root/scripts/restic-paths     │                          │
+ │  01:00+ restic-incus-backup.sh              │─────▶│ bucket restic-incus      │
+ │    • every replica, per file + config yaml  │      │  instance filesystems    │
+ │  Sun 14:00 restic-maintenance.sh            │      └──────────────────────────┘
+ │    • prune + check (rotating full verify)   │
+ └─────────────────────────────────────────────┘
+```
+
+Copies of any given byte: live → sdb replica (same box, other disk)
+→ nuc replica (other site) → S3 (other site, other technology).
+Databases additionally get application-consistent dumps nightly.
+
+## Schedule (root crontab on ks4)
+
+| when | what | log |
+|---|---|---|
+| 01:00 daily | `incus-copy.sh -p backup -s backup` — refresh all replicas onto sdb, then `restic-incus-backup.sh` → S3 | `/var/log/incus-copy.log`, `/var/log/restic-incus.log` |
+| 03:00 daily | instance snapshots (incus profile, 7-day expiry) — what keeps the refreshes incremental | `incus info <inst>` |
+| 05:00 daily | `restic-backup.sh` — dumps + data trees → S3 | `/var/log/restic-backup.log` |
+| 05:00 daily (nuc) | nuc pulls all replicas over WireGuard | nuc: `/var/log/incus-copy-ks4.log` |
+| Sun 14:00 | `restic-maintenance.sh` — prune (capped), structure check, 1/52 data verification | `/var/log/restic-maintenance.log` |
+| 1st of month 06:00 / quarterly 15th | seafile GC dry-run report / real GC ([ks4/seafile-gc.md](ks4/seafile-gc.md)) | `/var/log/seafile-gc.log` |
+
+Retention on S3: 14 daily, 8 weekly, 6 monthly snapshots.
+
+## "Did last night work?" — the 30-second check
+
+```sh
+grep -aE "done \(rc=|FAILED" /var/log/incus-copy.log | tail -2
+grep -aE "snapshot .* saved|done \(rc=|failed" /var/log/restic-backup.log | tail -3
+grep -aE "done \(rc=|failed" /var/log/restic-incus.log | tail -2
+```
+
+`rc=0` everywhere = fine. Any `failed`/`FAILED` line names the culprit
+(instance, database or path). All scripts exit non-zero on any error,
+never silently.
+
+## Restoring
+
+- **A whole instance, fast (same box)**: `incus copy backup:<inst>`
+  style — copy the replica from project `backup` back into `default`
+  ([ks4/incus-copy.md](ks4/incus-copy.md)); from nuc the same via the
+  remote.
+- **A file or directory** (any date within retention):
+  `restic -r s3:s3.sbg.io.cloud.ovh.net/restic-data restore <snapshot> --target /backup/restore-x --include <path>`
+  (never restore into `/tmp` — it is RAM). Env: `. /root/.restic-env`.
+- **A database**: restore the `.sql` dump from `restic-data`
+  (`/backup/dumps/mariadb/<inst>/<db>.sql`, `grants.sql` for users), load
+  it with `incus exec <inst> -- mariadb < dump.sql`.
+- **An instance from S3 only** (ks4 and both replica sites gone):
+  restore its tree + `<inst>.yaml` from `restic-incus`, recreate the
+  instance from the yaml/profile, push the tree back.
+
+Secrets you need for any of this: `/root/.restic-passphrase` and the S3
+keys in `/root/.restic-env` — **both are in the password manager**;
+without the passphrase the S3 buckets are unreadable.
+
+## Adding something to the backups
+
+New instances are picked up automatically by the replica and
+instance legs (all instances, opt-out only). Databases are
+auto-discovered. Only **data trees** need one line in
+`/root/scripts/restic-paths` — see [new-container.md](new-container.md).

new-container.md

@@ -0,0 +1,144 @@
+# Creating a new service on ks4 — conventions
+
+Audience: you have root on ks4 and want to add a service. Follow this
+and the service is automatically replicated, backed up and
+restorable, without touching the backup machinery.
+Companion page: [backup-strategy.md](backup-strategy.md).
+
+## 1. One service = one incus container
+
+```sh
+incus launch images:debian/13 myservice
+incus config device add myservice eth0 nic nictype=bridged parent=incusbr0 \
+    ipv4.address=192.168.1.30            # pick a free IP, note it in the doc
+incus exec myservice -- bash             # install and configure from here
+```
+
+Rules of thumb:
+
+- **Debian 13 or Ubuntu 24.04 images**, unprivileged (the default).
+  Privileged only with a written reason (`nuc/jellyfin-client.md` is
+  the one example: it needs raw device access).
+- **Configure through `incus exec` and write the commands down** in
+  `doc/ks4/<service>.md` as you go. The doc *is* the rebuild
+  procedure — that is the whole convention: re-running it recreates
+  the service.
+- The container is on the NAT bridge `incusbr0` (192.168.1.0/24). It
+  is **not** reachable from the internet by itself.
+- **Web services**: do not open ports. Add an nginx vhost in the
+  `gateway` container that proxies to `http://192.168.1.30:PORT`;
+  gateway is the single HTTP/S entry point and handles TLS.
+  Non-HTTP services (mail, wireguard) use an incus `proxy` device —
+  copy an existing one as a model.
+
+## 2. No Docker inside the container
+
+Docker-in-incus works (`login`/`outline` predate this rule) but costs
+a second layer of networking, storage and updates, and hides the
+service's data behind docker volumes.
+
+Since incus can run **OCI images natively**, an upstream
+`docker-compose.yml` becomes incus instances:
+
+```sh
+incus remote add docker https://docker.io --protocol=oci    # once per host
+incus launch docker:library/redis myservice-redis
+```
+
+and multi-container stacks are described with
+[`incus-compose`](https://github.com/lxc/incus-compose) (upstream,
+compose-file syntax → incus instances). Each piece is then a normal
+instance: same network, same snapshots, same backups, `incus exec` to
+debug, no nested runtime.
+
+If you truly need Docker (upstream ships only a compose stack you
+don't want to translate), say so in the service doc and note where
+the volumes live — you will need them in step 4.
+
+## 3. Snapshots (do nothing, but know why)
+
+The `default` profile gives every instance `snapshots.schedule =
+0 3 * * *`, `snapshots.expiry = 7d`. Leave it. Those snapshots are
+what make the nightly replication incremental — an instance without
+them forces a full re-send of its whole disk every night.
+
+## 4. Wire it into the backups
+
+**Automatic, nothing to do:**
+
+- the 01:00 replica leg copies *every* instance to the `backup` pool
+  and to nuc — new instances included;
+- the S3 instance leg backs up every replica the same way;
+- **databases are auto-discovered**: any running container with
+  MariaDB/MySQL or PostgreSQL gets every non-system database dumped
+  nightly, plus users/grants.
+
+**Manual, one line:** data that lives in the filesystem (uploads,
+repositories, mail spools…) must be listed in
+`/root/scripts/restic-paths`, one absolute host path per line:
+
+```
+/var/lib/incus/storage-pools/data/containers/myservice/rootfs/var/lib/myservice
+```
+
+(The container's `/x` is `…/containers/<name>/rootfs/x` on the host.)
+No comments in that file — every line is read as a path. Regenerable
+caches can be skipped by adding a pattern to
+`/root/scripts/restic-exclude`. Both files live in the `scripts` git
+repo: edit there, commit, `git pull` on ks4.
+
+For the database auto-discovery to work, keep the defaults:
+
+- MariaDB/MySQL: client **and** dump binaries installed, root access
+  over the local unix socket. **Do not leave a `database = …` line in
+  `/root/.my.cnf`** — it breaks `SHOW DATABASES` and the container is
+  then skipped (this happened once and went unnoticed for months).
+- PostgreSQL: reachable as the `postgres` system user (peer auth).
+- PostgreSQL in Docker: the container's image name must contain
+  "postgres" (official images do).
+
+A database you do *not* want dumped (huge, static, re-importable)
+goes into `/root/scripts/plakar-db-exclude` as `<instance>/<db>` —
+every skip is logged, so the list cannot rot silently.
+
+## 5. Verify, once
+
+```sh
+cd /root/scripts && git pull
+/root/scripts/restic-backup.sh -s dumps        # ~2 min: see your DB in the log
+ls /backup/dumps/mariadb/myservice/            # dump present?
+/root/scripts/restic-backup.sh                 # full run, then:
+. /root/.restic-env
+restic -r s3:s3.sbg.io.cloud.ovh.net/restic-data ls latest | grep myservice | head
+```
+
+Seeing your paths and dumps in that listing means the service is
+protected. Do this the day you create the service, not the day you
+need it.
+
+## 6. Pitfalls (learned the hard way)
+
+- **`/tmp` on ks4 is a 16 GiB tmpfs — it is RAM.** Never restore, dump
+  or stage data there; use `/backup/…`.
+- **Avoid millions of tiny files.** A service that keeps every version
+  of everything (seafile did, 24.9 M objects) makes every backup and
+  every filesystem walk take days. If the upstream software has a
+  garbage-collection or retention setting, turn it on *and schedule
+  it* the day you deploy — see [ks4/seafile-gc.md](ks4/seafile-gc.md).
+- **Keep data under one predictable path** per service; scattered data
+  means several `restic-paths` lines and things get forgotten.
+- **Static IP + a note in the service doc**; two containers fighting
+  over one IP is a confusing outage.
+- Cron on ks4 does not have `/usr/local/bin` in its `PATH` — use
+  absolute paths in anything you schedule.
+
+## 7. Checklist
+
+- [ ] container created from a Debian/Ubuntu image, static IP noted
+- [ ] no Docker (or a written reason + volume paths)
+- [ ] install/config steps written in `doc/ks4/<service>.md`
+- [ ] exposed through gateway (HTTP) or a proxy device (other)
+- [ ] snapshots left at the profile default
+- [ ] data paths added to `restic-paths` (committed to git)
+- [ ] `restic-backup.sh` run once; dumps and paths verified in the repo
+- [ ] service added to the table in [ks4/install.md](ks4/install.md)