diff --git a/ks4/restic-backup.md b/ks4/restic-backup.md new file mode 100644 index 0000000..182e2e4 --- /dev/null +++ b/ks4/restic-backup.md @@ -0,0 +1,206 @@ +# restic backups on ks4 — the giant fs trees + +Status: **drafted 2026-08-25, not yet deployed.** + +Why restic exists in this stack: plakar's incremental cost scales +with tree size, not churn (~800 MiB of parent metadata re-read from +S3 + hours of CPU per night on a 500k-file tree — measured, reported +as [PlakarKorp/plakar#2338](https://github.com/PlakarKorp/plakar/issues/2338)), +which priced `nextcloud-fs` and `seafile-fs` out of the plakar data +leg. restic keeps its comparison state in a **local cache**, so a +night costs a local stat-walk plus the churn — the rsync cost model. +plakar keeps everything it is good at (dumps, small sources, small +instances); restic covers only the two giants until #2338 is +resolved (see §7). + +Scope: + +| source | size | files | churn/night (measured) | +|---|---|---|---| +| `/nextcloud` (live, nextcloud container) | 737 G | ~500 k | 90–430 MiB | +| `/opt/seafile` (live, seafile container) | 933 G | millions of ~100 KiB block files | append-mostly | + +Read from the live rootfs paths on the host (crash-consistent, like +the plakar fs sources; DBs are covered by the dump leg). + +## 1. Repository layout — one bucket, two repos + +Two **separate repositories** under one bucket +(`s3:s3.sbg.io.cloud.ovh.net/restic/nextcloud` and `.../restic/seafile`): + +- restic loads the whole repo index into RAM per operation; the two + datasets share no content, so a shared repo only inflates every + run's memory for zero dedup gain. +- prune/check run per repo — seafile's heavy maintenance never blocks + or bloats nextcloud's. +- one bucket keeps credential handling identical to the plakar legs. + +Same passphrase discipline as plakar: generate to +`/root/.restic-passphrase` (mode 600), **copy to the password manager +immediately**. + +## 2. Option analysis (our numbers, not defaults-worship) + +### `--pack-size` (default 16 MiB, max 128) + +Packs are the S3 objects; blobs (~1 MiB avg for data) live inside +them. Pack size does **not** change index RAM (that scales with blob +count) — it changes object count, request count, and rewrite +amplification: + +| pack-size | S3 objects for ~1.2 T stored | trade-off | +|---|---|---| +| 16 MiB | ~75 000 | slow `check`/listing, more requests; prune rewrites are fine-grained | +| **64 MiB (chosen)** | ~19 000 | 4× fewer objects/requests; assembly RAM ≈ pack×connections ≈ 64 M × 5 ≈ 320 MiB — fine | +| 128 MiB | ~9 500 | fewer still, but prune rewrite amplification doubles (a pack with one dead blob rewrites 128 MiB) and upload buffers grow | + +`--pack-size 64` on **every backup/prune invocation** (it is not a +repo property). + +### Memory (the real constraint on this 23/31 GiB-used box) + +- Index RAM ≈ 200–300 B/blob. Estimate: nextcloud ~1 M blobs, + seafile ~2–4 M (tiny files = ≥1 blob each) → **~0.3 GiB and + ~0.6–1.2 GiB** respectively per backup run; prune peaks 2–3×. + This is why the repos are split (§1). +- `GOGC=20` env trades CPU for a substantially smaller Go heap — + enable it if the seafile prune ever pressures the box. + +### Cache location + +`~/.cache/restic` would land on `/` (14 G free) and can reach a few +GiB for repos this size → `--cache-dir /backup/restic-cache` (sdb +pool, plenty of room; cache reads are small but ARC-warm after first +use). + +### Concurrency + +- `--read-concurrency 8` (default 2): more outstanding reads lets the + HDD elevator help on the seek-bound seafile tree — the one knob that + directly targets our 275-IOPS wall. +- `-o s3.connections=8` (default 5): mild upload parallelism bump; + OVH same-region sustains it. + +### Compression + +Repo format v2 default `auto` is right; `max` only buys ~% on +already-compressed content (seafile blocks, images) for real CPU. + +### Excludes — the cheapest optimisation of all + +Nextcloud's `appdata_*/preview` (and `dav-photocache`) are +**regenerable caches holding a large fraction of the 500k files**; +excluding them shrinks walk and churn dramatically: + +``` +# /root/scripts/restic-nextcloud-exclude +/var/lib/incus/storage-pools/data/containers/nextcloud/rootfs/nextcloud/appdata_*/preview +/var/lib/incus/storage-pools/data/containers/nextcloud/rootfs/nextcloud/appdata_*/dav-photocache +``` + +Seafile: no excludes — the block store *is* the data. + +## 3. Install + init (root on ks4) + +```sh +apt install restic # Debian 13 ships ≥0.17 (repo v2, compression, pack-size ok) +head -c 32 /dev/urandom | base64 > /root/.restic-passphrase && chmod 600 /root/.restic-passphrase +# ⚠️ password manager, NOW. + +export AWS_ACCESS_KEY_ID= AWS_SECRET_ACCESS_KEY= # -> /root/.restic-env (mode 600) +export RESTIC_PASSWORD_FILE=/root/.restic-passphrase +restic -r s3:s3.sbg.io.cloud.ovh.net/restic/nextcloud init +restic -r s3:s3.sbg.io.cloud.ovh.net/restic/seafile init +mkdir -p /backup/restic-cache +``` + +## 4. Seed plan + +Order matters: + +1. **Bump the ARC floor first** so metadata walks stay warm between + nightly runs (also helps the plakar legs): + ```sh + echo $((8*1024*1024*1024)) > /sys/module/zfs/parameters/zfs_arc_min + echo "options zfs zfs_arc_min=8589934592" > /etc/modprobe.d/zfs-arc.conf + ``` +2. **nextcloud** (proven ~22 MiB/s from the live pool → ~8 h), in a + screen: + ```sh + restic -r s3:.../restic/nextcloud --cache-dir /backup/restic-cache \ + backup --pack-size 64 --read-concurrency 8 -o s3.connections=8 \ + --exclude-file /root/scripts/restic-nextcloud-exclude \ + /var/lib/incus/storage-pools/data/containers/nextcloud/rootfs/nextcloud + ``` +3. **seafile** — the one-time debt. Expect 1–2 days on the live pool + (warmer than sdb, and `--read-concurrency 8` helps); it is + restartable at any point (committed packs dedup on retry). + +## 5. Nightly runs + retention + verification + +Cron (05:00 — after the 04:30 plakar leg's dump phase, sources don't +overlap; wrap in a `restic-backup.sh` with flock + env, same style as +the plakar drivers): + +```cron +0 5 * * * /root/scripts/restic-backup.sh >> /var/log/restic-backup.log 2>&1 +``` + +Per repo, the script does: + +```sh +restic backup --pack-size 64 --read-concurrency 8 ... # as in §4 +restic forget --keep-daily 14 --keep-weekly 8 --keep-monthly 6 +restic unlock 2>/dev/null # clear stale locks after crashes +``` + +- `forget` without `--prune` is instant; actual **prune runs weekly** + (Sunday, offset from plakar's check): + ```cron + 0 14 * * 0 ... restic prune --max-unused 10% --pack-size 64 ... + ``` + `--max-unused 10%` caps rewrite amplification (leaves ≤10 % dead + data in place instead of rewriting packs eagerly). +- **Verification**: weekly `restic check` (structure only, cheap) + + `check --read-data-subset=1/52` rotating — a full data verification + of every byte once a year, ~25 GiB read per week, for free. + +Expected steady state: nightly = local stat-walk (minutes once ARC is +warm) + churn upload; no per-run S3 metadata storm — the exact +property #2338 is about. + +## 6. Restore test (gate before trusting it) + +```sh +restic -r s3:.../restic/nextcloud --cache-dir /backup/restic-cache \ + restore latest --target /backup/restore-test --include '/' +diff -r ... && rm -rf /backup/restore-test +``` + +Never restore to `/tmp` — it is a 16 GiB tmpfs on ks4. + +## 7. If restic ever takes over ALL the S3 backups + +Decision gate: **the upstream answer to #2338.** Two futures: + +- **plakar fixes the growth problem** → consolidate the giants *back* + onto plakar, retire restic (delete the `restic` bucket) — one tool, + and the incus integration investment keeps compounding. +- **No fix on a useful horizon** → migrate the data leg to restic: + - dumps + 8 small fs sources are ~50 G total: re-seeding into a + `restic/data` repo is an evening, and restic dedups the nightly + dump tree (incl. the static `geo.sql`) exactly like plakar does. + - **the incus leg is the sticking point**: the per-file instance + backups ride our plakar-native integration. restic equivalents + are all worse or heavier: backing up replica rootfs paths + directly (fights incus's on-demand mounts), `incus export` + tarballs (server-side materialization, no room), or `zfs send` + streams into restic's stdin mode (works — `restic backup --stdin` + — but chains restore through zfs+incus knowledge). + - So a full migration really means "restic for all *file* data + + keep plakar only for the incus leg" — two tools either way. + Which is the same operational cost as today's split, minus the + #2338 exposure. Conclusion: **full migration buys little; the + current split is not a temporary hack but a reasonable end + state.** Revisit only if plakar becomes unmaintained or the + incus leg moves to zfs send anyway.