From de72120105ae7ff0d05754263ec12308e3226557 Mon Sep 17 00:00:00 2001 From: Julien Lutran Date: Tue, 25 Aug 2026 12:50:07 +0200 Subject: [PATCH] restic-backup: two-bucket layout (restic-data + restic-incus), two drivers MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Giants merged into restic-data (single nightly invocation, one index load, --group-by host retention); instances leg gets per-instance mountpoints — restic picks parents by host+path, a shared mountpoint would cause nightly full re-reads. Co-Authored-By: Claude Fable 5 --- ks4/restic-backup.md | 98 ++++++++++++++++++++++++-------------------- 1 file changed, 54 insertions(+), 44 deletions(-) diff --git a/ks4/restic-backup.md b/ks4/restic-backup.md index 9f146c6..78aed25 100644 --- a/ks4/restic-backup.md +++ b/ks4/restic-backup.md @@ -17,20 +17,21 @@ needed: instances are reached via `incus file mount` (FUSE over the same per-instance sftp API our plakar integration used) — zero patches to maintain (see §6). -## 1. Repositories — one bucket per repo +## 1. Repositories — two buckets, two drivers (decided 2026-08-25) -| bucket = repo | contents | est. size | blobs (index RAM driver) | -|---|---|---|---| -| `restic-data` | `/backup/plakar-dumps` dump tree + the 8 small fs sources | ~50 G | small | -| `restic-nextcloud` | `/nextcloud` data (live rootfs path) | 737 G | ~1 M | -| `restic-seafile` | `/opt/seafile` (live rootfs path) | 933 G | ~2–4 M | -| `restic-instances` | the `backup`-project replicas, per-file via mount | ~16 small instances | medium | +| bucket = repo | contents | driver script | +|---|---|---| +| `restic-data` | SQL dumps + **all** fs sources (8 small + nextcloud + seafile) | `restic-backup.sh` | +| `restic-incus` | the `backup`-project replicas, per-file via mount | `restic-incus-backup.sh` | -Separate repos because restic loads the whole repo index into RAM per -operation and prune/check are per-repo; separate **buckets** (not -prefixes) for per-bucket OVH usage metrics, independent -versioning/object-lock decisions later, and bucket-native credential -scoping. One credential pair across all four to start. +Mirrors the plakar-era split (data leg / incus leg). Accepted +trade-off: `restic-data` carries the giants' blobs (~3–5 M once +seafile is in), so **every operation on it loads a ~1.5–2 GiB +index** — mitigated by running the whole nightly as a *single* +`restic backup` invocation (one index load, one snapshot covering +all paths), `GOGC=20` in reserve, rustic as the escape hatch. +Buckets (not prefixes) for per-bucket OVH metrics and independent +versioning/lock decisions later. Passphrase: `/root/.restic-passphrase` (mode 600), one for all repos, **copy to the password manager immediately**. S3 creds + env in @@ -118,7 +119,7 @@ export RESTIC_CACHE_DIR=/backup/restic-cache EOF chmod 600 /root/.restic-env && . /root/.restic-env && mkdir -p /backup/restic-cache -for r in restic-data restic-nextcloud restic-seafile restic-instances; do +for r in restic-data restic-incus; do restic -r s3:s3.sbg.io.cloud.ovh.net/$r init done ``` @@ -134,18 +135,15 @@ done ``` ⚠️ the floor is defended even under pressure — real headroom is ~31 G − services − 8 G; revisit before adding a fat service. -2. `restic-data` (~50 G — an hour). -3. `restic-nextcloud` (proven ~22 MiB/s from the live pool → ~8 h), - screen: - ```sh - restic -r s3:s3.sbg.io.cloud.ovh.net/restic-nextcloud \ - backup --pack-size 64 --read-concurrency 8 -o s3.connections=8 \ - --exclude-file /root/scripts/restic-nextcloud-exclude \ - /var/lib/incus/storage-pools/data/containers/nextcloud/rootfs/nextcloud - ``` -4. `restic-instances` (16 small replicas via §6 — a few hours). -5. `restic-seafile` — the one-time debt: 1–2 days, restartable at any - point (committed packs dedup on retry). +2. `restic-data`, staged (all into the same repo; separate + invocations so each stage is independently restartable — + `restic-backup.sh -s` runs the same thing): + - dumps + 8 small sources (~50 G — an hour) + - nextcloud path with the exclude file (proven ~22 MiB/s from the + live pool → ~8 h, screen) + - seafile — the one-time debt: 1–2 days, restartable at any point + (committed packs dedup on retry) +3. `restic-incus` (16 small replicas via §6 — a few hours). ## 5. Nightly driver — `scripts/restic-backup.sh` @@ -153,14 +151,21 @@ Same shape as the plakar drivers (flock, env, loud logging). Phases: 1. **dumps** — lifted verbatim from `plakar-backup.sh`: incus DB dumps + auto-discovered MariaDB/PostgreSQL (native and docker) - dumps into `/backup/plakar-dumps` (path kept), then - `restic -r …/restic-data backup` of the dump tree **and** the 8 - small source paths (one snapshot, paths from a list file). -2. **giants** — nextcloud (with excludes) and seafile into their - repos. -3. **instances** — §6 loop into `restic-instances`. -4. **retention** — per repo: `restic forget --keep-daily 14 - --keep-weekly 8 --keep-monthly 6` (instant without `--prune`). + dumps into the dump dir (during the parallel week restic dumps + into its own `/backup/dumps`; carry the archived `geo.sql` over + once with `cp -a`). +2. **one backup invocation** into `restic-data`: dump dir + all fs + paths from `/root/scripts/restic-paths` + (`--files-from-verbatim`), nextcloud excludes applied globally + (patterns only match nextcloud paths). One index load, one + snapshot per night. +3. **retention** — `restic forget --group-by host --keep-daily 14 + --keep-weekly 8 --keep-monthly 6` (instant without `--prune`; + `--group-by host` because seed-era snapshots have different path + sets than nightly ones and must age in one group). + +The instances leg is `restic-incus-backup.sh` (§6), chained after the +01:00 incus-copy once validated. ```cron 0 5 * * * /root/scripts/restic-backup.sh >> /var/log/restic-backup.log 2>&1 @@ -190,22 +195,27 @@ exposes any instance (including **stopped replicas**) as a FUSE mount over the same per-instance sftp API the plakar integration used: ```sh -mkdir -p /run/restic-incus for inst in $(incus list --project backup -c n -f csv); do case ",$EXCLUDE_INSTANCES," in *",$inst,"*) continue;; esac # nextcloud,seafile - incus config show "$inst" --project backup --expanded > "/backup/plakar-dumps/incus/$inst.yaml" - incus file mount "$inst/" /run/restic-incus --project backup & # foreground process; background + kill + mnt=/run/restic-incus/$inst # per-instance mountpoint: restic + mkdir -p "$mnt" # picks parents by host+path, so + incus config show "$inst" --project backup --expanded > "$mnt.yaml" + incus file mount "$inst/" "$mnt" --project backup & # foreground proc MPID=$!; sleep 2 - restic -r s3:…/restic-instances backup --pack-size 64 --tag "$inst" /run/restic-incus - kill $MPID; wait $MPID 2>/dev/null; umount /run/restic-incus 2>/dev/null + restic -r s3:…/restic-incus backup --pack-size 64 --tag "$inst" "$mnt" "$mnt.yaml" + kill $MPID; wait $MPID 2>/dev/null; umount "$mnt" 2>/dev/null done ``` -(Exact mount lifecycle to be hardened in the script — `incus file -mount` runs in the foreground and needs `sshfs` installed.) All -instances by default, opt-out via a variable — the no-manifest-drift -rule survives the tool change. Chained after the 01:00 incus-copy in -its cron entry once validated, same freshness reasoning as before. +⚠️ **per-instance mountpoints are load-bearing**: a shared mountpoint +would make every snapshot's parent the previous *other* instance's +snapshot (restic matches parents by host+path) → nightly full +re-reads. Distinct stable paths give correct parents and stat-skip. +Mount lifecycle hardened in `restic-incus-backup.sh` (`incus file +mount` runs foreground; needs `sshfs`). All instances by default, +opt-out via a variable — the no-manifest-drift rule survives the tool +change. Chained after the 01:00 incus-copy once validated, same +freshness reasoning as before. Bonus over plakar: one shared repo dedups the Ubuntu base across all 16 replicas. Restore granularity: per-file; instance definitions ride @@ -216,7 +226,7 @@ the dump tree (`incus/.yaml` + `incus-global-db.sql`). Per repo, one restore + diff (never to `/tmp` — 16 GiB tmpfs): ```sh -restic -r s3:…/restic-nextcloud restore latest --target /backup/restore-test --include '' +restic -r s3:…/restic-data restore latest --target /backup/restore-test --include '' diff -r … && rm -rf /backup/restore-test ```