restic-backup: two-bucket layout (restic-data + restic-incus), two drivers

Giants merged into restic-data (single nightly invocation, one index
load, --group-by host retention); instances leg gets per-instance
mountpoints — restic picks parents by host+path, a shared mountpoint
would cause nightly full re-reads.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

ks4/restic-backup.md

@@ -17,20 +17,21 @@ needed: instances are reached via `incus file mount` (FUSE over the
 same per-instance sftp API our plakar integration used) — zero
 patches to maintain (see §6).
 
-## 1. Repositories — one bucket per repo
+## 1. Repositories — two buckets, two drivers (decided 2026-08-25)
 
-| bucket = repo | contents | est. size | blobs (index RAM driver) |
-|---|---|---|---|
-| `restic-data` | `/backup/plakar-dumps` dump tree + the 8 small fs sources | ~50 G | small |
-| `restic-nextcloud` | `/nextcloud` data (live rootfs path) | 737 G | ~1 M |
-| `restic-seafile` | `/opt/seafile` (live rootfs path) | 933 G | ~2–4 M |
-| `restic-instances` | the `backup`-project replicas, per-file via mount | ~16 small instances | medium |
+| bucket = repo | contents | driver script |
+|---|---|---|
+| `restic-data` | SQL dumps + **all** fs sources (8 small + nextcloud + seafile) | `restic-backup.sh` |
+| `restic-incus` | the `backup`-project replicas, per-file via mount | `restic-incus-backup.sh` |
 
-Separate repos because restic loads the whole repo index into RAM per
-operation and prune/check are per-repo; separate **buckets** (not
-prefixes) for per-bucket OVH usage metrics, independent
-versioning/object-lock decisions later, and bucket-native credential
-scoping. One credential pair across all four to start.
+Mirrors the plakar-era split (data leg / incus leg). Accepted
+trade-off: `restic-data` carries the giants' blobs (~3–5 M once
+seafile is in), so **every operation on it loads a ~1.5–2 GiB
+index** — mitigated by running the whole nightly as a *single*
+`restic backup` invocation (one index load, one snapshot covering
+all paths), `GOGC=20` in reserve, rustic as the escape hatch.
+Buckets (not prefixes) for per-bucket OVH metrics and independent
+versioning/lock decisions later.
 
 Passphrase: `/root/.restic-passphrase` (mode 600), one for all repos,
 **copy to the password manager immediately**. S3 creds + env in
@@ -118,7 +119,7 @@ export RESTIC_CACHE_DIR=/backup/restic-cache
 EOF
 chmod 600 /root/.restic-env && . /root/.restic-env && mkdir -p /backup/restic-cache
 
-for r in restic-data restic-nextcloud restic-seafile restic-instances; do
+for r in restic-data restic-incus; do
     restic -r s3:s3.sbg.io.cloud.ovh.net/$r init
 done
 ```
@@ -134,18 +135,15 @@ done
    ```
    ⚠️ the floor is defended even under pressure — real headroom is
    ~31 G − services − 8 G; revisit before adding a fat service.
-2. `restic-data` (~50 G — an hour).
-3. `restic-nextcloud` (proven ~22 MiB/s from the live pool → ~8 h),
-   screen:
-   ```sh
-   restic -r s3:s3.sbg.io.cloud.ovh.net/restic-nextcloud \
-     backup --pack-size 64 --read-concurrency 8 -o s3.connections=8 \
-     --exclude-file /root/scripts/restic-nextcloud-exclude \
-     /var/lib/incus/storage-pools/data/containers/nextcloud/rootfs/nextcloud
-   ```
-4. `restic-instances` (16 small replicas via §6 — a few hours).
-5. `restic-seafile` — the one-time debt: 1–2 days, restartable at any
-   point (committed packs dedup on retry).
+2. `restic-data`, staged (all into the same repo; separate
+   invocations so each stage is independently restartable —
+   `restic-backup.sh -s` runs the same thing):
+   - dumps + 8 small sources (~50 G — an hour)
+   - nextcloud path with the exclude file (proven ~22 MiB/s from the
+     live pool → ~8 h, screen)
+   - seafile — the one-time debt: 1–2 days, restartable at any point
+     (committed packs dedup on retry)
+3. `restic-incus` (16 small replicas via §6 — a few hours).
 
 ## 5. Nightly driver — `scripts/restic-backup.sh`
 
@@ -153,14 +151,21 @@ Same shape as the plakar drivers (flock, env, loud logging). Phases:
 
 1. **dumps** — lifted verbatim from `plakar-backup.sh`: incus DB
    dumps + auto-discovered MariaDB/PostgreSQL (native and docker)
-   dumps into `/backup/plakar-dumps` (path kept), then
-   `restic -r …/restic-data backup` of the dump tree **and** the 8
-   small source paths (one snapshot, paths from a list file).
-2. **giants** — nextcloud (with excludes) and seafile into their
-   repos.
-3. **instances** — §6 loop into `restic-instances`.
-4. **retention** — per repo: `restic forget --keep-daily 14
-   --keep-weekly 8 --keep-monthly 6` (instant without `--prune`).
+   dumps into the dump dir (during the parallel week restic dumps
+   into its own `/backup/dumps`; carry the archived `geo.sql` over
+   once with `cp -a`).
+2. **one backup invocation** into `restic-data`: dump dir + all fs
+   paths from `/root/scripts/restic-paths`
+   (`--files-from-verbatim`), nextcloud excludes applied globally
+   (patterns only match nextcloud paths). One index load, one
+   snapshot per night.
+3. **retention** — `restic forget --group-by host --keep-daily 14
+   --keep-weekly 8 --keep-monthly 6` (instant without `--prune`;
+   `--group-by host` because seed-era snapshots have different path
+   sets than nightly ones and must age in one group).
+
+The instances leg is `restic-incus-backup.sh` (§6), chained after the
+01:00 incus-copy once validated.
 
 ```cron
 0 5 * * * /root/scripts/restic-backup.sh >> /var/log/restic-backup.log 2>&1
@@ -190,22 +195,27 @@ exposes any instance (including **stopped replicas**) as a FUSE mount
 over the same per-instance sftp API the plakar integration used:
 
 ```sh
-mkdir -p /run/restic-incus
 for inst in $(incus list --project backup -c n -f csv); do
     case ",$EXCLUDE_INSTANCES," in *",$inst,"*) continue;; esac   # nextcloud,seafile
-    incus config show "$inst" --project backup --expanded > "/backup/plakar-dumps/incus/$inst.yaml"
-    incus file mount "$inst/" /run/restic-incus --project backup &   # foreground process; background + kill
+    mnt=/run/restic-incus/$inst          # per-instance mountpoint: restic
+    mkdir -p "$mnt"                      # picks parents by host+path, so
+    incus config show "$inst" --project backup --expanded > "$mnt.yaml"
+    incus file mount "$inst/" "$mnt" --project backup &   # foreground proc
     MPID=$!; sleep 2
-    restic -r s3:…/restic-instances backup --pack-size 64 --tag "$inst" /run/restic-incus
-    kill $MPID; wait $MPID 2>/dev/null; umount /run/restic-incus 2>/dev/null
+    restic -r s3:…/restic-incus backup --pack-size 64 --tag "$inst" "$mnt" "$mnt.yaml"
+    kill $MPID; wait $MPID 2>/dev/null; umount "$mnt" 2>/dev/null
 done
 ```
 
-(Exact mount lifecycle to be hardened in the script — `incus file
-mount` runs in the foreground and needs `sshfs` installed.) All
-instances by default, opt-out via a variable — the no-manifest-drift
-rule survives the tool change. Chained after the 01:00 incus-copy in
-its cron entry once validated, same freshness reasoning as before.
+⚠️ **per-instance mountpoints are load-bearing**: a shared mountpoint
+would make every snapshot's parent the previous *other* instance's
+snapshot (restic matches parents by host+path) → nightly full
+re-reads. Distinct stable paths give correct parents and stat-skip.
+Mount lifecycle hardened in `restic-incus-backup.sh` (`incus file
+mount` runs foreground; needs `sshfs`). All instances by default,
+opt-out via a variable — the no-manifest-drift rule survives the tool
+change. Chained after the 01:00 incus-copy once validated, same
+freshness reasoning as before.
 
 Bonus over plakar: one shared repo dedups the Ubuntu base across all
 16 replicas. Restore granularity: per-file; instance definitions ride
@@ -216,7 +226,7 @@ the dump tree (`incus/<inst>.yaml` + `incus-global-db.sql`).
 Per repo, one restore + diff (never to `/tmp` — 16 GiB tmpfs):
 
 ```sh
-restic -r s3:…/restic-nextcloud restore latest --target /backup/restore-test --include '<subtree>'
+restic -r s3:…/restic-data restore latest --target /backup/restore-test --include '<subtree>'
 diff -r … && rm -rf /backup/restore-test
 ```