doc/ks4/restic-backup.md

restic backups on ks4 — the giant fs trees

Status: drafted 2026-08-25, not yet deployed.

Why restic exists in this stack: plakar's incremental cost scales with tree size, not churn (~800 MiB of parent metadata re-read from S3 + hours of CPU per night on a 500k-file tree — measured, reported as PlakarKorp/plakar#2338), which priced nextcloud-fs and seafile-fs out of the plakar data leg. restic keeps its comparison state in a local cache, so a night costs a local stat-walk plus the churn — the rsync cost model. plakar keeps everything it is good at (dumps, small sources, small instances); restic covers only the two giants until #2338 is resolved (see §7).

Scope:

source	size	files	churn/night (measured)
/nextcloud (live, nextcloud container)	737 G	~500 k	90–430 MiB
/opt/seafile (live, seafile container)	933 G	millions of ~100 KiB block files	append-mostly

Read from the live rootfs paths on the host (crash-consistent, like the plakar fs sources; DBs are covered by the dump leg).

1. Repository layout — one bucket, two repos

Two separate repositories under one bucket (s3:s3.sbg.io.cloud.ovh.net/restic/nextcloud and .../restic/seafile):

• restic loads the whole repo index into RAM per operation; the two datasets share no content, so a shared repo only inflates every run's memory for zero dedup gain.
• prune/check run per repo — seafile's heavy maintenance never blocks or bloats nextcloud's.
• one bucket keeps credential handling identical to the plakar legs.

Same passphrase discipline as plakar: generate to /root/.restic-passphrase (mode 600), copy to the password manager immediately.

2. Option analysis (our numbers, not defaults-worship)

--pack-size (default 16 MiB, max 128)

Packs are the S3 objects; blobs (~1 MiB avg for data) live inside them. Pack size does not change index RAM (that scales with blob count) — it changes object count, request count, and rewrite amplification:

pack-size	S3 objects for ~1.2 T stored	trade-off
16 MiB	~75 000	slow check/listing, more requests; prune rewrites are fine-grained
64 MiB (chosen)	~19 000	4× fewer objects/requests; assembly RAM ≈ pack×connections ≈ 64 M × 5 ≈ 320 MiB — fine
128 MiB	~9 500	fewer still, but prune rewrite amplification doubles (a pack with one dead blob rewrites 128 MiB) and upload buffers grow

--pack-size 64 on every backup/prune invocation (it is not a repo property).

Memory (the real constraint on this 23/31 GiB-used box)

• Index RAM ≈ 200–300 B/blob. Estimate: nextcloud ~1 M blobs, seafile ~2–4 M (tiny files = ≥1 blob each) → ~0.3 GiB and ~0.6–1.2 GiB respectively per backup run; prune peaks 2–3×. This is why the repos are split (§1).
• GOGC=20 env trades CPU for a substantially smaller Go heap — enable it if the seafile prune ever pressures the box.

Cache location

~/.cache/restic would land on / (14 G free) and can reach a few GiB for repos this size → --cache-dir /backup/restic-cache (sdb pool, plenty of room; cache reads are small but ARC-warm after first use).

Concurrency

• --read-concurrency 8 (default 2): more outstanding reads lets the HDD elevator help on the seek-bound seafile tree — the one knob that directly targets our 275-IOPS wall.
• -o s3.connections=8 (default 5): mild upload parallelism bump; OVH same-region sustains it.

Compression

Repo format v2 default auto is right; max only buys ~% on already-compressed content (seafile blocks, images) for real CPU.

Excludes — the cheapest optimisation of all

Nextcloud's appdata_*/preview (and dav-photocache) are regenerable caches holding a large fraction of the 500k files; excluding them shrinks walk and churn dramatically:

# /root/scripts/restic-nextcloud-exclude
/var/lib/incus/storage-pools/data/containers/nextcloud/rootfs/nextcloud/data/appdata_*/preview
/var/lib/incus/storage-pools/data/containers/nextcloud/rootfs/nextcloud/data/appdata_*/dav-photocache

Seafile: no excludes — the block store is the data.

Verify excludes before the seed: count the win with find .../appdata_* -path '*/preview/*' | wc -l, then restic backup --dry-run -vv --exclude-file ... prints every include/exclude decision without uploading. (Per-user <user>/cache/ dirs are a further regenerable candidate if the count justifies it.)

Cache on tmpfs was considered and rejected: the cache exists to avoid network metadata re-fetches; tmpfs loses it at reboot (forcing a full re-download), eats the RAM the ARC needs, and buys nothing — disk-backed cache reads are ARC-warm after first touch anyway.

3. Install + init (root on ks4)

apt install restic            # Debian 13 ships ≥0.17 (repo v2, compression, pack-size ok)
head -c 32 /dev/urandom | base64 > /root/.restic-passphrase && chmod 600 /root/.restic-passphrase
# ⚠️ password manager, NOW.

export AWS_ACCESS_KEY_ID=<AK> AWS_SECRET_ACCESS_KEY=<SK>   # -> /root/.restic-env (mode 600)
export RESTIC_PASSWORD_FILE=/root/.restic-passphrase
restic -r s3:s3.sbg.io.cloud.ovh.net/restic/nextcloud init
restic -r s3:s3.sbg.io.cloud.ovh.net/restic/seafile  init
mkdir -p /backup/restic-cache

4. Seed plan

Order matters:

1. Bump the ARC floor first so metadata walks stay warm between nightly runs (also helps the plakar legs):

   echo 8589934592 > /sys/module/zfs/parameters/zfs_arc_min
   echo "options zfs zfs_arc_min=8589934592" > /etc/modprobe.d/zfs-arc.conf
   update-initramfs -u    # zfs loads from the initramfs
   awk '/^c_min/ {printf "%.1f GiB\n", $3/1073741824}' /proc/spl/kstat/zfs/arcstats
   

   ⚠️ the floor is defended even under memory pressure — real headroom becomes ~31G − services − 8G; revisit before adding any fat new service.
2. nextcloud (proven ~22 MiB/s from the live pool → ~8 h), in a screen:

   restic -r s3:.../restic/nextcloud --cache-dir /backup/restic-cache \
     backup --pack-size 64 --read-concurrency 8 -o s3.connections=8 \
     --exclude-file /root/scripts/restic-nextcloud-exclude \
     /var/lib/incus/storage-pools/data/containers/nextcloud/rootfs/nextcloud
   

3. seafile — the one-time debt. Expect 1–2 days on the live pool (warmer than sdb, and --read-concurrency 8 helps); it is restartable at any point (committed packs dedup on retry).

5. Nightly runs + retention + verification

Cron (05:00 — after the 04:30 plakar leg's dump phase, sources don't overlap; wrap in a restic-backup.sh with flock + env, same style as the plakar drivers):

0 5 * * * /root/scripts/restic-backup.sh >> /var/log/restic-backup.log 2>&1

Per repo, the script does:

restic backup  --pack-size 64 --read-concurrency 8 ...   # as in §4
restic forget  --keep-daily 14 --keep-weekly 8 --keep-monthly 6
restic unlock 2>/dev/null                                  # clear stale locks after crashes

• forget without --prune is instant; actual prune runs weekly (Sunday, offset from plakar's check; prune takes an exclusive lock — never overlap the 05:00 backups):

  0 14 * * 0 ... restic prune --max-unused 10% --max-repack-size 4G --pack-size 64 ...
  

  Tuning rationale (our churn: ~0.1–0.4 GiB/day dead data vs ~700 GiB repos, seafile append-mostly ≈ nothing dies): --max-unused 10% lets prune skip repacking for months, carrying ≤10 % slack in cheap object storage; --max-repack-size 4G bounds any single Sunday's rewrite to ~10–15 min even after a mass deletion, deferring the rest to following weeks. (--repack-small only matters for repos with legacy sub-64 MiB packs — not us, we start at 64.)
• Verification: weekly restic check (structure only, cheap) + check --read-data-subset=1/52 rotating — a full data verification of every byte once a year, ~25 GiB read per week, for free.

Expected steady state: nightly = local stat-walk (minutes once ARC is warm) + churn upload; no per-run S3 metadata storm — the exact property #2338 is about.

6. Restore test (gate before trusting it)

restic -r s3:.../restic/nextcloud --cache-dir /backup/restic-cache \
  restore latest --target /backup/restore-test --include '/<some subtree>'
diff -r ... && rm -rf /backup/restore-test

Never restore to /tmp — it is a 16 GiB tmpfs on ks4.

7. If restic ever takes over ALL the S3 backups

Decision gate: the upstream answer to #2338. Two futures:

• plakar fixes the growth problem → consolidate the giants back onto plakar, retire restic (delete the restic bucket) — one tool, and the incus integration investment keeps compounding.
• No fix on a useful horizon → migrate the data leg to restic:
  • dumps + 8 small fs sources are ~50 G total: re-seeding into a restic/data repo is an evening, and restic dedups the nightly dump tree (incl. the static geo.sql) exactly like plakar does.
  • the incus leg is the sticking point: the per-file instance backups ride our plakar-native integration. restic equivalents are all worse or heavier: backing up replica rootfs paths directly (fights incus's on-demand mounts), incus export tarballs (server-side materialization, no room), or zfs send streams into restic's stdin mode (works — restic backup --stdin — but chains restore through zfs+incus knowledge).
  • So a full migration really means "restic for all file data + keep plakar only for the incus leg" — two tools either way. Which is the same operational cost as today's split, minus the #2338 exposure. Conclusion: full migration buys little; the current split is not a temporary hack but a reasonable end state. Revisit only if plakar becomes unmaintained or the incus leg moves to zfs send anyway.