doc/ks4/restic-backup.md

restic backups on ks4 — the S3 leg

Status: decided 2026-08-25 — restic replaces plakar as the S3 backup tool (evaluation first: parallel run + comparison, see §8). Final architecture: two tools — incus copy for replication (local sdb, nuc after FTTH, ks2 push as interim), restic for everything S3.

Why: plakar's incremental cost scales with tree size, not churn (~800 MiB parent metadata re-read from S3 + hours of CPU per night on a 500k-file tree — measured, reported as PlakarKorp/plakar#2338). restic keeps its comparison state in a local cache: a night costs a local stat-walk plus the churn — the rsync cost model. Neither restic nor rustic has plakar's source-plugin architecture, but none is needed: instances are reached via incus file mount (FUSE over the same per-instance sftp API our plakar integration used) — zero patches to maintain (see §6).

1. Repositories — one bucket per repo

bucket = repo	contents	est. size	blobs (index RAM driver)
restic-data	/backup/plakar-dumps dump tree + the 8 small fs sources	~50 G	small
restic-nextcloud	/nextcloud data (live rootfs path)	737 G	~1 M
restic-seafile	/opt/seafile (live rootfs path)	933 G	~2–4 M
restic-instances	the backup-project replicas, per-file via mount	~16 small instances	medium

Separate repos because restic loads the whole repo index into RAM per operation and prune/check are per-repo; separate buckets (not prefixes) for per-bucket OVH usage metrics, independent versioning/object-lock decisions later, and bucket-native credential scoping. One credential pair across all four to start.

Passphrase: /root/.restic-passphrase (mode 600), one for all repos, copy to the password manager immediately. S3 creds + env in /root/.restic-env (mode 600).

2. Option analysis (our numbers, not defaults-worship)

--pack-size (default 16 MiB, max 128)

Packs are the S3 objects; blobs (~1 MiB avg for data) live inside them. Pack size does not change index RAM (that scales with blob count) — it changes object count, request count, and rewrite amplification:

pack-size	S3 objects for ~1.2 T stored	trade-off
16 MiB	~75 000	slow check/listing, more requests; prune rewrites are fine-grained
64 MiB (chosen)	~19 000	4× fewer objects/requests; assembly RAM ≈ pack×connections ≈ 64 M × 5 ≈ 320 MiB — fine
128 MiB	~9 500	fewer still, but prune rewrite amplification doubles and upload buffers grow

--pack-size 64 on every backup/prune invocation (not a repo property).

Memory (the real constraint on this 23/31 GiB-used box)

• Index RAM ≈ 200–300 B/blob → ~0.3 GiB (nextcloud) and ~0.6–1.2 GiB (seafile) per backup run; prune peaks 2–3×. Hence the repo split (§1).
• GOGC=20 trades CPU for a smaller Go heap — enable if the seafile prune ever pressures the box.

Cache location

~/.cache/restic would land on / (14 G free) and can reach a few GiB at this scale → --cache-dir /backup/restic-cache (sdb pool; cache reads are ARC-warm after first touch). tmpfs was considered and rejected: the cache exists to avoid network metadata re-fetches; tmpfs loses it at reboot and eats the RAM the ARC needs.

Concurrency & compression

• --read-concurrency 8 (default 2): more outstanding reads lets the HDD elevator help on seek-bound trees (our 275-IOPS wall).
• -o s3.connections=8 (default 5): mild upload parallelism bump.
• Compression: repo-v2 default auto is right.

Excludes — the cheapest optimisation of all

Nextcloud's appdata_*/preview (and dav-photocache) are regenerable caches holding a large fraction of the 500k files:

# /root/scripts/restic-nextcloud-exclude
/var/lib/incus/storage-pools/data/containers/nextcloud/rootfs/nextcloud/data/appdata_*/preview
/var/lib/incus/storage-pools/data/containers/nextcloud/rootfs/nextcloud/data/appdata_*/dav-photocache

Verify before the seed: find .../data/appdata_* -path '*/preview/*' | wc -l for the win size, then restic backup --dry-run -vv --exclude-file … prints every decision without uploading. Per-user <user>/cache/ dirs are a further candidate. Seafile: no excludes — the block store is the data.

Escape hatch: rustic

rustic speaks the same repository format — switching later is a binary swap on the same repos (decided 2026-08-25: restic first, rustic if performance issues rise; manual upgrades accepted). It would bring lock-free prune and a lower RAM footprint. Don't run both concurrently on one repo (rustic ignores restic's locks); a clean switchover is fine.

3. Install + init (root on ks4)

apt install restic sshfs      # sshfs: needed by `incus file mount` (§6)
head -c 32 /dev/urandom | base64 > /root/.restic-passphrase && chmod 600 /root/.restic-passphrase
# ⚠️ password manager, NOW.

cat > /root/.restic-env <<'EOF'
export AWS_ACCESS_KEY_ID=<AK>
export AWS_SECRET_ACCESS_KEY=<SK>
export RESTIC_PASSWORD_FILE=/root/.restic-passphrase
export RESTIC_CACHE_DIR=/backup/restic-cache
EOF
chmod 600 /root/.restic-env && . /root/.restic-env && mkdir -p /backup/restic-cache

for r in restic-data restic-nextcloud restic-seafile restic-instances; do
    restic -r s3:s3.sbg.io.cloud.ovh.net/$r init
done

4. Seed plan

1. ARC floor first (helps every walk on the box):

   echo 8589934592 > /sys/module/zfs/parameters/zfs_arc_min
   echo "options zfs zfs_arc_min=8589934592" > /etc/modprobe.d/zfs-arc.conf
   update-initramfs -u
   awk '/^c_min/ {printf "%.1f GiB\n", $3/1073741824}' /proc/spl/kstat/zfs/arcstats
   

   ⚠️ the floor is defended even under pressure — real headroom is ~31 G − services − 8 G; revisit before adding a fat service.
2. restic-data (~50 G — an hour).
3. restic-nextcloud (proven ~22 MiB/s from the live pool → ~8 h), screen:

   restic -r s3:s3.sbg.io.cloud.ovh.net/restic-nextcloud \
     backup --pack-size 64 --read-concurrency 8 -o s3.connections=8 \
     --exclude-file /root/scripts/restic-nextcloud-exclude \
     /var/lib/incus/storage-pools/data/containers/nextcloud/rootfs/nextcloud
   

4. restic-instances (16 small replicas via §6 — a few hours).
5. restic-seafile — the one-time debt: 1–2 days, restartable at any point (committed packs dedup on retry).

5. Nightly driver — scripts/restic-backup.sh

Same shape as the plakar drivers (flock, env, loud logging). Phases:

1. dumps — lifted verbatim from plakar-backup.sh: incus DB dumps + auto-discovered MariaDB/PostgreSQL (native and docker) dumps into /backup/plakar-dumps (path kept), then restic -r …/restic-data backup of the dump tree and the 8 small source paths (one snapshot, paths from a list file).
2. giants — nextcloud (with excludes) and seafile into their repos.
3. instances — §6 loop into restic-instances.
4. retention — per repo: restic forget --keep-daily 14 --keep-weekly 8 --keep-monthly 6 (instant without --prune).

0 5 * * * /root/scripts/restic-backup.sh >> /var/log/restic-backup.log 2>&1

Weekly maintenance (Sunday; prune takes an exclusive lock — never overlap the 05:00 run):

0 14 * * 0 /root/scripts/restic-maintenance.sh >> /var/log/restic-maintenance.log 2>&1

Per repo: restic prune --max-unused 10% --max-repack-size 4G --pack-size 64 — our dead-data rate is ~0.1–0.4 GiB/day against ~700 GiB repos (seafile append-mostly ≈ nothing dies), so 10 % tolerated slack lets prune skip repacking for months, and the 4 G cap bounds any single Sunday to ~10–15 min even after a mass deletion. Then restic check (structure, cheap) + check --read-data-subset=1/52 rotating — full data verification of every byte once a year, ~25 GiB read per week.

6. Instances without a plugin — incus file mount

Neither restic nor rustic accepts source plugins, and forking the backup tool is the wrong place to carry a patch. Instead, incus exposes any instance (including stopped replicas) as a FUSE mount over the same per-instance sftp API the plakar integration used:

mkdir -p /run/restic-incus
for inst in $(incus list --project backup -c n -f csv); do
    case ",$EXCLUDE_INSTANCES," in *",$inst,"*) continue;; esac   # nextcloud,seafile
    incus config show "$inst" --project backup --expanded > "/backup/plakar-dumps/incus/$inst.yaml"
    incus file mount "$inst/" /run/restic-incus --project backup &   # foreground process; background + kill
    MPID=$!; sleep 2
    restic -r s3:…/restic-instances backup --pack-size 64 --tag "$inst" /run/restic-incus
    kill $MPID; wait $MPID 2>/dev/null; umount /run/restic-incus 2>/dev/null
done

(Exact mount lifecycle to be hardened in the script — incus file mount runs in the foreground and needs sshfs installed.) All instances by default, opt-out via a variable — the no-manifest-drift rule survives the tool change. Chained after the 01:00 incus-copy in its cron entry once validated, same freshness reasoning as before.

Bonus over plakar: one shared repo dedups the Ubuntu base across all 16 replicas. Restore granularity: per-file; instance definitions ride the dump tree (incus/<inst>.yaml + incus-global-db.sql).

7. Restore test (gate)

Per repo, one restore + diff (never to /tmp — 16 GiB tmpfs):

restic -r s3:…/restic-nextcloud restore latest --target /backup/restore-test --include '<subtree>'
diff -r … && rm -rf /backup/restore-test

8. Evaluation protocol + plakar retirement

• Parallel week: plakar 04:30 cron keeps running untouched; restic runs at 05:00 on the same data. Same nights, same churn → direct comparison (wall, bytes read from store, bytes written, peak RSS via /usr/bin/time -v) against plakar's measured table in plakar-s3-data.md. Restore-speed compared on the same 10 GiB dumps tree (plakar measured: 39 min).
• Cutover (after the comparison confirms + restore test passes): drop the plakar 04:30 + Sunday-check crons; 01:00 becomes incus-copy ; restic instances phase (or stays in the 05:00 run). Update plan.md legs table.
• plakar retirement, one exception: keep plakar installed and keep the plakar-data kloset until #2338 concludes — the issue offers debug runs against that repository. Delete the plakar-incus bucket immediately; plakar-data a few weeks later. The integration-incus repo retires from production; optionally still publish to PlakarKorp/hub as a community contribution.
• Interim ks2 push (until the nuc leg seeds after FTTH, expected before end of September): re-enable the replica push so instances keep an off-site copy meanwhile:

  0 2 * * * /root/scripts/incus-copy.sh -d ks2 -m push >> /var/log/incus-copy.log 2>&1