doc/ks4/restic-backup.md

restic backups on ks4 — the S3 leg

Status: live since 2026-08-28. This is the S3 backup leg of the strategy: database dumps + selected filesystem trees into the restic-data bucket, nightly at 05:00. Instances themselves are handled by replication (sdb, nuc after FTTH), not by this leg — an instance-filesystem leg (restic-incus, §6) was written and shelved; its script and empty repo stay in place should that change.

Why restic: its comparison state lives in a local cache, so a night costs a local stat-walk plus the churn — the rsync cost model, with dedup, compression and encryption on top. (The predecessor and the measurements that motivated the switch are in plakar-s3-data.md — reference only.)

1. Repositories — two buckets, two drivers (decided 2026-08-25)

bucket = repo	contents	driver script
restic-data	SQL dumps + all fs sources (8 small + nextcloud + seafile)	restic-backup.sh
restic-incus	(shelved — empty repo, see §6) the backup-project replicas, per-file via mount	restic-incus-backup.sh

Two repos so each can be pruned and checked independently. Accepted trade-off: restic-data carries the giants' blobs (~3–5 M once seafile is in), so every operation on it loads a ~1.5–2 GiB index — mitigated by running the whole nightly as a single restic backup invocation (one index load, one snapshot covering all paths), GOGC=20 in reserve, rustic as the escape hatch. Buckets (not prefixes) for per-bucket OVH metrics and independent versioning/lock decisions later.

Passphrase: /root/.restic-passphrase (mode 600), one for all repos, copy to the password manager immediately. S3 creds + env in /root/.restic-env (mode 600).

2. Option analysis (our numbers, not defaults-worship)

--pack-size (default 16 MiB, max 128)

Packs are the S3 objects; blobs (~1 MiB avg for data) live inside them. Pack size does not change index RAM (that scales with blob count) — it changes object count, request count, and rewrite amplification:

pack-size	S3 objects for ~1.2 T stored	trade-off
16 MiB	~75 000	slow check/listing, more requests; prune rewrites are fine-grained
64 MiB (chosen)	~19 000	4× fewer objects/requests; assembly RAM ≈ pack×connections ≈ 64 M × 5 ≈ 320 MiB — fine
128 MiB	~9 500	fewer still, but prune rewrite amplification doubles and upload buffers grow

--pack-size 64 on every backup/prune invocation (not a repo property).

Memory (the real constraint on this 23/31 GiB-used box)

• Index RAM ≈ 200–300 B/blob → combined restic-data index ~1 GiB per operation; prune peaks 2–3×. The seafile side of that estimate collapsed after its garbage collection (24.9 M → 0.5 M objects, seafile-gc.md). Measure peak RSS with /usr/bin/time -v if the box ever feels tight.
• GOGC=20 trades CPU for a smaller Go heap — enable if the seafile prune ever pressures the box.

Cache location

~/.cache/restic would land on / (14 G free) and can reach a few GiB at this scale → --cache-dir /backup/restic-cache (sdb pool; cache reads are ARC-warm after first touch). tmpfs was considered and rejected: the cache exists to avoid network metadata re-fetches; tmpfs loses it at reboot and eats the RAM the ARC needs.

Concurrency & compression

• --read-concurrency 8 (default 2): more outstanding reads lets the HDD elevator help on seek-bound trees (our 275-IOPS wall).
• -o s3.connections=8 (default 5): mild upload parallelism bump.
• Compression: repo-v2 default auto is right.

Excludes — the cheapest optimisation of all

Nextcloud's appdata_*/preview (and dav-photocache) are regenerable caches holding a large fraction of the 500k files:

# /root/scripts/restic-exclude
/var/lib/incus/storage-pools/data/containers/nextcloud/rootfs/nextcloud/data/appdata_*/preview
/var/lib/incus/storage-pools/data/containers/nextcloud/rootfs/nextcloud/data/appdata_*/dav-photocache

Verify before the seed: find .../data/appdata_* -path '*/preview/*' | wc -l for the win size, then restic backup --dry-run -vv --exclude-file … prints every decision without uploading. Per-user <user>/cache/ dirs are a further candidate. Seafile: no excludes — the block store is the data.

Escape hatch: rustic

rustic speaks the same repository format — switching later is a binary swap on the same repos (decided 2026-08-25: restic first, rustic if performance issues rise; manual upgrades accepted). It would bring lock-free prune and a lower RAM footprint. Don't run both concurrently on one repo (rustic ignores restic's locks); a clean switchover is fine.

3. Install + init (root on ks4)

Debian 13 ships restic 0.18.0, which misses fixes we specifically need (checked 2026-08-25 against upstream changelogs): the 0.18.1 crash fix for directories deleted mid-walk (#5421 — our live trees do that nightly), 0.19.0's much faster index loading (#5713 — the merged restic-data repo loads its index every op) and lower check memory (#5610), index-repair robustness (#21827/28), clean exit codes (#4467/#5363 — our drivers key off rc), and 0.19.1's skip-inaccessible-source fix (#5667 — an unmounted source must not commit an empty "successful" snapshot). → install upstream 0.19.1; future upgrades via restic self-update (signature-verified).

apt install sshfs bzip2       # sshfs: needed by `incus file mount` (§6)
curl -LO https://github.com/restic/restic/releases/download/v0.19.1/restic_0.19.1_linux_amd64.bz2
bunzip2 restic_0.19.1_linux_amd64.bz2
install -m755 restic_0.19.1_linux_amd64 /usr/local/bin/restic
restic version
head -c 32 /dev/urandom | base64 > /root/.restic-passphrase && chmod 600 /root/.restic-passphrase
# ⚠️ password manager, NOW.

cat > /root/.restic-env <<'EOF'
export AWS_ACCESS_KEY_ID=<AK>
export AWS_SECRET_ACCESS_KEY=<SK>
export RESTIC_PASSWORD_FILE=/root/.restic-passphrase
export RESTIC_CACHE_DIR=/backup/restic-cache
EOF
chmod 600 /root/.restic-env && . /root/.restic-env && mkdir -p /backup/restic-cache

for r in restic-data restic-incus; do
    restic -r s3:s3.sbg.io.cloud.ovh.net/$r init
done

4. Seed plan (executed 2026-08-25 → 08-28)

1. ARC floor first (helps every walk on the box):

   echo 8589934592 > /sys/module/zfs/parameters/zfs_arc_min
   echo "options zfs zfs_arc_min=8589934592" > /etc/modprobe.d/zfs-arc.conf
   update-initramfs -u
   awk '/^c_min/ {printf "%.1f GiB\n", $3/1073741824}' /proc/spl/kstat/zfs/arcstats
   

   ⚠️ the floor is defended even under pressure — real headroom is ~31 G − services − 8 G; revisit before adding a fat service.
2. restic-data, staged (all into the same repo; separate invocations so each stage is independently restartable — restic-backup.sh -s runs the same thing):
   • dumps + 8 small sources (~50 G — an hour)
   • nextcloud path with the exclude file (proven ~22 MiB/s from the live pool → ~8 h, screen)
   • seafile — the one-time debt: 1–2 days, restartable at any point (committed packs dedup on retry)
3. restic-incus (16 small replicas via §6 — a few hours).

5. Nightly driver — scripts/restic-backup.sh

Shape: flock, env file, loud logging. Phases:

1. dumps — incus DB dumps + auto-discovered MariaDB/PostgreSQL (native and docker) dumps into the dump dir (during the parallel week restic dumps into its own /backup/dumps; carry the archived geo.sql over once with cp -a).
2. one backup invocation into restic-data: dump dir + all fs paths from /root/scripts/restic-paths (--files-from-verbatim), nextcloud excludes applied globally (patterns only match nextcloud paths). One index load, one snapshot per night.
3. retention — restic forget --group-by host --keep-daily 14 --keep-weekly 8 --keep-monthly 6 (instant without --prune; --group-by host because seed-era snapshots have different path sets than nightly ones and must age in one group).

The instances leg (restic-incus-backup.sh, §6) is shelved — nothing chains onto the 01:00 line today.

0 5 * * * /root/scripts/restic-backup.sh >> /var/log/restic-backup.log 2>&1

⚠️ cron gotcha (bit us 2026-08-28): cron's PATH lacks /usr/local/bin, so the scripts call /usr/local/bin/restic by absolute path.

Weekly maintenance (Sunday; prune takes an exclusive lock — never overlap the 05:00 run):

0 14 * * 0 /root/scripts/restic-maintenance.sh >> /var/log/restic-maintenance.log 2>&1

Per repo: restic prune --max-unused 10% --max-repack-size 4G --pack-size 64 — our dead-data rate is ~0.1–0.4 GiB/day against ~700 GiB repos (seafile append-mostly ≈ nothing dies), so 10 % tolerated slack lets prune skip repacking for months, and the 4 G cap bounds any single Sunday to ~10–15 min even after a mass deletion. Then restic check (structure, cheap) + check --read-data-subset=1/52 rotating — full data verification of every byte once a year, ~25 GiB read per week.

6. Instances without a plugin — incus file mount (SHELVED)

Decided 2026-08-28: not deployed. Instances are replicated (sdb, nuc after FTTH, ks2 in the meantime) and everything needed to rebuild one is in restic-data, so a third copy of their filesystems was judged not worth the nightly moving parts. The design and the script remain, ready if that changes. If the restic-incus bucket is ever deleted, drop it from REPOS in restic-maintenance.sh too.

Neither restic nor rustic accepts source plugins, and forking the backup tool is the wrong place to carry a patch. Instead, incus exposes any instance (including stopped replicas) as a FUSE mount over the per-instance sftp API:

for inst in $(incus list --project backup -c n -f csv); do
    case ",$EXCLUDE_INSTANCES," in *",$inst,"*) continue;; esac   # nextcloud,seafile
    mnt=/run/restic-incus/$inst          # per-instance mountpoint: restic
    mkdir -p "$mnt"                      # picks parents by host+path, so
    incus config show "$inst" --project backup --expanded > "$mnt.yaml"
    incus file mount "$inst/" "$mnt" --project backup &   # foreground proc
    MPID=$!; sleep 2
    restic -r s3:…/restic-incus backup --pack-size 64 --tag "$inst" "$mnt" "$mnt.yaml"
    kill $MPID; wait $MPID 2>/dev/null; umount "$mnt" 2>/dev/null
done

⚠️ per-instance mountpoints are load-bearing: a shared mountpoint would make every snapshot's parent the previous other instance's snapshot (restic matches parents by host+path) → nightly full re-reads. Distinct stable paths give correct parents and stat-skip. Mount lifecycle hardened in restic-incus-backup.sh (incus file mount runs foreground; needs sshfs). All instances by default, opt-out via a variable — the no-manifest-drift rule survives the tool change. Chained after the 01:00 incus-copy once validated, same freshness reasoning as before.

One shared repo would dedup the Ubuntu base across all replicas. Restore granularity: per-file; instance definitions ride the dump tree (incus/<inst>.yaml + incus-global-db.sql).

7. Restore drill (gate, then quarterly)

Scripted — restic-restore-test.sh:

/root/scripts/restic-restore-test.sh        # tree + dump, ~2 min
/root/scripts/restic-restore-test.sh -d     # also loads the dump into a
                                            # throwaway container (strongest)

It restores a filesystem tree from the latest snapshot and diffs it against the live one, restores a database dump and checks it is well-formed (CREATE TABLE count + Dump completed marker), then cleans up (-k keeps the files, -w changes the work dir). Never restore into /tmp — it is a 16 GiB tmpfs on ks4; the script writes under /backup.

A backup leg that has never been restored from is a hypothesis, not a backup: run this at the cutover gate and once a quarter after.

8. Cutover (done) + what is left

Legacy on the box: plakar is still installed and its plakar-data bucket still exists, kept only to reproduce #2338 if upstream asks; no cron references it. The empty plakar-incus bucket can be deleted.

• Cutover 2026-08-28: the previous S3 leg's crons are off and restic's 05:00 run is the S3 backup. The formal parallel-week comparison was cut short — the evidence had already settled it (predecessor: ~800 MiB metadata re-read and 3–4 h per night on the nextcloud tree, churn-independent; restic: seed at 60–115 MiB/s, local-cache incrementals). Details in plakar-s3-data.md.
• Done 2026-08-28: initial sync (1,114,212 files / 1.239 TiB in 4h10m, only 2.4 GiB added thanks to dedup + the seafile GC), restore drill passed (§7), interim ks2 push re-enabled.
• Steady state, measured 2026-08-29 (first full nightly run): 1,114,343 files / 1.239 TiB walked in 1 min 43 s, 201 MiB added (63 MiB stored), rc=0. For comparison, the previous S3 tool needed 3–4 h per night for one of those trees. This is the local-cache cost model working as intended: the walk is stat-only, and only the churn is read, chunked and uploaded.
• First maintenance run 2026-08-30: prune 0 blobs / 0 B removed, check and the rotating --read-data-subset slice both clean, 7 min, rc=0. Nothing to reclaim because the killed seed's packs were adopted by the successful seed's dedup rather than orphaned — worth knowing before assuming an interrupted backup wastes storage.
• Left: the nuc leg after FTTH, then the ks2 decommission (../ks2/plan.md). The instance leg (§6) is shelved, not pending.
• Old tool: see the legacy note above; the integration-incus repo retires from production (optionally still published to PlakarKorp/hub as a community contribution).
• Interim ks2 push (until the nuc leg seeds after FTTH, expected before end of September): re-enable the replica push so instances keep an off-site copy meanwhile:

  0 2 * * * /root/scripts/incus-copy.sh -d ks2 -m push >> /var/log/incus-copy.log 2>&1