doc/ks4/restic-backup.md

restic backups on ks4 — the S3 leg

Status: live since 2026-08-28 — restic replaced plakar as the S3 backup tool (plakar crons off, its kloset kept for #2338; see §8). Architecture: two tools — incus copy for replication (local sdb, nuc after FTTH), restic for everything S3. Data leg (restic-data) runs nightly at 05:00. The instance leg (restic-incus, §6) is shelved (2026-08-28): instances are covered by replication, so duplicating their filesystems into S3 buys little. The script and the (empty) repo stay in place should that change.

Why: plakar's incremental cost scales with tree size, not churn (~800 MiB parent metadata re-read from S3 + hours of CPU per night on a 500k-file tree — measured, reported as PlakarKorp/plakar#2338). restic keeps its comparison state in a local cache: a night costs a local stat-walk plus the churn — the rsync cost model. Neither restic nor rustic has plakar's source-plugin architecture, but none is needed: instances are reached via incus file mount (FUSE over the same per-instance sftp API our plakar integration used) — zero patches to maintain (see §6).

1. Repositories — two buckets, two drivers (decided 2026-08-25)

bucket = repo	contents	driver script
restic-data	SQL dumps + all fs sources (8 small + nextcloud + seafile)	restic-backup.sh
restic-incus	(shelved — empty repo, see §6) the backup-project replicas, per-file via mount	restic-incus-backup.sh

Mirrors the plakar-era split (data leg / incus leg). Accepted trade-off: restic-data carries the giants' blobs (~3–5 M once seafile is in), so every operation on it loads a ~1.5–2 GiB index — mitigated by running the whole nightly as a single restic backup invocation (one index load, one snapshot covering all paths), GOGC=20 in reserve, rustic as the escape hatch. Buckets (not prefixes) for per-bucket OVH metrics and independent versioning/lock decisions later.

Passphrase: /root/.restic-passphrase (mode 600), one for all repos, copy to the password manager immediately. S3 creds + env in /root/.restic-env (mode 600).

2. Option analysis (our numbers, not defaults-worship)

--pack-size (default 16 MiB, max 128)

Packs are the S3 objects; blobs (~1 MiB avg for data) live inside them. Pack size does not change index RAM (that scales with blob count) — it changes object count, request count, and rewrite amplification:

pack-size	S3 objects for ~1.2 T stored	trade-off
16 MiB	~75 000	slow check/listing, more requests; prune rewrites are fine-grained
64 MiB (chosen)	~19 000	4× fewer objects/requests; assembly RAM ≈ pack×connections ≈ 64 M × 5 ≈ 320 MiB — fine
128 MiB	~9 500	fewer still, but prune rewrite amplification doubles and upload buffers grow

--pack-size 64 on every backup/prune invocation (not a repo property).

Memory (the real constraint on this 23/31 GiB-used box)

• Index RAM ≈ 200–300 B/blob → combined restic-data index ~1 GiB per operation; prune peaks 2–3×. The seafile side of that estimate collapsed after its garbage collection (24.9 M → 0.5 M objects, seafile-gc.md). Measure peak RSS with /usr/bin/time -v if the box ever feels tight.
• GOGC=20 trades CPU for a smaller Go heap — enable if the seafile prune ever pressures the box.

Cache location

~/.cache/restic would land on / (14 G free) and can reach a few GiB at this scale → --cache-dir /backup/restic-cache (sdb pool; cache reads are ARC-warm after first touch). tmpfs was considered and rejected: the cache exists to avoid network metadata re-fetches; tmpfs loses it at reboot and eats the RAM the ARC needs.

Concurrency & compression

• --read-concurrency 8 (default 2): more outstanding reads lets the HDD elevator help on seek-bound trees (our 275-IOPS wall).
• -o s3.connections=8 (default 5): mild upload parallelism bump.
• Compression: repo-v2 default auto is right.

Excludes — the cheapest optimisation of all

Nextcloud's appdata_*/preview (and dav-photocache) are regenerable caches holding a large fraction of the 500k files:

# /root/scripts/restic-exclude
/var/lib/incus/storage-pools/data/containers/nextcloud/rootfs/nextcloud/data/appdata_*/preview
/var/lib/incus/storage-pools/data/containers/nextcloud/rootfs/nextcloud/data/appdata_*/dav-photocache

Verify before the seed: find .../data/appdata_* -path '*/preview/*' | wc -l for the win size, then restic backup --dry-run -vv --exclude-file … prints every decision without uploading. Per-user <user>/cache/ dirs are a further candidate. Seafile: no excludes — the block store is the data.

Escape hatch: rustic

rustic speaks the same repository format — switching later is a binary swap on the same repos (decided 2026-08-25: restic first, rustic if performance issues rise; manual upgrades accepted). It would bring lock-free prune and a lower RAM footprint. Don't run both concurrently on one repo (rustic ignores restic's locks); a clean switchover is fine.

3. Install + init (root on ks4)

Debian 13 ships restic 0.18.0, which misses fixes we specifically need (checked 2026-08-25 against upstream changelogs): the 0.18.1 crash fix for directories deleted mid-walk (#5421 — our live trees do that nightly), 0.19.0's much faster index loading (#5713 — the merged restic-data repo loads its index every op) and lower check memory (#5610), index-repair robustness (#21827/28), clean exit codes (#4467/#5363 — our drivers key off rc), and 0.19.1's skip-inaccessible-source fix (#5667 — an unmounted source must not commit an empty "successful" snapshot). → install upstream 0.19.1; future upgrades via restic self-update (signature-verified).

apt install sshfs bzip2       # sshfs: needed by `incus file mount` (§6)
curl -LO https://github.com/restic/restic/releases/download/v0.19.1/restic_0.19.1_linux_amd64.bz2
bunzip2 restic_0.19.1_linux_amd64.bz2
install -m755 restic_0.19.1_linux_amd64 /usr/local/bin/restic
restic version
head -c 32 /dev/urandom | base64 > /root/.restic-passphrase && chmod 600 /root/.restic-passphrase
# ⚠️ password manager, NOW.

cat > /root/.restic-env <<'EOF'
export AWS_ACCESS_KEY_ID=<AK>
export AWS_SECRET_ACCESS_KEY=<SK>
export RESTIC_PASSWORD_FILE=/root/.restic-passphrase
export RESTIC_CACHE_DIR=/backup/restic-cache
EOF
chmod 600 /root/.restic-env && . /root/.restic-env && mkdir -p /backup/restic-cache

for r in restic-data restic-incus; do
    restic -r s3:s3.sbg.io.cloud.ovh.net/$r init
done

4. Seed plan (executed 2026-08-25 → 08-28)

1. ARC floor first (helps every walk on the box):

   echo 8589934592 > /sys/module/zfs/parameters/zfs_arc_min
   echo "options zfs zfs_arc_min=8589934592" > /etc/modprobe.d/zfs-arc.conf
   update-initramfs -u
   awk '/^c_min/ {printf "%.1f GiB\n", $3/1073741824}' /proc/spl/kstat/zfs/arcstats
   

   ⚠️ the floor is defended even under pressure — real headroom is ~31 G − services − 8 G; revisit before adding a fat service.
2. restic-data, staged (all into the same repo; separate invocations so each stage is independently restartable — restic-backup.sh -s runs the same thing):
   • dumps + 8 small sources (~50 G — an hour)
   • nextcloud path with the exclude file (proven ~22 MiB/s from the live pool → ~8 h, screen)
   • seafile — the one-time debt: 1–2 days, restartable at any point (committed packs dedup on retry)
3. restic-incus (16 small replicas via §6 — a few hours).

5. Nightly driver — scripts/restic-backup.sh

Same shape as the plakar drivers (flock, env, loud logging). Phases:

1. dumps — lifted verbatim from plakar-backup.sh: incus DB dumps + auto-discovered MariaDB/PostgreSQL (native and docker) dumps into the dump dir (during the parallel week restic dumps into its own /backup/dumps; carry the archived geo.sql over once with cp -a).
2. one backup invocation into restic-data: dump dir + all fs paths from /root/scripts/restic-paths (--files-from-verbatim), nextcloud excludes applied globally (patterns only match nextcloud paths). One index load, one snapshot per night.
3. retention — restic forget --group-by host --keep-daily 14 --keep-weekly 8 --keep-monthly 6 (instant without --prune; --group-by host because seed-era snapshots have different path sets than nightly ones and must age in one group).

The instances leg (restic-incus-backup.sh, §6) is shelved — nothing chains onto the 01:00 line today.

0 5 * * * /root/scripts/restic-backup.sh >> /var/log/restic-backup.log 2>&1

⚠️ cron gotcha (bit us 2026-08-28): cron's PATH lacks /usr/local/bin, so the scripts call /usr/local/bin/restic by absolute path.

Weekly maintenance (Sunday; prune takes an exclusive lock — never overlap the 05:00 run):

0 14 * * 0 /root/scripts/restic-maintenance.sh >> /var/log/restic-maintenance.log 2>&1

Per repo: restic prune --max-unused 10% --max-repack-size 4G --pack-size 64 — our dead-data rate is ~0.1–0.4 GiB/day against ~700 GiB repos (seafile append-mostly ≈ nothing dies), so 10 % tolerated slack lets prune skip repacking for months, and the 4 G cap bounds any single Sunday to ~10–15 min even after a mass deletion. Then restic check (structure, cheap) + check --read-data-subset=1/52 rotating — full data verification of every byte once a year, ~25 GiB read per week.

6. Instances without a plugin — incus file mount (SHELVED)

Decided 2026-08-28: not deployed. Instances are replicated (sdb, nuc after FTTH, ks2 in the meantime) and everything needed to rebuild one is in restic-data, so a third copy of their filesystems was judged not worth the nightly moving parts. The design and the script remain, ready if that changes. If the restic-incus bucket is ever deleted, drop it from REPOS in restic-maintenance.sh too.

Neither restic nor rustic accepts source plugins, and forking the backup tool is the wrong place to carry a patch. Instead, incus exposes any instance (including stopped replicas) as a FUSE mount over the same per-instance sftp API the plakar integration used:

for inst in $(incus list --project backup -c n -f csv); do
    case ",$EXCLUDE_INSTANCES," in *",$inst,"*) continue;; esac   # nextcloud,seafile
    mnt=/run/restic-incus/$inst          # per-instance mountpoint: restic
    mkdir -p "$mnt"                      # picks parents by host+path, so
    incus config show "$inst" --project backup --expanded > "$mnt.yaml"
    incus file mount "$inst/" "$mnt" --project backup &   # foreground proc
    MPID=$!; sleep 2
    restic -r s3:…/restic-incus backup --pack-size 64 --tag "$inst" "$mnt" "$mnt.yaml"
    kill $MPID; wait $MPID 2>/dev/null; umount "$mnt" 2>/dev/null
done

⚠️ per-instance mountpoints are load-bearing: a shared mountpoint would make every snapshot's parent the previous other instance's snapshot (restic matches parents by host+path) → nightly full re-reads. Distinct stable paths give correct parents and stat-skip. Mount lifecycle hardened in restic-incus-backup.sh (incus file mount runs foreground; needs sshfs). All instances by default, opt-out via a variable — the no-manifest-drift rule survives the tool change. Chained after the 01:00 incus-copy once validated, same freshness reasoning as before.

Bonus over plakar: one shared repo dedups the Ubuntu base across all 16 replicas. Restore granularity: per-file; instance definitions ride the dump tree (incus/<inst>.yaml + incus-global-db.sql).

7. Restore test (gate)

Per repo, one restore + diff (never to /tmp — 16 GiB tmpfs):

restic -r s3:…/restic-data restore latest --target /backup/restore-test --include '<subtree>'
diff -r … && rm -rf /backup/restore-test

8. Cutover (done) + what is left

• Cutover 2026-08-28: plakar's 04:30 and Sunday crons are off; restic's 05:00 data leg is the S3 backup. The formal parallel-week comparison was cut short — the evidence had already settled it (plakar: ~800 MiB metadata re-read and 3–4 h per night on the nextcloud tree, churn-independent; restic: seed at 60–115 MiB/s, local-cache incrementals). plakar's measurements stay in plakar-s3-data.md.
• Left to do: restore test on the complete restic-data; re-enable the interim ks2 push until FTTH; then the ks2 decommission gates (../ks2/plan.md). The instance leg (§6) is shelved, not pending.
• plakar retirement, one exception: keep plakar installed and keep the plakar-data kloset until #2338 concludes — the issue offers debug runs against that repository. Delete the plakar-incus bucket immediately; plakar-data a few weeks later. The integration-incus repo retires from production; optionally still publish to PlakarKorp/hub as a community contribution.
• Interim ks2 push (until the nuc leg seeds after FTTH, expected before end of September): re-enable the replica push so instances keep an off-site copy meanwhile:

  0 2 * * * /root/scripts/incus-copy.sh -d ks2 -m push >> /var/log/incus-copy.log 2>&1