229 lines
9.8 KiB
Markdown
229 lines
9.8 KiB
Markdown
# restic backups on ks4 — the giant fs trees
|
||
|
||
Status: **drafted 2026-08-25, not yet deployed.**
|
||
|
||
Why restic exists in this stack: plakar's incremental cost scales
|
||
with tree size, not churn (~800 MiB of parent metadata re-read from
|
||
S3 + hours of CPU per night on a 500k-file tree — measured, reported
|
||
as [PlakarKorp/plakar#2338](https://github.com/PlakarKorp/plakar/issues/2338)),
|
||
which priced `nextcloud-fs` and `seafile-fs` out of the plakar data
|
||
leg. restic keeps its comparison state in a **local cache**, so a
|
||
night costs a local stat-walk plus the churn — the rsync cost model.
|
||
plakar keeps everything it is good at (dumps, small sources, small
|
||
instances); restic covers only the two giants until #2338 is
|
||
resolved (see §7).
|
||
|
||
Scope:
|
||
|
||
| source | size | files | churn/night (measured) |
|
||
|---|---|---|---|
|
||
| `/nextcloud` (live, nextcloud container) | 737 G | ~500 k | 90–430 MiB |
|
||
| `/opt/seafile` (live, seafile container) | 933 G | millions of ~100 KiB block files | append-mostly |
|
||
|
||
Read from the live rootfs paths on the host (crash-consistent, like
|
||
the plakar fs sources; DBs are covered by the dump leg).
|
||
|
||
## 1. Repository layout — one bucket, two repos
|
||
|
||
Two **separate repositories** under one bucket
|
||
(`s3:s3.sbg.io.cloud.ovh.net/restic/nextcloud` and `.../restic/seafile`):
|
||
|
||
- restic loads the whole repo index into RAM per operation; the two
|
||
datasets share no content, so a shared repo only inflates every
|
||
run's memory for zero dedup gain.
|
||
- prune/check run per repo — seafile's heavy maintenance never blocks
|
||
or bloats nextcloud's.
|
||
- one bucket keeps credential handling identical to the plakar legs.
|
||
|
||
Same passphrase discipline as plakar: generate to
|
||
`/root/.restic-passphrase` (mode 600), **copy to the password manager
|
||
immediately**.
|
||
|
||
## 2. Option analysis (our numbers, not defaults-worship)
|
||
|
||
### `--pack-size` (default 16 MiB, max 128)
|
||
|
||
Packs are the S3 objects; blobs (~1 MiB avg for data) live inside
|
||
them. Pack size does **not** change index RAM (that scales with blob
|
||
count) — it changes object count, request count, and rewrite
|
||
amplification:
|
||
|
||
| pack-size | S3 objects for ~1.2 T stored | trade-off |
|
||
|---|---|---|
|
||
| 16 MiB | ~75 000 | slow `check`/listing, more requests; prune rewrites are fine-grained |
|
||
| **64 MiB (chosen)** | ~19 000 | 4× fewer objects/requests; assembly RAM ≈ pack×connections ≈ 64 M × 5 ≈ 320 MiB — fine |
|
||
| 128 MiB | ~9 500 | fewer still, but prune rewrite amplification doubles (a pack with one dead blob rewrites 128 MiB) and upload buffers grow |
|
||
|
||
`--pack-size 64` on **every backup/prune invocation** (it is not a
|
||
repo property).
|
||
|
||
### Memory (the real constraint on this 23/31 GiB-used box)
|
||
|
||
- Index RAM ≈ 200–300 B/blob. Estimate: nextcloud ~1 M blobs,
|
||
seafile ~2–4 M (tiny files = ≥1 blob each) → **~0.3 GiB and
|
||
~0.6–1.2 GiB** respectively per backup run; prune peaks 2–3×.
|
||
This is why the repos are split (§1).
|
||
- `GOGC=20` env trades CPU for a substantially smaller Go heap —
|
||
enable it if the seafile prune ever pressures the box.
|
||
|
||
### Cache location
|
||
|
||
`~/.cache/restic` would land on `/` (14 G free) and can reach a few
|
||
GiB for repos this size → `--cache-dir /backup/restic-cache` (sdb
|
||
pool, plenty of room; cache reads are small but ARC-warm after first
|
||
use).
|
||
|
||
### Concurrency
|
||
|
||
- `--read-concurrency 8` (default 2): more outstanding reads lets the
|
||
HDD elevator help on the seek-bound seafile tree — the one knob that
|
||
directly targets our 275-IOPS wall.
|
||
- `-o s3.connections=8` (default 5): mild upload parallelism bump;
|
||
OVH same-region sustains it.
|
||
|
||
### Compression
|
||
|
||
Repo format v2 default `auto` is right; `max` only buys ~% on
|
||
already-compressed content (seafile blocks, images) for real CPU.
|
||
|
||
### Excludes — the cheapest optimisation of all
|
||
|
||
Nextcloud's `appdata_*/preview` (and `dav-photocache`) are
|
||
**regenerable caches holding a large fraction of the 500k files**;
|
||
excluding them shrinks walk and churn dramatically:
|
||
|
||
```
|
||
# /root/scripts/restic-nextcloud-exclude
|
||
/var/lib/incus/storage-pools/data/containers/nextcloud/rootfs/nextcloud/data/appdata_*/preview
|
||
/var/lib/incus/storage-pools/data/containers/nextcloud/rootfs/nextcloud/data/appdata_*/dav-photocache
|
||
```
|
||
|
||
Seafile: no excludes — the block store *is* the data.
|
||
|
||
Verify excludes before the seed: count the win with
|
||
`find .../appdata_* -path '*/preview/*' | wc -l`, then
|
||
`restic backup --dry-run -vv --exclude-file ...` prints every
|
||
include/exclude decision without uploading. (Per-user `<user>/cache/`
|
||
dirs are a further regenerable candidate if the count justifies it.)
|
||
|
||
Cache on tmpfs was considered and rejected: the cache exists to avoid
|
||
*network* metadata re-fetches; tmpfs loses it at reboot (forcing a
|
||
full re-download), eats the RAM the ARC needs, and buys nothing —
|
||
disk-backed cache reads are ARC-warm after first touch anyway.
|
||
|
||
## 3. Install + init (root on ks4)
|
||
|
||
```sh
|
||
apt install restic # Debian 13 ships ≥0.17 (repo v2, compression, pack-size ok)
|
||
head -c 32 /dev/urandom | base64 > /root/.restic-passphrase && chmod 600 /root/.restic-passphrase
|
||
# ⚠️ password manager, NOW.
|
||
|
||
export AWS_ACCESS_KEY_ID=<AK> AWS_SECRET_ACCESS_KEY=<SK> # -> /root/.restic-env (mode 600)
|
||
export RESTIC_PASSWORD_FILE=/root/.restic-passphrase
|
||
restic -r s3:s3.sbg.io.cloud.ovh.net/restic/nextcloud init
|
||
restic -r s3:s3.sbg.io.cloud.ovh.net/restic/seafile init
|
||
mkdir -p /backup/restic-cache
|
||
```
|
||
|
||
## 4. Seed plan
|
||
|
||
Order matters:
|
||
|
||
1. **Bump the ARC floor first** so metadata walks stay warm between
|
||
nightly runs (also helps the plakar legs):
|
||
```sh
|
||
echo 8589934592 > /sys/module/zfs/parameters/zfs_arc_min
|
||
echo "options zfs zfs_arc_min=8589934592" > /etc/modprobe.d/zfs-arc.conf
|
||
update-initramfs -u # zfs loads from the initramfs
|
||
awk '/^c_min/ {printf "%.1f GiB\n", $3/1073741824}' /proc/spl/kstat/zfs/arcstats
|
||
```
|
||
⚠️ the floor is defended even under memory pressure — real
|
||
headroom becomes ~31G − services − 8G; revisit before adding any
|
||
fat new service.
|
||
2. **nextcloud** (proven ~22 MiB/s from the live pool → ~8 h), in a
|
||
screen:
|
||
```sh
|
||
restic -r s3:.../restic/nextcloud --cache-dir /backup/restic-cache \
|
||
backup --pack-size 64 --read-concurrency 8 -o s3.connections=8 \
|
||
--exclude-file /root/scripts/restic-nextcloud-exclude \
|
||
/var/lib/incus/storage-pools/data/containers/nextcloud/rootfs/nextcloud
|
||
```
|
||
3. **seafile** — the one-time debt. Expect 1–2 days on the live pool
|
||
(warmer than sdb, and `--read-concurrency 8` helps); it is
|
||
restartable at any point (committed packs dedup on retry).
|
||
|
||
## 5. Nightly runs + retention + verification
|
||
|
||
Cron (05:00 — after the 04:30 plakar leg's dump phase, sources don't
|
||
overlap; wrap in a `restic-backup.sh` with flock + env, same style as
|
||
the plakar drivers):
|
||
|
||
```cron
|
||
0 5 * * * /root/scripts/restic-backup.sh >> /var/log/restic-backup.log 2>&1
|
||
```
|
||
|
||
Per repo, the script does:
|
||
|
||
```sh
|
||
restic backup --pack-size 64 --read-concurrency 8 ... # as in §4
|
||
restic forget --keep-daily 14 --keep-weekly 8 --keep-monthly 6
|
||
restic unlock 2>/dev/null # clear stale locks after crashes
|
||
```
|
||
|
||
- `forget` without `--prune` is instant; actual **prune runs weekly**
|
||
(Sunday, offset from plakar's check; prune takes an exclusive lock —
|
||
never overlap the 05:00 backups):
|
||
```cron
|
||
0 14 * * 0 ... restic prune --max-unused 10% --max-repack-size 4G --pack-size 64 ...
|
||
```
|
||
Tuning rationale (our churn: ~0.1–0.4 GiB/day dead data vs ~700 GiB
|
||
repos, seafile append-mostly ≈ nothing dies): `--max-unused 10%`
|
||
lets prune skip repacking for months, carrying ≤10 % slack in cheap
|
||
object storage; `--max-repack-size 4G` bounds any single Sunday's
|
||
rewrite to ~10–15 min even after a mass deletion, deferring the
|
||
rest to following weeks. (`--repack-small` only matters for repos
|
||
with legacy sub-64 MiB packs — not us, we start at 64.)
|
||
- **Verification**: weekly `restic check` (structure only, cheap) +
|
||
`check --read-data-subset=1/52` rotating — a full data verification
|
||
of every byte once a year, ~25 GiB read per week, for free.
|
||
|
||
Expected steady state: nightly = local stat-walk (minutes once ARC is
|
||
warm) + churn upload; no per-run S3 metadata storm — the exact
|
||
property #2338 is about.
|
||
|
||
## 6. Restore test (gate before trusting it)
|
||
|
||
```sh
|
||
restic -r s3:.../restic/nextcloud --cache-dir /backup/restic-cache \
|
||
restore latest --target /backup/restore-test --include '/<some subtree>'
|
||
diff -r ... && rm -rf /backup/restore-test
|
||
```
|
||
|
||
Never restore to `/tmp` — it is a 16 GiB tmpfs on ks4.
|
||
|
||
## 7. If restic ever takes over ALL the S3 backups
|
||
|
||
Decision gate: **the upstream answer to #2338.** Two futures:
|
||
|
||
- **plakar fixes the growth problem** → consolidate the giants *back*
|
||
onto plakar, retire restic (delete the `restic` bucket) — one tool,
|
||
and the incus integration investment keeps compounding.
|
||
- **No fix on a useful horizon** → migrate the data leg to restic:
|
||
- dumps + 8 small fs sources are ~50 G total: re-seeding into a
|
||
`restic/data` repo is an evening, and restic dedups the nightly
|
||
dump tree (incl. the static `geo.sql`) exactly like plakar does.
|
||
- **the incus leg is the sticking point**: the per-file instance
|
||
backups ride our plakar-native integration. restic equivalents
|
||
are all worse or heavier: backing up replica rootfs paths
|
||
directly (fights incus's on-demand mounts), `incus export`
|
||
tarballs (server-side materialization, no room), or `zfs send`
|
||
streams into restic's stdin mode (works — `restic backup --stdin`
|
||
— but chains restore through zfs+incus knowledge).
|
||
- So a full migration really means "restic for all *file* data +
|
||
keep plakar only for the incus leg" — two tools either way.
|
||
Which is the same operational cost as today's split, minus the
|
||
#2338 exposure. Conclusion: **full migration buys little; the
|
||
current split is not a temporary hack but a reasonable end
|
||
state.** Revisit only if plakar becomes unmaintained or the
|
||
incus leg moves to zfs send anyway.
|