julien/doc
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

restic-backup: restic becomes THE S3 tool — full rewrite

Browse Source 
Four buckets (data/nextcloud/seafile/instances), instances leg via
incus file mount (no plugin architecture in restic/rustic — mount the
source instead of forking the tool), nightly driver spec, parallel
plakar comparison week, plakar retirement plan (kloset kept for
PlakarKorp/plakar#2338 debugging), interim ks2 replica push until the
FTTH/nuc leg.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Julien Lutran
2026-08-25 12:44:48 +02:00
co-authored by Claude Fable 5
parent 3dec0c0a7b
commit fb410259c5
1 changed files with 166 additions and 159 deletions
Download Patch File Download Diff File Expand all files Collapse all files
ks4/restic-backup.md
+159 -152
View File
@@ -1,43 +1,40 @@
# restic backups on ks4 — the giant fs trees # restic backups on ks4 — the S3 leg
Status: **drafted 2026-08-25, not yet deployed.** Status: **decided 2026-08-25 — restic replaces plakar as the S3
backup tool** (evaluation first: parallel run + comparison, see §8).
Final architecture: **two tools** — `incus copy` for replication
(local sdb, nuc after FTTH, ks2 push as interim), **restic** for
everything S3.
Why restic exists in this stack: plakar's incremental cost scales Why: plakar's incremental cost scales with tree size, not churn
with tree size, not churn (~800 MiB of parent metadata re-read from (~800 MiB parent metadata re-read from S3 + hours of CPU per night on
S3 + hours of CPU per night on a 500k-file tree — measured, reported a 500k-file tree — measured, reported as
as [PlakarKorp/plakar#2338](https://github.com/PlakarKorp/plakar/issues/2338)), [PlakarKorp/plakar#2338](https://github.com/PlakarKorp/plakar/issues/2338)).
which priced `nextcloud-fs` and `seafile-fs` out of the plakar data restic keeps its comparison state in a local cache: a night costs a
leg. restic keeps its comparison state in a **local cache**, so a local stat-walk plus the churn — the rsync cost model. Neither restic
night costs a local stat-walk plus the churn — the rsync cost model. nor rustic has plakar's source-plugin architecture, but none is
plakar keeps everything it is good at (dumps, small sources, small needed: instances are reached via `incus file mount` (FUSE over the
instances); restic covers only the two giants until #2338 is same per-instance sftp API our plakar integration used) — zero
resolved (see §7). patches to maintain (see §6).
Scope: ## 1. Repositories — one bucket per repo
| source | size | files | churn/night (measured) | | bucket = repo | contents | est. size | blobs (index RAM driver) |
|---|---|---|---| |---|---|---|---|
| `/nextcloud` (live, nextcloud container) | 737 G | ~500 k | 90–430 MiB | | `restic-data` | `/backup/plakar-dumps` dump tree + the 8 small fs sources | ~50 G | small |
| `/opt/seafile` (live, seafile container) | 933 G | millions of ~100 KiB block files | append-mostly | | `restic-nextcloud` | `/nextcloud` data (live rootfs path) | 737 G | ~1 M |
| `restic-seafile` | `/opt/seafile` (live rootfs path) | 933 G | ~2–4 M |
| `restic-instances` | the `backup`-project replicas, per-file via mount | ~16 small instances | medium |
Read from the live rootfs paths on the host (crash-consistent, like Separate repos because restic loads the whole repo index into RAM per
the plakar fs sources; DBs are covered by the dump leg). operation and prune/check are per-repo; separate **buckets** (not
prefixes) for per-bucket OVH usage metrics, independent
versioning/object-lock decisions later, and bucket-native credential
scoping. One credential pair across all four to start.
## 1. Repository layout — one bucket, two repos Passphrase: `/root/.restic-passphrase` (mode 600), one for all repos,
**copy to the password manager immediately**. S3 creds + env in
Two **separate repositories** under one bucket `/root/.restic-env` (mode 600).
(`s3:s3.sbg.io.cloud.ovh.net/restic/nextcloud` and `.../restic/seafile`):
- restic loads the whole repo index into RAM per operation; the two
datasets share no content, so a shared repo only inflates every
run's memory for zero dedup gain.
- prune/check run per repo — seafile's heavy maintenance never blocks
or bloats nextcloud's.
- one bucket keeps credential handling identical to the plakar legs.
Same passphrase discipline as plakar: generate to
`/root/.restic-passphrase` (mode 600), **copy to the password manager
immediately**.
## 2. Option analysis (our numbers, not defaults-worship) ## 2. Option analysis (our numbers, not defaults-worship)
@@ -52,45 +49,38 @@ amplification:
|---|---|---| |---|---|---|
| 16 MiB | ~75 000 | slow `check`/listing, more requests; prune rewrites are fine-grained | | 16 MiB | ~75 000 | slow `check`/listing, more requests; prune rewrites are fine-grained |
| **64 MiB (chosen)** | ~19 000 | 4× fewer objects/requests; assembly RAM ≈ pack×connections ≈ 64 M × 5 ≈ 320 MiB — fine | | **64 MiB (chosen)** | ~19 000 | 4× fewer objects/requests; assembly RAM ≈ pack×connections ≈ 64 M × 5 ≈ 320 MiB — fine |
| 128 MiB | ~9 500 | fewer still, but prune rewrite amplification doubles (a pack with one dead blob rewrites 128 MiB) and upload buffers grow | | 128 MiB | ~9 500 | fewer still, but prune rewrite amplification doubles and upload buffers grow |
`--pack-size 64` on **every backup/prune invocation** (it is not a `--pack-size 64` on **every backup/prune invocation** (not a repo
repo property). property).
### Memory (the real constraint on this 23/31 GiB-used box) ### Memory (the real constraint on this 23/31 GiB-used box)
- Index RAM ≈ 200–300 B/blob. Estimate: nextcloud ~1 M blobs, - Index RAM ≈ 200–300 B/blob → ~0.3 GiB (nextcloud) and
seafile ~2–4 M (tiny files = ≥1 blob each) → **~0.3 GiB and ~0.6–1.2 GiB (seafile) per backup run; prune peaks 2–3×. Hence the
~0.6–1.2 GiB** respectively per backup run; prune peaks 2–3×. repo split (§1).
This is why the repos are split (§1). - `GOGC=20` trades CPU for a smaller Go heap — enable if the seafile
- `GOGC=20` env trades CPU for a substantially smaller Go heap — prune ever pressures the box.
enable it if the seafile prune ever pressures the box.
### Cache location ### Cache location
`~/.cache/restic` would land on `/` (14 G free) and can reach a few `~/.cache/restic` would land on `/` (14 G free) and can reach a few
GiB for repos this size → `--cache-dir /backup/restic-cache` (sdb GiB at this scale → `--cache-dir /backup/restic-cache` (sdb pool;
pool, plenty of room; cache reads are small but ARC-warm after first cache reads are ARC-warm after first touch). tmpfs was considered and
use). rejected: the cache exists to avoid *network* metadata re-fetches;
tmpfs loses it at reboot and eats the RAM the ARC needs.
### Concurrency ### Concurrency & compression
- `--read-concurrency 8` (default 2): more outstanding reads lets the - `--read-concurrency 8` (default 2): more outstanding reads lets the
HDD elevator help on the seek-bound seafile tree — the one knob that HDD elevator help on seek-bound trees (our 275-IOPS wall).
directly targets our 275-IOPS wall. - `-o s3.connections=8` (default 5): mild upload parallelism bump.
- `-o s3.connections=8` (default 5): mild upload parallelism bump; - Compression: repo-v2 default `auto` is right.
OVH same-region sustains it.
### Compression
Repo format v2 default `auto` is right; `max` only buys ~% on
already-compressed content (seafile blocks, images) for real CPU.
### Excludes — the cheapest optimisation of all ### Excludes — the cheapest optimisation of all
Nextcloud's `appdata_*/preview` (and `dav-photocache`) are Nextcloud's `appdata_*/preview` (and `dav-photocache`) are
**regenerable caches holding a large fraction of the 500k files**; regenerable caches holding a large fraction of the 500k files:
excluding them shrinks walk and churn dramatically:
``` ```
# /root/scripts/restic-nextcloud-exclude # /root/scripts/restic-nextcloud-exclude
@@ -98,143 +88,160 @@ excluding them shrinks walk and churn dramatically:
/var/lib/incus/storage-pools/data/containers/nextcloud/rootfs/nextcloud/data/appdata_*/dav-photocache /var/lib/incus/storage-pools/data/containers/nextcloud/rootfs/nextcloud/data/appdata_*/dav-photocache
``` ```
Seafile: no excludes — the block store *is* the data. Verify before the seed: `find .../data/appdata_* -path '*/preview/*' | wc -l`
for the win size, then `restic backup --dry-run -vv --exclude-file …`
Verify excludes before the seed: count the win with prints every decision without uploading. Per-user `<user>/cache/`
`find .../appdata_* -path '*/preview/*' | wc -l`, then dirs are a further candidate. Seafile: no excludes — the block store
`restic backup --dry-run -vv --exclude-file ...` prints every *is* the data.
include/exclude decision without uploading. (Per-user `<user>/cache/`
dirs are a further regenerable candidate if the count justifies it.)
Cache on tmpfs was considered and rejected: the cache exists to avoid
*network* metadata re-fetches; tmpfs loses it at reboot (forcing a
full re-download), eats the RAM the ARC needs, and buys nothing —
disk-backed cache reads are ARC-warm after first touch anyway.
### Escape hatch: rustic ### Escape hatch: rustic
[rustic](https://github.com/rustic-rs/rustic) speaks the **same [rustic](https://github.com/rustic-rs/rustic) speaks the same
repository format** — switching later is a binary swap on the same repository format — switching later is a binary swap on the same
repos, no re-seed. It would solve two specific pains if they repos (decided 2026-08-25: restic first, rustic if performance
materialize: **lock-free prune** (no backup/prune scheduling issues rise; manual upgrades accepted). It would bring lock-free
constraint) and a **lower RAM footprint** (the seafile-index prune and a lower RAM footprint. Don't run both concurrently on one
concern). Not chosen first because it isn't in Debian (manual repo (rustic ignores restic's locks); a clean switchover is fine.
upgrades) and has a far smaller community than restic. Don't run
both tools concurrently on one repo (rustic ignores restic's locks);
a clean switchover is fine.
## 3. Install + init (root on ks4) ## 3. Install + init (root on ks4)
```sh ```sh
apt install restic # Debian 13 ships ≥0.17 (repo v2, compression, pack-size ok) apt install restic sshfs # sshfs: needed by `incus file mount` (§6)
head -c 32 /dev/urandom | base64 > /root/.restic-passphrase && chmod 600 /root/.restic-passphrase head -c 32 /dev/urandom | base64 > /root/.restic-passphrase && chmod 600 /root/.restic-passphrase
# ⚠️ password manager, NOW. # ⚠️ password manager, NOW.
export AWS_ACCESS_KEY_ID=<AK> AWS_SECRET_ACCESS_KEY=<SK> # -> /root/.restic-env (mode 600) cat > /root/.restic-env <<'EOF'
export AWS_ACCESS_KEY_ID=<AK>
export AWS_SECRET_ACCESS_KEY=<SK>
export RESTIC_PASSWORD_FILE=/root/.restic-passphrase export RESTIC_PASSWORD_FILE=/root/.restic-passphrase
restic -r s3:s3.sbg.io.cloud.ovh.net/restic/nextcloud init export RESTIC_CACHE_DIR=/backup/restic-cache
restic -r s3:s3.sbg.io.cloud.ovh.net/restic/seafile init EOF
mkdir -p /backup/restic-cache chmod 600 /root/.restic-env && . /root/.restic-env && mkdir -p /backup/restic-cache
for r in restic-data restic-nextcloud restic-seafile restic-instances; do
restic -r s3:s3.sbg.io.cloud.ovh.net/$r init
done
``` ```
## 4. Seed plan ## 4. Seed plan
Order matters: 1. **ARC floor first** (helps every walk on the box):
1. **Bump the ARC floor first** so metadata walks stay warm between
nightly runs (also helps the plakar legs):
```sh ```sh
echo 8589934592 > /sys/module/zfs/parameters/zfs_arc_min echo 8589934592 > /sys/module/zfs/parameters/zfs_arc_min
echo "options zfs zfs_arc_min=8589934592" > /etc/modprobe.d/zfs-arc.conf echo "options zfs zfs_arc_min=8589934592" > /etc/modprobe.d/zfs-arc.conf
update-initramfs -u # zfs loads from the initramfs update-initramfs -u
awk '/^c_min/ {printf "%.1f GiB\n", $3/1073741824}' /proc/spl/kstat/zfs/arcstats awk '/^c_min/ {printf "%.1f GiB\n", $3/1073741824}' /proc/spl/kstat/zfs/arcstats
``` ```
⚠️ the floor is defended even under memory pressure — real ⚠️ the floor is defended even under pressure — real headroom is
headroom becomes ~31G − services − 8G; revisit before adding any ~31 G − services − 8 G; revisit before adding a fat service.
fat new service. 2. `restic-data` (~50 G — an hour).
2. **nextcloud** (proven ~22 MiB/s from the live pool → ~8 h), in a 3. `restic-nextcloud` (proven ~22 MiB/s from the live pool → ~8 h),
screen: screen:
```sh ```sh
restic -r s3:.../restic/nextcloud --cache-dir /backup/restic-cache \ restic -r s3:s3.sbg.io.cloud.ovh.net/restic-nextcloud \
backup --pack-size 64 --read-concurrency 8 -o s3.connections=8 \ backup --pack-size 64 --read-concurrency 8 -o s3.connections=8 \
--exclude-file /root/scripts/restic-nextcloud-exclude \ --exclude-file /root/scripts/restic-nextcloud-exclude \
/var/lib/incus/storage-pools/data/containers/nextcloud/rootfs/nextcloud /var/lib/incus/storage-pools/data/containers/nextcloud/rootfs/nextcloud
``` ```
3. **seafile** — the one-time debt. Expect 1–2 days on the live pool 4. `restic-instances` (16 small replicas via §6 — a few hours).
(warmer than sdb, and `--read-concurrency 8` helps); it is 5. `restic-seafile` — the one-time debt: 1–2 days, restartable at any
restartable at any point (committed packs dedup on retry). point (committed packs dedup on retry).
## 5. Nightly runs + retention + verification ## 5. Nightly driver — `scripts/restic-backup.sh`
Cron (05:00 — after the 04:30 plakar leg's dump phase, sources don't Same shape as the plakar drivers (flock, env, loud logging). Phases:
overlap; wrap in a `restic-backup.sh` with flock + env, same style as
the plakar drivers): 1. **dumps** — lifted verbatim from `plakar-backup.sh`: incus DB
dumps + auto-discovered MariaDB/PostgreSQL (native and docker)
dumps into `/backup/plakar-dumps` (path kept), then
`restic -r …/restic-data backup` of the dump tree **and** the 8
small source paths (one snapshot, paths from a list file).
2. **giants** — nextcloud (with excludes) and seafile into their
repos.
3. **instances** — §6 loop into `restic-instances`.
4. **retention** — per repo: `restic forget --keep-daily 14
--keep-weekly 8 --keep-monthly 6` (instant without `--prune`).
```cron ```cron
0 5 * * * /root/scripts/restic-backup.sh >> /var/log/restic-backup.log 2>&1 0 5 * * * /root/scripts/restic-backup.sh >> /var/log/restic-backup.log 2>&1
``` ```
Per repo, the script does: Weekly maintenance (Sunday; prune takes an **exclusive lock** — never
overlap the 05:00 run):
```sh
restic backup --pack-size 64 --read-concurrency 8 ... # as in §4
restic forget --keep-daily 14 --keep-weekly 8 --keep-monthly 6
restic unlock 2>/dev/null # clear stale locks after crashes
```
- `forget` without `--prune` is instant; actual **prune runs weekly**
(Sunday, offset from plakar's check; prune takes an exclusive lock —
never overlap the 05:00 backups):
```cron ```cron
0 14 * * 0 ... restic prune --max-unused 10% --max-repack-size 4G --pack-size 64 ... 0 14 * * 0 /root/scripts/restic-maintenance.sh >> /var/log/restic-maintenance.log 2>&1
``` ```
Tuning rationale (our churn: ~0.1–0.4 GiB/day dead data vs ~700 GiB
repos, seafile append-mostly ≈ nothing dies): `--max-unused 10%`
lets prune skip repacking for months, carrying ≤10 % slack in cheap
object storage; `--max-repack-size 4G` bounds any single Sunday's
rewrite to ~10–15 min even after a mass deletion, deferring the
rest to following weeks. (`--repack-small` only matters for repos
with legacy sub-64 MiB packs — not us, we start at 64.)
- **Verification**: weekly `restic check` (structure only, cheap) +
`check --read-data-subset=1/52` rotating — a full data verification
of every byte once a year, ~25 GiB read per week, for free.
Expected steady state: nightly = local stat-walk (minutes once ARC is Per repo: `restic prune --max-unused 10% --max-repack-size 4G
warm) + churn upload; no per-run S3 metadata storm — the exact --pack-size 64` — our dead-data rate is ~0.1–0.4 GiB/day against
property #2338 is about. ~700 GiB repos (seafile append-mostly ≈ nothing dies), so 10 %
tolerated slack lets prune skip repacking for months, and the 4 G cap
bounds any single Sunday to ~10–15 min even after a mass deletion.
Then `restic check` (structure, cheap) + `check
--read-data-subset=1/52` rotating — full data verification of every
byte once a year, ~25 GiB read per week.
## 6. Restore test (gate before trusting it) ## 6. Instances without a plugin — `incus file mount`
Neither restic nor rustic accepts source plugins, and forking the
backup tool is the wrong place to carry a patch. Instead, incus
exposes any instance (including **stopped replicas**) as a FUSE mount
over the same per-instance sftp API the plakar integration used:
```sh ```sh
restic -r s3:.../restic/nextcloud --cache-dir /backup/restic-cache \ mkdir -p /run/restic-incus
restore latest --target /backup/restore-test --include '/<some subtree>' for inst in $(incus list --project backup -c n -f csv); do
diff -r ... && rm -rf /backup/restore-test case ",$EXCLUDE_INSTANCES," in *",$inst,"*) continue;; esac # nextcloud,seafile
incus config show "$inst" --project backup --expanded > "/backup/plakar-dumps/incus/$inst.yaml"
incus file mount "$inst/" /run/restic-incus --project backup & # foreground process; background + kill
MPID=$!; sleep 2
restic -r s3:…/restic-instances backup --pack-size 64 --tag "$inst" /run/restic-incus
kill $MPID; wait $MPID 2>/dev/null; umount /run/restic-incus 2>/dev/null
done
``` ```
Never restore to `/tmp` — it is a 16 GiB tmpfs on ks4. (Exact mount lifecycle to be hardened in the script — `incus file
mount` runs in the foreground and needs `sshfs` installed.) All
instances by default, opt-out via a variable — the no-manifest-drift
rule survives the tool change. Chained after the 01:00 incus-copy in
its cron entry once validated, same freshness reasoning as before.
## 7. If restic ever takes over ALL the S3 backups Bonus over plakar: one shared repo dedups the Ubuntu base across all
16 replicas. Restore granularity: per-file; instance definitions ride
the dump tree (`incus/<inst>.yaml` + `incus-global-db.sql`).
Decision gate: **the upstream answer to #2338.** Two futures: ## 7. Restore test (gate)
- **plakar fixes the growth problem** → consolidate the giants *back* Per repo, one restore + diff (never to `/tmp` — 16 GiB tmpfs):
onto plakar, retire restic (delete the `restic` bucket) — one tool,
and the incus integration investment keeps compounding. ```sh
- **No fix on a useful horizon** → migrate the data leg to restic: restic -r s3:…/restic-nextcloud restore latest --target /backup/restore-test --include '<subtree>'
- dumps + 8 small fs sources are ~50 G total: re-seeding into a diff -r … && rm -rf /backup/restore-test
`restic/data` repo is an evening, and restic dedups the nightly ```
dump tree (incl. the static `geo.sql`) exactly like plakar does.
- **the incus leg is the sticking point**: the per-file instance ## 8. Evaluation protocol + plakar retirement
backups ride our plakar-native integration. restic equivalents
are all worse or heavier: backing up replica rootfs paths - **Parallel week**: plakar 04:30 cron keeps running untouched;
directly (fights incus's on-demand mounts), `incus export` restic runs at 05:00 on the same data. Same nights, same churn →
tarballs (server-side materialization, no room), or `zfs send` direct comparison (wall, bytes read from store, bytes written, peak
streams into restic's stdin mode (works — `restic backup --stdin` RSS via `/usr/bin/time -v`) against plakar's measured table in
— but chains restore through zfs+incus knowledge). [plakar-s3-data.md](plakar-s3-data.md). Restore-speed compared on
- So a full migration really means "restic for all *file* data + the same 10 GiB dumps tree (plakar measured: 39 min).
keep plakar only for the incus leg" — two tools either way. - **Cutover** (after the comparison confirms + restore test passes):
Which is the same operational cost as today's split, minus the drop the plakar 04:30 + Sunday-check crons; 01:00 becomes
#2338 exposure. Conclusion: **full migration buys little; the `incus-copy ; restic instances phase` (or stays in the 05:00 run).
current split is not a temporary hack but a reasonable end Update plan.md legs table.
state.** Revisit only if plakar becomes unmaintained or the - **plakar retirement, one exception**: keep plakar installed and
incus leg moves to zfs send anyway. keep the `plakar-data` kloset until
[#2338](https://github.com/PlakarKorp/plakar/issues/2338) concludes
— the issue offers debug runs against that repository. Delete the
`plakar-incus` bucket immediately; `plakar-data` a few weeks later.
The `integration-incus` repo retires from production; optionally
still publish to PlakarKorp/hub as a community contribution.
- **Interim ks2 push** (until the nuc leg seeds after FTTH, expected
before end of September): re-enable the replica push so instances
keep an off-site copy meanwhile:
```cron
0 2 * * * /root/scripts/incus-copy.sh -d ks2 -m push >> /var/log/incus-copy.log 2>&1
```