diff --git a/ks4/plakar-incus-integration.md b/ks4/plakar-incus-integration.md index 0ba0cb7..7ce34a1 100644 --- a/ks4/plakar-incus-integration.md +++ b/ks4/plakar-incus-integration.md @@ -119,6 +119,41 @@ to the installed plakar line (v1.1.x), rebuild the ptar on a plakar major upgrade; a mismatch surfaces at backup time, so run a manual `plakar backup @ks4-incus` after plakar upgrades. +## The giant-instance problem (2026-08-24, seed attempt #1) + +The per-file S3 seed of the whole `backup` project stalled at +~1 MiB/s once it reached the big replicas: sdb at 96 % util doing +~275 × 4 KiB reads/s — **cold random tiny-file reads on an HDD are +seek-bound**, regardless of importer (the fs importer hit the same +wall on the live pool; sftp changes nothing). At ~275 IOPS, +nextcloud (725 G, ~500 k files) + seafile (933 G, millions of block +files) need *weeks* to seed. The 16 small instances are unaffected +(solar smoke test: 5 GiB in 2m38s). + +Options considered for nextcloud/seafile: + +1. **backups-API tarball (`mode=image`)** — sequential-ish but incus + materializes the tarball server-side first (no room on `/`; would + need `storage.backups_volume` on the backup pool) and the tar walk + itself seeks like any tree walk. Double IO nightly. Weak. +2. **`zfs send` of the replica dataset** — the only truly sequential + read (disk block order, ~100+ MiB/s, no temp space): seed in + hours. Full send nightly is ~4 h read for both giants; CDC chunking + dedups unchanged stream regions. Restore = `zfs receive` + + `incus admin recover` (document!). Backend-specific — acceptable as + an opt-in mode for exactly these two. +3. **Per-file + warm metadata** — after a seed, nightly walks are + stat-only; keeping dnodes resident (`zfs_arc_min` ≈ 8 G) could make + them fast. Doesn't solve the *seed*. +4. **Exclude giants from this leg** — leaves seafile without any + fresh off-site copy (data leg already excludes it). Not acceptable + long-term. + +Direction (pending decision): seed the 16 small instances per-file +(`exclude_instances=nextcloud,seafile`), then implement option 2 as a +`zfs+send://` mode or side-channel for the two giants, and revisit 3 +for nightly stat-walk speed. + ## Open questions - uid/gid view through the instance sftp endpoint (idmap handling for