From 6c392356e8471fdebc73cb5beaebdb7662b139a7 Mon Sep 17 00:00:00 2001 From: Julien Lutran Date: Mon, 24 Aug 2026 17:04:40 +0200 Subject: [PATCH] plakar-s3-data: leg live; restore tests must target /backup, not tmpfs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - status: S3 kloset seeded, restore-tested, nightly cron installed - restore-test paths moved off /tmp — it is a 16G tmpfs on ks4 and the 10 GiB dumps restore sat in RAM squeezing the ZFS ARC - seafile-fs note rewritten: cause unknown, local-kloset bisect next; nextcloud 3-4h/night cost tracked for the upstream growth issue Co-Authored-By: Claude Fable 5 --- ks4/plakar-s3-data.md | 51 +++++++++++++++++++++++-------------------- 1 file changed, 27 insertions(+), 24 deletions(-) diff --git a/ks4/plakar-s3-data.md b/ks4/plakar-s3-data.md index 692e6fc..758430f 100644 --- a/ks4/plakar-s3-data.md +++ b/ks4/plakar-s3-data.md @@ -1,13 +1,13 @@ # plakar data + DB backups on ks4 (leg 3) -Status: **pipeline validated 2026-08-22** against a local test kloset -(`/backup/plakar-test`): full dump auto-discovery incl. grants/globals, -exclude mechanism, fs sources, prune + maintenance, rc=0; second run -confirmed incremental behavior (21 GiB dumps snapshot → 60 MiB -written). **S3 credentials ready — next: store add, create, seed.** -Replaces `incus-backup.sh` (the 04:00 rsync to ks2 stays alive as a -stopgap until this leg is verified — see -[local-backup-cron.md](local-backup-cron.md)). +Status: **live since 2026-08-24** — S3 kloset +(`lutran-ks4-plakar-data`) seeded (dumps + 9 fs sources), restore +test passed (file tree + 10 GiB dumps snapshot), 04:30 nightly cron + +Sunday `check -latest` installed. Replaces `incus-backup.sh` (its +04:00 ks2 rsync cron is disabled). Open items: `seafile-fs` excluded +(see §3 warning), `nextcloud-fs` costs 3–4 h/night (cost scales with +tree size — upstream issue drafted, observing a few nights before +filing). Architecture: plakar runs **on ks4** (decision recorded in [plan.md](../ks2/plan.md)) and backs up **directly into an S3 kloset** @@ -112,16 +112,19 @@ Beyond databases, `outline`/`login` may hold non-DB state (uploaded files, docker volumes) — decide per instance whether an fs source is needed in §4. -⚠️ **seafile-fs excluded from the data leg (2026-08-24).** The S3 -seed wedged for 12 h inside a single `lstat()` in seafile's block -store (millions of tiny files on the contended live pool). SIGQUIT -goroutine dump: main goroutine looping in `os.ignoringEINTR` around -`fstatat` — a Go preemption/EINTR **livelock** (100 % CPU, no -progress; dmesg clean, so not a kernel hang). `seafile-fs` is -commented out of `plakar-sources`; seafile's S3 coverage comes from -the incus leg instead, which reads the quiesced replica on the idle -`backup` pool. If a live-pool retry is ever wanted: -`GODEBUG=asyncpreemptoff=1` is the standard mitigation. +⚠️ **seafile-fs excluded from the data leg (2026-08-24).** Its S3 +seed ran 12 h with no visible upload and was killed. Successive +theories (EINTR livelock — `GODEBUG=asyncpreemptoff=1` changed +nothing; kernel reclaim stall — PSI counters too low; ARC-starved +lstats — disproved by `find` walking nextcloud's whole tree in 8m20 +cold / 19 s warm) all fell to evidence; cause **unknown**. Next +diagnostic: bisect by seeding seafile-fs into a *local* scratch +kloset (`plakar at /backup/plakar-scratch create` + timed backup) — +completes fine → S3 store path implicated; crawls → importer-side on +that tree shape (millions of tiny block files); then the +maintainer-suggested `-trace=all` (writes to stdout). Until resolved, +seafile's fresh off-site coverage waits on the incus leg reading the +sdb replica. Findings from the first dry-run (2026-08-22): @@ -207,19 +210,19 @@ export PLAKAR_PASSPHRASE=$(cat /root/.plakar-passphrase) plakar at @s3 ls # expect 11 snapshots: dumps + 10 fs sources # (a) file tree — restore solar-fs and diff against the live tree -plakar at @s3 restore -to /tmp/restore-test -diff -r /tmp/restore-test/var/lib/incus/storage-pools/data/containers/solar/rootfs/var/www/html/solar \ +plakar at @s3 restore -to /backup/restore-test +diff -r /backup/restore-test/var/lib/incus/storage-pools/data/containers/solar/rootfs/var/www/html/solar \ /var/lib/incus/storage-pools/data/containers/solar/rootfs/var/www/html/solar -rm -rf /tmp/restore-test +rm -rf /backup/restore-test # (b) database — restore a dump and load it into a scratch container -plakar at @s3 restore -to /tmp/restore-db +plakar at @s3 restore -to /backup/restore-db incus launch images:ubuntu/24.04 dbtest incus exec dbtest -- apt-get install -y mariadb-server -incus file push /tmp/restore-db/backup/plakar-dumps/mariadb/freshrss/freshrss.sql dbtest/root/ +incus file push /backup/restore-db/backup/plakar-dumps/mariadb/freshrss/freshrss.sql dbtest/root/ incus exec dbtest -- sh -c 'mariadb < /root/freshrss.sql' # dump embeds CREATE DATABASE incus exec dbtest -- mariadb -e 'SELECT COUNT(*) FROM information_schema.tables WHERE table_schema="freshrss"' -incus delete -f dbtest && rm -rf /tmp/restore-db +incus delete -f dbtest && rm -rf /backup/restore-db ``` Pass = diff empty (modulo files changed since the snapshot) and the