plakar-s3-data: leg live; restore tests must target /backup, not tmpfs

- status: S3 kloset seeded, restore-tested, nightly cron installed
- restore-test paths moved off /tmp — it is a 16G tmpfs on ks4 and the
  10 GiB dumps restore sat in RAM squeezing the ZFS ARC
- seafile-fs note rewritten: cause unknown, local-kloset bisect next;
  nextcloud 3-4h/night cost tracked for the upstream growth issue

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

ks4/plakar-s3-data.md

@@ -1,13 +1,13 @@
 # plakar data + DB backups on ks4 (leg 3)
 
-Status: **pipeline validated 2026-08-22** against a local test kloset
+Status: **live since 2026-08-24** — S3 kloset
-(`/backup/plakar-test`): full dump auto-discovery incl. grants/globals,
+(`lutran-ks4-plakar-data`) seeded (dumps + 9 fs sources), restore
-exclude mechanism, fs sources, prune + maintenance, rc=0; second run
+test passed (file tree + 10 GiB dumps snapshot), 04:30 nightly cron +
-confirmed incremental behavior (21 GiB dumps snapshot → 60 MiB
+Sunday `check -latest` installed. Replaces `incus-backup.sh` (its
-written). **S3 credentials ready — next: store add, create, seed.**
+04:00 ks2 rsync cron is disabled). Open items: `seafile-fs` excluded
-Replaces `incus-backup.sh` (the 04:00 rsync to ks2 stays alive as a
+(see §3 warning), `nextcloud-fs` costs 3–4 h/night (cost scales with
-stopgap until this leg is verified — see
+tree size — upstream issue drafted, observing a few nights before
-[local-backup-cron.md](local-backup-cron.md)).
+filing).
 
 Architecture: plakar runs **on ks4** (decision recorded in
 [plan.md](../ks2/plan.md)) and backs up **directly into an S3 kloset**
@@ -112,16 +112,19 @@ Beyond databases, `outline`/`login` may hold non-DB state (uploaded
 files, docker volumes) — decide per instance whether an fs source is
 needed in §4.
 
-⚠️ **seafile-fs excluded from the data leg (2026-08-24).** The S3
+⚠️ **seafile-fs excluded from the data leg (2026-08-24).** Its S3
-seed wedged for 12 h inside a single `lstat()` in seafile's block
+seed ran 12 h with no visible upload and was killed. Successive
-store (millions of tiny files on the contended live pool). SIGQUIT
+theories (EINTR livelock — `GODEBUG=asyncpreemptoff=1` changed
-goroutine dump: main goroutine looping in `os.ignoringEINTR` around
+nothing; kernel reclaim stall — PSI counters too low; ARC-starved
-`fstatat` — a Go preemption/EINTR **livelock** (100 % CPU, no
+lstats — disproved by `find` walking nextcloud's whole tree in 8m20
-progress; dmesg clean, so not a kernel hang). `seafile-fs` is
+cold / 19 s warm) all fell to evidence; cause **unknown**. Next
-commented out of `plakar-sources`; seafile's S3 coverage comes from
+diagnostic: bisect by seeding seafile-fs into a *local* scratch
-the incus leg instead, which reads the quiesced replica on the idle
+kloset (`plakar at /backup/plakar-scratch create` + timed backup) —
-`backup` pool. If a live-pool retry is ever wanted:
+completes fine → S3 store path implicated; crawls → importer-side on
-`GODEBUG=asyncpreemptoff=1` is the standard mitigation.
+that tree shape (millions of tiny block files); then the
+maintainer-suggested `-trace=all` (writes to stdout). Until resolved,
+seafile's fresh off-site coverage waits on the incus leg reading the
+sdb replica.
 
 Findings from the first dry-run (2026-08-22):
 
@@ -207,19 +210,19 @@ export PLAKAR_PASSPHRASE=$(cat /root/.plakar-passphrase)
 plakar at @s3 ls          # expect 11 snapshots: dumps + 10 fs sources
 
 # (a) file tree — restore solar-fs and diff against the live tree
-plakar at @s3 restore -to /tmp/restore-test <solar-fs-snap-id>
+plakar at @s3 restore -to /backup/restore-test <solar-fs-snap-id>
-diff -r /tmp/restore-test/var/lib/incus/storage-pools/data/containers/solar/rootfs/var/www/html/solar \
+diff -r /backup/restore-test/var/lib/incus/storage-pools/data/containers/solar/rootfs/var/www/html/solar \
         /var/lib/incus/storage-pools/data/containers/solar/rootfs/var/www/html/solar
-rm -rf /tmp/restore-test
+rm -rf /backup/restore-test
 
 # (b) database — restore a dump and load it into a scratch container
-plakar at @s3 restore -to /tmp/restore-db <dumps-snap-id>
+plakar at @s3 restore -to /backup/restore-db <dumps-snap-id>
 incus launch images:ubuntu/24.04 dbtest
 incus exec dbtest -- apt-get install -y mariadb-server
-incus file push /tmp/restore-db/backup/plakar-dumps/mariadb/freshrss/freshrss.sql dbtest/root/
+incus file push /backup/restore-db/backup/plakar-dumps/mariadb/freshrss/freshrss.sql dbtest/root/
 incus exec dbtest -- sh -c 'mariadb < /root/freshrss.sql'   # dump embeds CREATE DATABASE
 incus exec dbtest -- mariadb -e 'SELECT COUNT(*) FROM information_schema.tables WHERE table_schema="freshrss"'
-incus delete -f dbtest && rm -rf /tmp/restore-db
+incus delete -f dbtest && rm -rf /backup/restore-db
 ```
 
 Pass = diff empty (modulo files changed since the snapshot) and the