plakar-s3-data: seed done except seafile-fs (EINTR livelock, excluded)

Seafile's block store wedged the fs walker for 12h in an lstat
preemption/EINTR livelock on the live pool; coverage moves to the
incus leg reading the sdb replica. Restore test + cron are the
remaining steps of the data leg.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

ks4/plakar-s3-data.md

@@ -112,6 +112,17 @@ Beyond databases, `outline`/`login` may hold non-DB state (uploaded
 files, docker volumes) — decide per instance whether an fs source is
 needed in §4.
 
+⚠️ **seafile-fs excluded from the data leg (2026-08-24).** The S3
+seed wedged for 12 h inside a single `lstat()` in seafile's block
+store (millions of tiny files on the contended live pool). SIGQUIT
+goroutine dump: main goroutine looping in `os.ignoringEINTR` around
+`fstatat` — a Go preemption/EINTR **livelock** (100 % CPU, no
+progress; dmesg clean, so not a kernel hang). `seafile-fs` is
+commented out of `plakar-sources`; seafile's S3 coverage comes from
+the incus leg instead, which reads the quiesced replica on the idle
+`backup` pool. If a live-pool retry is ever wanted:
+`GODEBUG=asyncpreemptoff=1` is the standard mitigation.
+
 Findings from the first dry-run (2026-08-22):
 
 - `livetrail`'s `/root/.my.cnf` had a stale `database = spot` default

ks4/seafile-gcid-fix.md

@@ -0,0 +1,109 @@
+# seafile — GCID table loss, 6-month CPU burn (fixed 2026-08-22)
+
+`seaf-server` in the `seafile` container sat at 40–47 % CPU for months
+while appearing to do nothing: negligible client traffic on `eth0`
+(344 MB sent over 102 days) but 82 GB and 385 M packets over loopback,
+and ~11 **days** of accumulated CPU time.
+
+## Root cause
+
+The `seafile-db` MariaDB database was missing the `GCID` and
+`LastGCID` tables. Since Seafile 11 every client sync operation checks
+a per-repo GC id (`SELECT gc_id FROM GCID WHERE repo_id = ? FOR
+UPDATE`); with the table gone the query failed, the client retried,
+and the cycle repeated forever — no sync ever completed:
+
+- 6 of seaf-server's fileserver worker threads spinning in pure
+  user-space, ~3 new MySQL connections/s (167 loopback `TIME_WAIT`
+  to :3306 at any moment),
+- `seafile.log` grew to **1.8 GB** of the single warning
+  `Failed to prepare sql SELECT gc_id FROM GCID … Table
+  'seafile-db.GCID' doesn't exist`, several times per second.
+
+First occurrence **2026-03-01 22:17** — under 11.0.9, i.e. *before*
+both the 2026-03-16 restart and the 2026-05-11 upgrade to 12.0.14.
+`GCID` is an old core table (the 6.3.0 upgrade SQL already ALTERs
+it), so it existed and was lost. **What dropped it around 2026-03-01
+was never identified** — if tables vanish again, start there.
+
+## Diagnosis trail (what worked without container root)
+
+From the host as `julien` (sudo grant covers read-only `incus`
+subcommands only):
+
+```sh
+sudo incus info seafile                    # PID 2410, 1005 GiB, 2 GiB RAM
+ps -eo pid,%cpu,args --sort=-%cpu          # seaf-server pid 10217 top hog
+# per-thread CPU from /proc (world-readable, no root needed):
+for t in /proc/10217/task/*; do awk '{print $1, $14+$15}' $t/stat; done
+# utime vs stime split → pure user-space compute, ~0 syscalls
+# container netns TCP table via the process:
+awk 'NR>1 && $4=="06"' /proc/10217/net/tcp | wc -l   # 167 TIME_WAIT → :3306
+```
+
+Reading `/opt/seafile/logs/seafile.log` needed a temporary sudoers
+extension (`/usr/bin/incus exec *`, `/usr/bin/incus file pull *` in a
+drop-in — **removed again after the fix**). The log line then made the
+cause obvious.
+
+## Fix
+
+The exact DDL ships inside the server binary:
+
+```sh
+strings /opt/seafile/seafile-server-12.0.14/seafile/bin/seaf-server \
+  | grep 'CREATE TABLE IF NOT EXISTS.*GCID'
+```
+
+Applied (root in the container; additive, no restart needed):
+
+```sql
+CREATE TABLE IF NOT EXISTS GCID (
+  id BIGINT NOT NULL PRIMARY KEY AUTO_INCREMENT,
+  repo_id CHAR(36), gc_id CHAR(36),
+  UNIQUE INDEX(repo_id)) ENGINE=INNODB;
+CREATE TABLE IF NOT EXISTS LastGCID (
+  id BIGINT NOT NULL PRIMARY KEY AUTO_INCREMENT,
+  repo_id CHAR(36), client_id VARCHAR(128), gc_id CHAR(36),
+  UNIQUE INDEX(repo_id, client_id)) ENGINE=INNODB;
+```
+
+The warning storm stopped the same second, seaf-server dropped to
+0 % CPU, host load fell from ~8–12 toward normal. `seafile.log` was
+truncated (`truncate -s 0`).
+
+## Log rotation (added 2026-08-22)
+
+Seafile ships **no rotation** — that's how one warning reached 1.8 GB
+unnoticed. `/etc/logrotate.d/seafile` in the container:
+
+```
+/opt/seafile/logs/*.log {
+	weekly
+	maxsize 100M
+	rotate 8
+	missingok
+	notifempty
+	compress
+	delaycompress
+	copytruncate
+}
+```
+
+`copytruncate` avoids pid/SIGUSR1 handling for the various daemons
+(seaf-server, seahub, seafevents…) at the cost of possibly losing a
+few lines at rotation — fine here. `maxsize 100M` caps a future spam
+storm at the next daily logrotate run instead of next week.
+
+## Related cleanup / leftovers
+
+- Legacy 4 AM cron on ks4 rsyncing seafile's rootfs to old **ks2**
+  (`164.132.173.57`) removed 2026-08-22 — redundant with the
+  [incus-copy](incus-copy.md) legs and it burned another ~47 % CPU
+  per run.
+- Still missing from `seafile-db` (harmless, nothing queries them
+  unless per-folder permissions are used): `FolderUserPerm`,
+  `FolderGroupPerm`, `FolderPermTimestamp`. Same `strings`/upgrade-SQL
+  trick provides the DDL if ever needed.
+- Host swap (1 GB) was 100 % full during the incident; unrelated but
+  worth keeping an eye on.