plakar-s3-data: seed done except seafile-fs (EINTR livelock, excluded)
Seafile's block store wedged the fs walker for 12h in an lstat preemption/EINTR livelock on the live pool; coverage moves to the incus leg reading the sdb replica. Restore test + cron are the remaining steps of the data leg. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Julien Lutran
co-authored by
Claude Fable 5
Claude Fable 5
parent
4b73d77803
commit
ae82d9f682
@@ -0,0 +1,109 @@
|
||||
# seafile — GCID table loss, 6-month CPU burn (fixed 2026-08-22)
|
||||
|
||||
`seaf-server` in the `seafile` container sat at 40–47 % CPU for months
|
||||
while appearing to do nothing: negligible client traffic on `eth0`
|
||||
(344 MB sent over 102 days) but 82 GB and 385 M packets over loopback,
|
||||
and ~11 **days** of accumulated CPU time.
|
||||
|
||||
## Root cause
|
||||
|
||||
The `seafile-db` MariaDB database was missing the `GCID` and
|
||||
`LastGCID` tables. Since Seafile 11 every client sync operation checks
|
||||
a per-repo GC id (`SELECT gc_id FROM GCID WHERE repo_id = ? FOR
|
||||
UPDATE`); with the table gone the query failed, the client retried,
|
||||
and the cycle repeated forever — no sync ever completed:
|
||||
|
||||
- 6 of seaf-server's fileserver worker threads spinning in pure
|
||||
user-space, ~3 new MySQL connections/s (167 loopback `TIME_WAIT`
|
||||
to :3306 at any moment),
|
||||
- `seafile.log` grew to **1.8 GB** of the single warning
|
||||
`Failed to prepare sql SELECT gc_id FROM GCID … Table
|
||||
'seafile-db.GCID' doesn't exist`, several times per second.
|
||||
|
||||
First occurrence **2026-03-01 22:17** — under 11.0.9, i.e. *before*
|
||||
both the 2026-03-16 restart and the 2026-05-11 upgrade to 12.0.14.
|
||||
`GCID` is an old core table (the 6.3.0 upgrade SQL already ALTERs
|
||||
it), so it existed and was lost. **What dropped it around 2026-03-01
|
||||
was never identified** — if tables vanish again, start there.
|
||||
|
||||
## Diagnosis trail (what worked without container root)
|
||||
|
||||
From the host as `julien` (sudo grant covers read-only `incus`
|
||||
subcommands only):
|
||||
|
||||
```sh
|
||||
sudo incus info seafile # PID 2410, 1005 GiB, 2 GiB RAM
|
||||
ps -eo pid,%cpu,args --sort=-%cpu # seaf-server pid 10217 top hog
|
||||
# per-thread CPU from /proc (world-readable, no root needed):
|
||||
for t in /proc/10217/task/*; do awk '{print $1, $14+$15}' $t/stat; done
|
||||
# utime vs stime split → pure user-space compute, ~0 syscalls
|
||||
# container netns TCP table via the process:
|
||||
awk 'NR>1 && $4=="06"' /proc/10217/net/tcp | wc -l # 167 TIME_WAIT → :3306
|
||||
```
|
||||
|
||||
Reading `/opt/seafile/logs/seafile.log` needed a temporary sudoers
|
||||
extension (`/usr/bin/incus exec *`, `/usr/bin/incus file pull *` in a
|
||||
drop-in — **removed again after the fix**). The log line then made the
|
||||
cause obvious.
|
||||
|
||||
## Fix
|
||||
|
||||
The exact DDL ships inside the server binary:
|
||||
|
||||
```sh
|
||||
strings /opt/seafile/seafile-server-12.0.14/seafile/bin/seaf-server \
|
||||
| grep 'CREATE TABLE IF NOT EXISTS.*GCID'
|
||||
```
|
||||
|
||||
Applied (root in the container; additive, no restart needed):
|
||||
|
||||
```sql
|
||||
CREATE TABLE IF NOT EXISTS GCID (
|
||||
id BIGINT NOT NULL PRIMARY KEY AUTO_INCREMENT,
|
||||
repo_id CHAR(36), gc_id CHAR(36),
|
||||
UNIQUE INDEX(repo_id)) ENGINE=INNODB;
|
||||
CREATE TABLE IF NOT EXISTS LastGCID (
|
||||
id BIGINT NOT NULL PRIMARY KEY AUTO_INCREMENT,
|
||||
repo_id CHAR(36), client_id VARCHAR(128), gc_id CHAR(36),
|
||||
UNIQUE INDEX(repo_id, client_id)) ENGINE=INNODB;
|
||||
```
|
||||
|
||||
The warning storm stopped the same second, seaf-server dropped to
|
||||
0 % CPU, host load fell from ~8–12 toward normal. `seafile.log` was
|
||||
truncated (`truncate -s 0`).
|
||||
|
||||
## Log rotation (added 2026-08-22)
|
||||
|
||||
Seafile ships **no rotation** — that's how one warning reached 1.8 GB
|
||||
unnoticed. `/etc/logrotate.d/seafile` in the container:
|
||||
|
||||
```
|
||||
/opt/seafile/logs/*.log {
|
||||
weekly
|
||||
maxsize 100M
|
||||
rotate 8
|
||||
missingok
|
||||
notifempty
|
||||
compress
|
||||
delaycompress
|
||||
copytruncate
|
||||
}
|
||||
```
|
||||
|
||||
`copytruncate` avoids pid/SIGUSR1 handling for the various daemons
|
||||
(seaf-server, seahub, seafevents…) at the cost of possibly losing a
|
||||
few lines at rotation — fine here. `maxsize 100M` caps a future spam
|
||||
storm at the next daily logrotate run instead of next week.
|
||||
|
||||
## Related cleanup / leftovers
|
||||
|
||||
- Legacy 4 AM cron on ks4 rsyncing seafile's rootfs to old **ks2**
|
||||
(`164.132.173.57`) removed 2026-08-22 — redundant with the
|
||||
[incus-copy](incus-copy.md) legs and it burned another ~47 % CPU
|
||||
per run.
|
||||
- Still missing from `seafile-db` (harmless, nothing queries them
|
||||
unless per-folder permissions are used): `FolderUserPerm`,
|
||||
`FolderGroupPerm`, `FolderPermTimestamp`. Same `strings`/upgrade-SQL
|
||||
trick provides the DDL if ever needed.
|
||||
- Host swap (1 GB) was 100 % full during the incident; unrelated but
|
||||
worth keeping an eye on.
|
||||
Reference in New Issue
Block a user