Julien LutranandClaude Fable 5
ae82d9f682
Seafile's block store wedged the fs walker for 12h in an lstat preemption/EINTR livelock on the live pool; coverage moves to the incus leg reading the sdb replica. Restore test + cron are the remaining steps of the data leg. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
110 lines
4.0 KiB
Markdown
110 lines
4.0 KiB
Markdown
# seafile — GCID table loss, 6-month CPU burn (fixed 2026-08-22)
|
||
|
||
`seaf-server` in the `seafile` container sat at 40–47 % CPU for months
|
||
while appearing to do nothing: negligible client traffic on `eth0`
|
||
(344 MB sent over 102 days) but 82 GB and 385 M packets over loopback,
|
||
and ~11 **days** of accumulated CPU time.
|
||
|
||
## Root cause
|
||
|
||
The `seafile-db` MariaDB database was missing the `GCID` and
|
||
`LastGCID` tables. Since Seafile 11 every client sync operation checks
|
||
a per-repo GC id (`SELECT gc_id FROM GCID WHERE repo_id = ? FOR
|
||
UPDATE`); with the table gone the query failed, the client retried,
|
||
and the cycle repeated forever — no sync ever completed:
|
||
|
||
- 6 of seaf-server's fileserver worker threads spinning in pure
|
||
user-space, ~3 new MySQL connections/s (167 loopback `TIME_WAIT`
|
||
to :3306 at any moment),
|
||
- `seafile.log` grew to **1.8 GB** of the single warning
|
||
`Failed to prepare sql SELECT gc_id FROM GCID … Table
|
||
'seafile-db.GCID' doesn't exist`, several times per second.
|
||
|
||
First occurrence **2026-03-01 22:17** — under 11.0.9, i.e. *before*
|
||
both the 2026-03-16 restart and the 2026-05-11 upgrade to 12.0.14.
|
||
`GCID` is an old core table (the 6.3.0 upgrade SQL already ALTERs
|
||
it), so it existed and was lost. **What dropped it around 2026-03-01
|
||
was never identified** — if tables vanish again, start there.
|
||
|
||
## Diagnosis trail (what worked without container root)
|
||
|
||
From the host as `julien` (sudo grant covers read-only `incus`
|
||
subcommands only):
|
||
|
||
```sh
|
||
sudo incus info seafile # PID 2410, 1005 GiB, 2 GiB RAM
|
||
ps -eo pid,%cpu,args --sort=-%cpu # seaf-server pid 10217 top hog
|
||
# per-thread CPU from /proc (world-readable, no root needed):
|
||
for t in /proc/10217/task/*; do awk '{print $1, $14+$15}' $t/stat; done
|
||
# utime vs stime split → pure user-space compute, ~0 syscalls
|
||
# container netns TCP table via the process:
|
||
awk 'NR>1 && $4=="06"' /proc/10217/net/tcp | wc -l # 167 TIME_WAIT → :3306
|
||
```
|
||
|
||
Reading `/opt/seafile/logs/seafile.log` needed a temporary sudoers
|
||
extension (`/usr/bin/incus exec *`, `/usr/bin/incus file pull *` in a
|
||
drop-in — **removed again after the fix**). The log line then made the
|
||
cause obvious.
|
||
|
||
## Fix
|
||
|
||
The exact DDL ships inside the server binary:
|
||
|
||
```sh
|
||
strings /opt/seafile/seafile-server-12.0.14/seafile/bin/seaf-server \
|
||
| grep 'CREATE TABLE IF NOT EXISTS.*GCID'
|
||
```
|
||
|
||
Applied (root in the container; additive, no restart needed):
|
||
|
||
```sql
|
||
CREATE TABLE IF NOT EXISTS GCID (
|
||
id BIGINT NOT NULL PRIMARY KEY AUTO_INCREMENT,
|
||
repo_id CHAR(36), gc_id CHAR(36),
|
||
UNIQUE INDEX(repo_id)) ENGINE=INNODB;
|
||
CREATE TABLE IF NOT EXISTS LastGCID (
|
||
id BIGINT NOT NULL PRIMARY KEY AUTO_INCREMENT,
|
||
repo_id CHAR(36), client_id VARCHAR(128), gc_id CHAR(36),
|
||
UNIQUE INDEX(repo_id, client_id)) ENGINE=INNODB;
|
||
```
|
||
|
||
The warning storm stopped the same second, seaf-server dropped to
|
||
0 % CPU, host load fell from ~8–12 toward normal. `seafile.log` was
|
||
truncated (`truncate -s 0`).
|
||
|
||
## Log rotation (added 2026-08-22)
|
||
|
||
Seafile ships **no rotation** — that's how one warning reached 1.8 GB
|
||
unnoticed. `/etc/logrotate.d/seafile` in the container:
|
||
|
||
```
|
||
/opt/seafile/logs/*.log {
|
||
weekly
|
||
maxsize 100M
|
||
rotate 8
|
||
missingok
|
||
notifempty
|
||
compress
|
||
delaycompress
|
||
copytruncate
|
||
}
|
||
```
|
||
|
||
`copytruncate` avoids pid/SIGUSR1 handling for the various daemons
|
||
(seaf-server, seahub, seafevents…) at the cost of possibly losing a
|
||
few lines at rotation — fine here. `maxsize 100M` caps a future spam
|
||
storm at the next daily logrotate run instead of next week.
|
||
|
||
## Related cleanup / leftovers
|
||
|
||
- Legacy 4 AM cron on ks4 rsyncing seafile's rootfs to old **ks2**
|
||
(`164.132.173.57`) removed 2026-08-22 — redundant with the
|
||
[incus-copy](incus-copy.md) legs and it burned another ~47 % CPU
|
||
per run.
|
||
- Still missing from `seafile-db` (harmless, nothing queries them
|
||
unless per-folder permissions are used): `FolderUserPerm`,
|
||
`FolderGroupPerm`, `FolderPermTimestamp`. Same `strings`/upgrade-SQL
|
||
trick provides the DDL if ever needed.
|
||
- Host swap (1 GB) was 100 % full during the incident; unrelated but
|
||
worth keeping an eye on.
|