doc/ks4/seafile-gcid-fix.md

seafile — GCID table loss, 6-month CPU burn (fixed 2026-08-22)

seaf-server in the seafile container sat at 40–47 % CPU for months while appearing to do nothing: negligible client traffic on eth0 (344 MB sent over 102 days) but 82 GB and 385 M packets over loopback, and ~11 days of accumulated CPU time.

Root cause

The seafile-db MariaDB database was missing the GCID and LastGCID tables. Since Seafile 11 every client sync operation checks a per-repo GC id (SELECT gc_id FROM GCID WHERE repo_id = ? FOR UPDATE); with the table gone the query failed, the client retried, and the cycle repeated forever — no sync ever completed:

• 6 of seaf-server's fileserver worker threads spinning in pure user-space, ~3 new MySQL connections/s (167 loopback TIME_WAIT to :3306 at any moment),
• seafile.log grew to 1.8 GB of the single warning Failed to prepare sql SELECT gc_id FROM GCID … Table 'seafile-db.GCID' doesn't exist, several times per second.

First occurrence 2026-03-01 22:17 — under 11.0.9, i.e. before both the 2026-03-16 restart and the 2026-05-11 upgrade to 12.0.14. GCID is an old core table (the 6.3.0 upgrade SQL already ALTERs it), so it existed and was lost. What dropped it around 2026-03-01 was never identified — if tables vanish again, start there.

Diagnosis trail (what worked without container root)

From the host as julien (sudo grant covers read-only incus subcommands only):

sudo incus info seafile                    # PID 2410, 1005 GiB, 2 GiB RAM
ps -eo pid,%cpu,args --sort=-%cpu          # seaf-server pid 10217 top hog
# per-thread CPU from /proc (world-readable, no root needed):
for t in /proc/10217/task/*; do awk '{print $1, $14+$15}' $t/stat; done
# utime vs stime split → pure user-space compute, ~0 syscalls
# container netns TCP table via the process:
awk 'NR>1 && $4=="06"' /proc/10217/net/tcp | wc -l   # 167 TIME_WAIT → :3306

Reading /opt/seafile/logs/seafile.log needed a temporary sudoers extension (/usr/bin/incus exec *, /usr/bin/incus file pull * in a drop-in — removed again after the fix). The log line then made the cause obvious.

Fix

The exact DDL ships inside the server binary:

strings /opt/seafile/seafile-server-12.0.14/seafile/bin/seaf-server \
  | grep 'CREATE TABLE IF NOT EXISTS.*GCID'

Applied (root in the container; additive, no restart needed):

CREATE TABLE IF NOT EXISTS GCID (
  id BIGINT NOT NULL PRIMARY KEY AUTO_INCREMENT,
  repo_id CHAR(36), gc_id CHAR(36),
  UNIQUE INDEX(repo_id)) ENGINE=INNODB;
CREATE TABLE IF NOT EXISTS LastGCID (
  id BIGINT NOT NULL PRIMARY KEY AUTO_INCREMENT,
  repo_id CHAR(36), client_id VARCHAR(128), gc_id CHAR(36),
  UNIQUE INDEX(repo_id, client_id)) ENGINE=INNODB;

The warning storm stopped the same second, seaf-server dropped to 0 % CPU, host load fell from ~8–12 toward normal. seafile.log was truncated (truncate -s 0).

Log rotation (added 2026-08-22)

Seafile ships no rotation — that's how one warning reached 1.8 GB unnoticed. /etc/logrotate.d/seafile in the container:

/opt/seafile/logs/*.log {
	weekly
	maxsize 100M
	rotate 8
	missingok
	notifempty
	compress
	delaycompress
	copytruncate
}

copytruncate avoids pid/SIGUSR1 handling for the various daemons (seaf-server, seahub, seafevents…) at the cost of possibly losing a few lines at rotation — fine here. maxsize 100M caps a future spam storm at the next daily logrotate run instead of next week.

Related cleanup / leftovers

• Legacy 4 AM cron on ks4 rsyncing seafile's rootfs to old ks2 (164.132.173.57) removed 2026-08-22 — redundant with the incus-copy legs and it burned another ~47 % CPU per run.
• Still missing from seafile-db (harmless, nothing queries them unless per-folder permissions are used): FolderUserPerm, FolderGroupPerm, FolderPermTimestamp. Same strings/upgrade-SQL trick provides the DDL if ever needed.
• Host swap (1 GB) was 100 % full during the incident; unrelated but worth keeping an eye on.