doc/ks4/seafile-gc.md

seafile: garbage collection + history retention

Status: GC executed 2026-08-26/27. Results:

metric	before	after
dataset inodes	24,885,504	515,842 (after Documents recreation + final -r, 2026-08-28)
storage/blocks/	867 G	632 G (−235 G)
fs objects	~19.3 M	~100 k (19,207,400 removed)
ghost library stores	13	0 (blocks/fs eaten by GC; 4 empty commit husks removed manually — --rm-deleted couldn't see them, they predate GarbageRepos)
wall time		16 min (blocks) + 0.2 s (-r) + 21 h 33 m (-R)

Root-root-cause found: one library — Documents (c4ec7247, 2.9 G data) — held 19.02 M fs objects and ~4.9 M commits: a client has been committing at extreme frequency for years. The ~4.9 M commit objects survive GC (CE never deletes commit objects) and are most of the remaining inodes. Remedy: identify and fix the committing client, then recreate the library (sync down current state → new library → delete old → GC -r, which works for 12.x-era deletions) — that retires the commit store entirely and takes the dataset to ~600 k inodes, at which point per-file backup of seafile becomes ordinary.

All crontab backup jobs on ks4 remain commented out until restart + validation completes.

Root cause of the 24.9 M-inode / 933 G dataset

Investigated after per-file backups of /opt/seafile proved unwalkable (restic seed: ~20 M objects remaining at ~66 files/s). Findings (seafile-server 12.0.14, binary install, upgraded from 8.0 → 9.0 → 12.0):

• GC has never run — no gc.log ever, no cron in the container. seaf-gc is the only mechanism that deletes blocks/fs/commits objects; without it nothing is ever reclaimed.
• 13 ghost libraries: storage/commits/ holds 30 library dirs; the DB has 17 live repos and an empty trash. Deleted libraries' object stores persist until seaf-gc --rm-deleted.
• Unlimited history on 16 of 17 repos: only one library (the 240 G one) has a retention limit (30 days, RepoHistoryLimit). Everything else keeps every version of every file since creation.
• Sizes: live repo data ≈ 710 G (DB RepoSize) vs 867 G in blocks/ → ≥150 G dead blocks; the rest of the dataset is the fs/commits object swarm (~5 M commits objects counted; fs likely 15 M+, sub-4K each) — the thing that makes every per-file walk take days.

References: https://manual.seafile.com/12.0/maintain/seafile_gc/

Procedure

0. Safety net (host, instant)

zfs snapshot data/containers/seafile@pre-gc      # rollbackable; delete after validation

Leg-1 replica on sdb is the second net (crontab currently disabled — the replica is pre-GC state, which is exactly what we want here).

1. Default history retention (inside the container)

/opt/seafile/conf/seafile.conf:

[history]
keep_days = 90

Applies as the default for all libraries without a per-library setting (the existing per-library 30 d row stays). Users can still override per library in the web UI if ENABLE_REPO_HISTORY_SETTING = True in seahub_settings.py.

2. GC — offline, maintenance window

CE's GC requires seafile stopped. On this pathological store expect hours (it traverses fs/commits on a cold HDD); the dry-run sizes the window. Inside the container:

cd /opt/seafile/seafile-server-latest
./seahub.sh stop && ./seafile.sh stop     # or the container's systemd units — check systemctl

./seaf-gc.sh --dry-run                    # per-repo report of what would be reclaimed
./seaf-gc.sh                              # blocks GC, honors keep_days
./seaf-gc.sh --rm-deleted                 # purge the 13 ghost libraries
./seaf-gc.sh --rm-fs                      # unreferenced fs objects — the inode bulk
                                          # (verify flag names: ./seaf-gc.sh --help; manual 12.0)

./seafile.sh start && ./seahub.sh start

3. Measure the result (host)

df -i /var/lib/incus/storage-pools/data/containers/seafile   # was 24,885,504 inodes
du -sh .../seafile/rootfs/opt/seafile/seafile-data/storage/blocks   # was 867 G

4. Aftermath

• validate seafile works (client sync, web, download a versioned file), then zfs destroy data/containers/seafile@pre-gc
• re-enable the crontab (incus-copy leg first; its next refresh transfers the shrunken dataset)
• recurring GC (done 2026-08-27 design): the MySQL backend runs GC online (verified — the runs printed "use online GC" and -r ran with services up), so no stop/start is needed. Host crontab:

  # monthly dry-run report (cheap health check, read-only)
  0 6 1 * * incus exec seafile -- sh -c "cd /opt/seafile/seafile-server-latest && ./seaf-gc.sh -D" >> /var/log/seafile-gc.log 2>&1
  # quarterly real GC: blocks + trashed repos + fs objects
  0 6 15 1,4,7,10 * incus exec seafile -- sh -c "cd /opt/seafile/seafile-server-latest && ./seaf-gc.sh && ./seaf-gc.sh -r && ./seaf-gc.sh -R" >> /var/log/seafile-gc.log 2>&1
  

  At the post-cleanup scale (~120 k live files) the full sequence is minutes, not hours. Keep it clear of the 01:00/04:30/05:00 backup slots.
• 2026-08-27 follow-ups executed: 8 old libraries deleted by user → trash emptied → seaf-gc -r purged their stores (18 m, blocks now 590 G, −277 G total); 4 pre-GarbageRepos husks removed manually; Documents to be recreated from a local re-sync (retires its ~4.9 M fossil commit objects → expected final ~600 k inodes)
• re-decide seafile's S3 backup with real post-GC numbers: per-file restic redux if the object count collapses far enough, zfs send | restic --stdin-from-command otherwise (see restic-backup.md)
• then plan the 13.x upgrade (Docker-only — see the incus-compose notes and https://manual.seafile.com/13.0/upgrade/upgrade_notes_for_13.0.x/) against a much smaller estate

Related cleanup while in there

• logs/seafevents.log.2024-* and friends — years of rotated logs
• the old seafile-server-8.0.7 / 9.0.10 dirs once 12.x is proven

Closure (2026-08-28)

Documents recreated from a local re-sync, old library trashed → trash emptied → seaf-gc.sh -r (78 m) retired its ~4.9 M fossil commit objects. Final: 515,842 inodes (from 24,885,504, −98 %), blocks 590 G (from 867 G), 8 libraries / ~120 k live files / 703 GB. The @pre-gc snapshot can be destroyed once the new Documents library has proven itself for a few days (it pins the freed space until then).