seafile-gc: diagnose never-run GC (24.9M inodes) and the reclamation procedure

GC never ran in the server's life: 13 ghost libraries, unlimited
history on 16/17 repos, 867G blocks vs 710G live data, and an
fs/commits object swarm that makes per-file backup walks take days.
Procedure: pre-GC zfs snapshot, keep_days=90 default, offline
seaf-gc (+ --rm-deleted, --rm-fs), measure, quarterly GC schedule,
then re-decide the S3 backup mechanism and the 13.x upgrade.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

ks4/seafile-gc.md

@@ -0,0 +1,102 @@
+# seafile: garbage collection + history retention
+
+Status: **diagnosed 2026-08-26, GC not yet run.** All crontab backup
+jobs on ks4 are commented out until this completes.
+
+## Root cause of the 24.9 M-inode / 933 G dataset
+
+Investigated after per-file backups of `/opt/seafile` proved
+unwalkable (restic seed: ~20 M objects remaining at ~66 files/s).
+Findings (seafile-server **12.0.14**, binary install, upgraded from
+8.0 → 9.0 → 12.0):
+
+- **GC has never run** — no `gc.log` ever, no cron in the container.
+  `seaf-gc` is the *only* mechanism that deletes blocks/fs/commits
+  objects; without it nothing is ever reclaimed.
+- **13 ghost libraries**: `storage/commits/` holds 30 library dirs;
+  the DB has 17 live repos and an empty trash. Deleted libraries'
+  object stores persist until `seaf-gc --rm-deleted`.
+- **Unlimited history on 16 of 17 repos**: only one library (the
+  240 G one) has a retention limit (30 days, `RepoHistoryLimit`).
+  Everything else keeps every version of every file since creation.
+- Sizes: live repo data ≈ 710 G (DB `RepoSize`) vs **867 G** in
+  `blocks/` → ≥150 G dead blocks; the rest of the dataset is the
+  fs/commits object swarm (~5 M commits objects counted; fs likely
+  15 M+, sub-4K each) — the thing that makes every per-file walk
+  take days.
+
+References: <https://manual.seafile.com/12.0/maintain/seafile_gc/>
+
+## Procedure
+
+### 0. Safety net (host, instant)
+
+```sh
+zfs snapshot data/containers/seafile@pre-gc      # rollbackable; delete after validation
+```
+
+Leg-1 replica on sdb is the second net (crontab currently disabled —
+the replica is pre-GC state, which is exactly what we want here).
+
+### 1. Default history retention (inside the container)
+
+`/opt/seafile/conf/seafile.conf`:
+
+```ini
+[history]
+keep_days = 90
+```
+
+Applies as the default for all libraries without a per-library
+setting (the existing per-library 30 d row stays). Users can still
+override per library in the web UI if
+`ENABLE_REPO_HISTORY_SETTING = True` in `seahub_settings.py`.
+
+### 2. GC — offline, maintenance window
+
+CE's GC requires seafile stopped. On this pathological store expect
+**hours** (it traverses fs/commits on a cold HDD); the dry-run sizes
+the window. Inside the container:
+
+```sh
+cd /opt/seafile/seafile-server-latest
+./seahub.sh stop && ./seafile.sh stop     # or the container's systemd units — check systemctl
+
+./seaf-gc.sh --dry-run                    # per-repo report of what would be reclaimed
+./seaf-gc.sh                              # blocks GC, honors keep_days
+./seaf-gc.sh --rm-deleted                 # purge the 13 ghost libraries
+./seaf-gc.sh --rm-fs                      # unreferenced fs objects — the inode bulk
+                                          # (verify flag names: ./seaf-gc.sh --help; manual 12.0)
+
+./seafile.sh start && ./seahub.sh start
+```
+
+### 3. Measure the result (host)
+
+```sh
+df -i /var/lib/incus/storage-pools/data/containers/seafile   # was 24,885,504 inodes
+du -sh .../seafile/rootfs/opt/seafile/seafile-data/storage/blocks   # was 867 G
+```
+
+### 4. Aftermath
+
+- validate seafile works (client sync, web, download a versioned
+  file), then `zfs destroy data/containers/seafile@pre-gc`
+- re-enable the crontab (incus-copy leg first; its next refresh
+  transfers the shrunken dataset)
+- **schedule GC** so this never regrows: quarterly maintenance window
+  (offline GC is the CE price); host-side cron via
+  `incus exec seafile -- ...` or a documented manual ritual
+- re-decide seafile's S3 backup with real post-GC numbers:
+  per-file restic redux if the object count collapses far enough,
+  `zfs send | restic --stdin-from-command` otherwise
+  (see [restic-backup.md](restic-backup.md))
+- then plan the 13.x upgrade (Docker-only — see the incus-compose
+  notes and
+  <https://manual.seafile.com/13.0/upgrade/upgrade_notes_for_13.0.x/>)
+  against a much smaller estate
+
+## Related cleanup while in there
+
+- `logs/seafevents.log.2024-*` and friends — years of rotated logs
+- the old `seafile-server-8.0.7` / `9.0.10` dirs once 12.x is proven