# seafile: garbage collection + history retention

Status: **GC executed 2026-08-26/27.** Results:

| metric | before | after |
|---|---|---|
| dataset inodes | 24,885,504 | **515,842** (after Documents recreation + final `-r`, 2026-08-28) |
| `storage/blocks/` | 867 G | **632 G** (−235 G) |
| fs objects | ~19.3 M | ~100 k (19,207,400 removed) |
| ghost library stores | 13 | 0 (blocks/fs eaten by GC; 4 empty commit husks removed manually — `--rm-deleted` couldn't see them, they predate `GarbageRepos`) |
| wall time | | 16 min (blocks) + 0.2 s (`-r`) + **21 h 33 m** (`-R`) |

**Root-root-cause found**: one library — Documents
(`c4ec7247`, 2.9 G data) — held **19.02 M fs objects and ~4.9 M
commits**: a client has been committing at extreme frequency for
years. The ~4.9 M commit objects survive GC (CE never deletes commit
objects) and are most of the remaining inodes. Remedy: identify and
fix the committing client, then **recreate the library** (sync down
current state → new library → delete old → GC `-r`, which works for
12.x-era deletions) — that retires the commit store entirely and
takes the dataset to ~600 k inodes, at which point per-file backup of
seafile becomes ordinary.

(The ks4 backup crons were paused during the GC window and are back
on since 2026-08-27.)

## Root cause of the 24.9 M-inode / 933 G dataset

Investigated after per-file backups of `/opt/seafile` proved
unwalkable (restic seed: ~20 M objects remaining at ~66 files/s).
Findings (seafile-server **12.0.14**, binary install, upgraded from
8.0 → 9.0 → 12.0):

- **GC has never run** — no `gc.log` ever, no cron in the container.
  `seaf-gc` is the *only* mechanism that deletes blocks/fs/commits
  objects; without it nothing is ever reclaimed.
- **13 ghost libraries**: `storage/commits/` holds 30 library dirs;
  the DB has 17 live repos and an empty trash. Deleted libraries'
  object stores persist until `seaf-gc --rm-deleted`.
- **Unlimited history on 16 of 17 repos**: only one library (the
  240 G one) has a retention limit (30 days, `RepoHistoryLimit`).
  Everything else keeps every version of every file since creation.
- Sizes: live repo data ≈ 710 G (DB `RepoSize`) vs **867 G** in
  `blocks/` → ≥150 G dead blocks; the rest of the dataset is the
  fs/commits object swarm (~5 M commits objects counted; fs likely
  15 M+, sub-4K each) — the thing that makes every per-file walk
  take days.

References: <https://manual.seafile.com/12.0/maintain/seafile_gc/>

## Procedure

### 0. Safety net (host, instant)

```sh
zfs snapshot data/containers/seafile@pre-gc      # rollbackable; delete after validation
```

Leg-1 replica on sdb is the second net (crontab currently disabled —
the replica is pre-GC state, which is exactly what we want here).

### 1. Default history retention (inside the container)

`/opt/seafile/conf/seafile.conf`:

```ini
[history]
keep_days = 90
```

Applies as the default for all libraries without a per-library
setting (the existing per-library 30 d row stays). Users can still
override per library in the web UI if
`ENABLE_REPO_HISTORY_SETTING = True` in `seahub_settings.py`.

### 2. GC — maintenance window

⚠️ Superseded by experience: with the **MySQL backend seafile runs
*online* GC** (the runs print "use online GC"), so stopping the
services is not required — the recurring cron below does not. The
2026-08-26 run stopped them anyway, out of caution. What follows is
that run's procedure.

CE's GC traditionally requires seafile stopped. On this pathological store expect
**hours** (it traverses fs/commits on a cold HDD); the dry-run sizes
the window. Inside the container:

```sh
cd /opt/seafile/seafile-server-latest
./seahub.sh stop && ./seafile.sh stop     # or the container's systemd units — check systemctl

./seaf-gc.sh --dry-run                    # per-repo report of what would be reclaimed
./seaf-gc.sh                              # blocks GC, honors keep_days
./seaf-gc.sh --rm-deleted                 # purge the 13 ghost libraries
./seaf-gc.sh --rm-fs                      # unreferenced fs objects — the inode bulk
                                          # (verify flag names: ./seaf-gc.sh --help; manual 12.0)

./seafile.sh start && ./seahub.sh start
```

### 3. Measure the result (host)

```sh
df -i /var/lib/incus/storage-pools/data/containers/seafile   # was 24,885,504 inodes
du -sh .../seafile/rootfs/opt/seafile/seafile-data/storage/blocks   # was 867 G
```

### 4. Aftermath

- validate seafile works (client sync, web, download a versioned
  file), then `zfs destroy data/containers/seafile@pre-gc`
- re-enable the crontab (incus-copy leg first; its next refresh
  transfers the shrunken dataset)
- **recurring GC** (done 2026-08-27 design): the MySQL backend runs
  GC *online* (verified — the runs printed "use online GC" and `-r`
  ran with services up), so no stop/start is needed. Host crontab:
  ```cron
  # monthly dry-run report (cheap health check, read-only)
  0 6 1 * * incus exec seafile -- sh -c "cd /opt/seafile/seafile-server-latest && ./seaf-gc.sh -D" >> /var/log/seafile-gc.log 2>&1
  # quarterly real GC: blocks + trashed repos + fs objects
  0 6 15 1,4,7,10 * incus exec seafile -- sh -c "cd /opt/seafile/seafile-server-latest && ./seaf-gc.sh && ./seaf-gc.sh -r && ./seaf-gc.sh -R" >> /var/log/seafile-gc.log 2>&1
  ```
  At the post-cleanup scale (~120 k live files) the full sequence is
  minutes, not hours. Keep it clear of the 01:00/05:00 backup
  slots.
- 2026-08-27 follow-ups executed: 8 old libraries deleted by user →
  trash emptied → `seaf-gc -r` purged their stores (18 m, blocks now
  590 G, −277 G total); 4 pre-`GarbageRepos` husks removed manually;
  Documents to be recreated from a local re-sync (retires its ~4.9 M
  fossil commit objects → expected final ~600 k inodes)
- re-decide seafile's S3 backup with real post-GC numbers:
  per-file restic redux if the object count collapses far enough,
  `zfs send | restic --stdin-from-command` otherwise
  (see [restic-backup.md](restic-backup.md))
- then plan the 13.x upgrade (Docker-only — see the incus-compose
  notes and
  <https://manual.seafile.com/13.0/upgrade/upgrade_notes_for_13.0.x/>)
  against a much smaller estate

## Related cleanup while in there

- `logs/seafevents.log.2024-*` and friends — years of rotated logs
- the old `seafile-server-8.0.7` / `9.0.10` dirs once 12.x is proven

## Closure (2026-08-28)

Documents recreated from a local re-sync, old library trashed → trash
emptied → `seaf-gc.sh -r` (78 m) retired its ~4.9 M fossil commit
objects. **Final: 515,842 inodes (from 24,885,504, −98 %), blocks
590 G (from 867 G), 8 libraries / ~120 k live files / 703 GB.** The
`@pre-gc` snapshot can be destroyed once the new Documents library has
proven itself for a few days (it pins the freed space until then).