Julien LutranandClaude Fable 5
a0e3a7dd86
- nuc/README.md and ks4/README.md carry the host sections (+ network flows) that lived in the top-level README; links rebased - top README: repo links (doc/scripts on git.lutran.fr), index points at the new per-host pages - cross-repo references now use https://git.lutran.fr/julien/scripts instead of relative ../scripts paths that resolve nowhere - plakar-s3-data.md and plakar-incus-integration.md marked SUPERSEDED / RETIRED with pointers to restic-backup.md; their measurements and rationale kept - install.md, local-backup-cron.md, incus-copy.md: crontab sections updated to the live schedule (01:00 replicas, 05:00 restic, Sun maintenance); retired legs labelled as such - restic-backup.md: status live, cutover recorded, post-GC memory estimate, seed plan dated - seafile-gc.md: online GC noted, stale 'crons commented out' removed - ks2/: what-ks2-does-today rewritten (nothing writes to it any more), legs table and gates reflect restic, decommission steps updated - db-exclude replaces the plakar-era config name (script keeps a fallback) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
153 lines
6.7 KiB
Markdown
153 lines
6.7 KiB
Markdown
# seafile: garbage collection + history retention
|
||
|
||
Status: **GC executed 2026-08-26/27.** Results:
|
||
|
||
| metric | before | after |
|
||
|---|---|---|
|
||
| dataset inodes | 24,885,504 | **515,842** (after Documents recreation + final `-r`, 2026-08-28) |
|
||
| `storage/blocks/` | 867 G | **632 G** (−235 G) |
|
||
| fs objects | ~19.3 M | ~100 k (19,207,400 removed) |
|
||
| ghost library stores | 13 | 0 (blocks/fs eaten by GC; 4 empty commit husks removed manually — `--rm-deleted` couldn't see them, they predate `GarbageRepos`) |
|
||
| wall time | | 16 min (blocks) + 0.2 s (`-r`) + **21 h 33 m** (`-R`) |
|
||
|
||
**Root-root-cause found**: one library — Documents
|
||
(`c4ec7247`, 2.9 G data) — held **19.02 M fs objects and ~4.9 M
|
||
commits**: a client has been committing at extreme frequency for
|
||
years. The ~4.9 M commit objects survive GC (CE never deletes commit
|
||
objects) and are most of the remaining inodes. Remedy: identify and
|
||
fix the committing client, then **recreate the library** (sync down
|
||
current state → new library → delete old → GC `-r`, which works for
|
||
12.x-era deletions) — that retires the commit store entirely and
|
||
takes the dataset to ~600 k inodes, at which point per-file backup of
|
||
seafile becomes ordinary.
|
||
|
||
(The ks4 backup crons were paused during the GC window and are back
|
||
on since 2026-08-27.)
|
||
|
||
## Root cause of the 24.9 M-inode / 933 G dataset
|
||
|
||
Investigated after per-file backups of `/opt/seafile` proved
|
||
unwalkable (restic seed: ~20 M objects remaining at ~66 files/s).
|
||
Findings (seafile-server **12.0.14**, binary install, upgraded from
|
||
8.0 → 9.0 → 12.0):
|
||
|
||
- **GC has never run** — no `gc.log` ever, no cron in the container.
|
||
`seaf-gc` is the *only* mechanism that deletes blocks/fs/commits
|
||
objects; without it nothing is ever reclaimed.
|
||
- **13 ghost libraries**: `storage/commits/` holds 30 library dirs;
|
||
the DB has 17 live repos and an empty trash. Deleted libraries'
|
||
object stores persist until `seaf-gc --rm-deleted`.
|
||
- **Unlimited history on 16 of 17 repos**: only one library (the
|
||
240 G one) has a retention limit (30 days, `RepoHistoryLimit`).
|
||
Everything else keeps every version of every file since creation.
|
||
- Sizes: live repo data ≈ 710 G (DB `RepoSize`) vs **867 G** in
|
||
`blocks/` → ≥150 G dead blocks; the rest of the dataset is the
|
||
fs/commits object swarm (~5 M commits objects counted; fs likely
|
||
15 M+, sub-4K each) — the thing that makes every per-file walk
|
||
take days.
|
||
|
||
References: <https://manual.seafile.com/12.0/maintain/seafile_gc/>
|
||
|
||
## Procedure
|
||
|
||
### 0. Safety net (host, instant)
|
||
|
||
```sh
|
||
zfs snapshot data/containers/seafile@pre-gc # rollbackable; delete after validation
|
||
```
|
||
|
||
Leg-1 replica on sdb is the second net (crontab currently disabled —
|
||
the replica is pre-GC state, which is exactly what we want here).
|
||
|
||
### 1. Default history retention (inside the container)
|
||
|
||
`/opt/seafile/conf/seafile.conf`:
|
||
|
||
```ini
|
||
[history]
|
||
keep_days = 90
|
||
```
|
||
|
||
Applies as the default for all libraries without a per-library
|
||
setting (the existing per-library 30 d row stays). Users can still
|
||
override per library in the web UI if
|
||
`ENABLE_REPO_HISTORY_SETTING = True` in `seahub_settings.py`.
|
||
|
||
### 2. GC — maintenance window
|
||
|
||
⚠️ Superseded by experience: with the **MySQL backend seafile runs
|
||
*online* GC** (the runs print "use online GC"), so stopping the
|
||
services is not required — the recurring cron below does not. The
|
||
2026-08-26 run stopped them anyway, out of caution. What follows is
|
||
that run's procedure.
|
||
|
||
CE's GC traditionally requires seafile stopped. On this pathological store expect
|
||
**hours** (it traverses fs/commits on a cold HDD); the dry-run sizes
|
||
the window. Inside the container:
|
||
|
||
```sh
|
||
cd /opt/seafile/seafile-server-latest
|
||
./seahub.sh stop && ./seafile.sh stop # or the container's systemd units — check systemctl
|
||
|
||
./seaf-gc.sh --dry-run # per-repo report of what would be reclaimed
|
||
./seaf-gc.sh # blocks GC, honors keep_days
|
||
./seaf-gc.sh --rm-deleted # purge the 13 ghost libraries
|
||
./seaf-gc.sh --rm-fs # unreferenced fs objects — the inode bulk
|
||
# (verify flag names: ./seaf-gc.sh --help; manual 12.0)
|
||
|
||
./seafile.sh start && ./seahub.sh start
|
||
```
|
||
|
||
### 3. Measure the result (host)
|
||
|
||
```sh
|
||
df -i /var/lib/incus/storage-pools/data/containers/seafile # was 24,885,504 inodes
|
||
du -sh .../seafile/rootfs/opt/seafile/seafile-data/storage/blocks # was 867 G
|
||
```
|
||
|
||
### 4. Aftermath
|
||
|
||
- validate seafile works (client sync, web, download a versioned
|
||
file), then `zfs destroy data/containers/seafile@pre-gc`
|
||
- re-enable the crontab (incus-copy leg first; its next refresh
|
||
transfers the shrunken dataset)
|
||
- **recurring GC** (done 2026-08-27 design): the MySQL backend runs
|
||
GC *online* (verified — the runs printed "use online GC" and `-r`
|
||
ran with services up), so no stop/start is needed. Host crontab:
|
||
```cron
|
||
# monthly dry-run report (cheap health check, read-only)
|
||
0 6 1 * * incus exec seafile -- sh -c "cd /opt/seafile/seafile-server-latest && ./seaf-gc.sh -D" >> /var/log/seafile-gc.log 2>&1
|
||
# quarterly real GC: blocks + trashed repos + fs objects
|
||
0 6 15 1,4,7,10 * incus exec seafile -- sh -c "cd /opt/seafile/seafile-server-latest && ./seaf-gc.sh && ./seaf-gc.sh -r && ./seaf-gc.sh -R" >> /var/log/seafile-gc.log 2>&1
|
||
```
|
||
At the post-cleanup scale (~120 k live files) the full sequence is
|
||
minutes, not hours. Keep it clear of the 01:00/05:00 backup
|
||
slots.
|
||
- 2026-08-27 follow-ups executed: 8 old libraries deleted by user →
|
||
trash emptied → `seaf-gc -r` purged their stores (18 m, blocks now
|
||
590 G, −277 G total); 4 pre-`GarbageRepos` husks removed manually;
|
||
Documents to be recreated from a local re-sync (retires its ~4.9 M
|
||
fossil commit objects → expected final ~600 k inodes)
|
||
- re-decide seafile's S3 backup with real post-GC numbers:
|
||
per-file restic redux if the object count collapses far enough,
|
||
`zfs send | restic --stdin-from-command` otherwise
|
||
(see [restic-backup.md](restic-backup.md))
|
||
- then plan the 13.x upgrade (Docker-only — see the incus-compose
|
||
notes and
|
||
<https://manual.seafile.com/13.0/upgrade/upgrade_notes_for_13.0.x/>)
|
||
against a much smaller estate
|
||
|
||
## Related cleanup while in there
|
||
|
||
- `logs/seafevents.log.2024-*` and friends — years of rotated logs
|
||
- the old `seafile-server-8.0.7` / `9.0.10` dirs once 12.x is proven
|
||
|
||
## Closure (2026-08-28)
|
||
|
||
Documents recreated from a local re-sync, old library trashed → trash
|
||
emptied → `seaf-gc.sh -r` (78 m) retired its ~4.9 M fossil commit
|
||
objects. **Final: 515,842 inodes (from 24,885,504, −98 %), blocks
|
||
590 G (from 867 G), 8 libraries / ~120 k live files / 703 GB.** The
|
||
`@pre-gc` snapshot can be destroyed once the new Documents library has
|
||
proven itself for a few days (it pins the freed space until then).
|