julien/doc
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

doc: split per-host READMEs, gitea cross-repo links, consistency pass

Browse Source 
- nuc/README.md and ks4/README.md carry the host sections (+ network
  flows) that lived in the top-level README; links rebased
- top README: repo links (doc/scripts on git.lutran.fr), index points
  at the new per-host pages
- cross-repo references now use https://git.lutran.fr/julien/scripts
  instead of relative ../scripts paths that resolve nowhere
- plakar-s3-data.md and plakar-incus-integration.md marked SUPERSEDED
  / RETIRED with pointers to restic-backup.md; their measurements and
  rationale kept
- install.md, local-backup-cron.md, incus-copy.md: crontab sections
  updated to the live schedule (01:00 replicas, 05:00 restic, Sun
  maintenance); retired legs labelled as such
- restic-backup.md: status live, cutover recorded, post-GC memory
  estimate, seed plan dated
- seafile-gc.md: online GC noted, stale 'crons commented out' removed
- ks2/: what-ks2-does-today rewritten (nothing writes to it any more),
  legs table and gates reflect restic, decommission steps updated
- db-exclude replaces the plakar-era config name (script keeps a
  fallback)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Julien Lutran
2026-08-28 16:04:07 +02:00
co-authored by Claude Fable 5
parent 80f0219716
commit a0e3a7dd86
16 changed files with 246 additions and 193 deletions
Download Patch File Download Diff File Expand all files Collapse all files
ks4/README.md
+55
View File
@@ -0,0 +1,55 @@
# Prod: ks4
Incus host at OVH — public-facing self-hosted services.
- Debian 13, Xeon D-1521 / 32 GiB, 2× 6 TB (OS on mdraid RAID1,
ZFS pool `data` on `sda5`) — setup & rebuild:
[install.md](install.md)
- SSH: `ssh -i id_rsa_claude -p 2233 julien@193.70.35.17`
(`julien` has passwordless `sudo incus …`)
- Instances are on a NAT bridge `incusbr0` (192.168.1.0/24); public
ingress via Incus proxy devices. `gateway` (nginx) is the sole HTTP/S
entry point and fans out to the app containers.
- Services: gateway, mail, nextcloud, seafile, git, freshrss, bitwarden,
outline, login (SSO), wireguard, + more — full table in
[install.md](install.md).
- ⚠️ The ZFS `data` pool is single-disk (not mirrored); durability
rests on nightly cron jobs — 01:00 `incus copy --refresh` of all
instances to the local `backup` pool on sdb5, then the instance leg
to S3; 05:00 restic (DB dumps + data trees) to S3; nuc pulls the
replicas over WireGuard. Full picture and restore procedures:
**[backup-strategy.md](../backup-strategy.md)**
([local-backup-cron.md](local-backup-cron.md),
[incus-copy.md](incus-copy.md),
[restic-backup.md](restic-backup.md)).
## Network flows (nuc ↔ ks4)
```
nuc — home LAN 192.168.0.0/24 ks4 — OVH 193.70.35.17
+-----------------------------------+ +-------------------------------------+
| | | |
| host: wg-ks4 (10.8.0.20) | | [wireguard] 192.168.1.18 |
| incus remote "ks4" ------+--WG-->| wg0 10.8.0.1/24, udp 51845 |
| pull ks4:* -> pool ks4backup | udp | | masquerade -> eth0 |
| on usb4t [pending FTTH seed] | 51845 | | |
| | | +-> incus API 192.168.1.1:8443 |
| [transmission-bt] wg0 (10.8.0.21) | | | (ufw: only from .18) |
| full tunnel 0.0.0.0/0 ------+--WG-->| | |
| kill switch: no default route | udp | +-> WAN egress: torrents + |
| downloads -> /srv/media | 51845 | apt of transmission-bt |
| (usb4t/media, read by jellyfin) | | exit as 193.70.35.17 |
| | | |
| 03:00 instance snapshots | | 03:00 instance snapshots |
| 03:30 incus-copy: all instances | | 01:00 incus-copy: all instances |
| -> project backup, pool | | -> project backup, zpool sdb5 |
| nucbackup (usb4t/backup/nuc) | | then restic instance leg -> S3 |
| 05:00 pull ks4:* -> ks4backup | | 05:00 restic: DB dumps + data |
| [pending FTTH] | | trees -> S3 (restic-data) |
| 05:30 apt upgrade all containers | | Sun 14:00 restic maintenance |
+-----------------------------------+ +-------------------------------------+
phones/laptops: WG peers 10.8.0.2-3 reach 192.168.1.x through the same endpoint
```
Both tunnels initiate **from** nuc (home NAT, dynamic IP) toward ks4's
fixed endpoint; ks4's incus API is never exposed to the internet.
ks4/incus-copy.md
+5 -3
View File
@@ -1,7 +1,7 @@
# incus-copy — ks4 instance replication
Incremental replication of all ks4 Incus instances via
[`scripts/incus-copy.sh`](../scripts/incus-copy.sh)
[`scripts/incus-copy.sh`](https://git.lutran.fr/julien/scripts/src/branch/main/incus-copy.sh)
(`incus copy --refresh`: ZFS-incremental, produces ready-to-start
replicas instead of tarballs). Two legs replace the historical nightly
push to `ks2` (decommissioning):
@@ -35,7 +35,7 @@ incus-copy.sh -p <dest-project> -s <dest-pool> # local
`--refresh-exclude-older` keeps expired snapshots from being re-sent.
- Consistency: copies of running instances are crash-consistent;
databases may need recovery on restore. Snapshot history mitigates,
and [`incus-backup.sh`](../scripts/incus-backup.sh) (proper
and [`incus-backup.sh`](https://git.lutran.fr/julien/scripts/src/branch/main/incus-backup.sh) (proper
`mariadb-dump`s + selected paths, manifest-driven) covers the DBs.
- **VMs**: `--refresh` re-sends a VM's **full block volume** every run
unless source and replica share a common snapshot — containers diff
@@ -63,7 +63,9 @@ Cron (root on ks4) — replaces both ks2 jobs:
```cron
0 1 * * * /root/scripts/incus-copy.sh -p backup -s backup >> /var/log/incus-copy.log 2>&1
0 4 * * * /root/scripts/incus-backup.sh -f /root/scripts/incus-backup.db -s data >> /var/log/incus-backup.log 2>&1
# retired 2026-08-28 — dumps + data trees are restic's job (05:00),
# see ../backup-strategy.md and restic-backup.md
#0 4 * * * /root/scripts/incus-backup.sh -f /root/scripts/incus-backup.db -s data >> /var/log/incus-backup.log 2>&1
```
(`incus-backup.sh` without `-d` now writes locally to
ks4/install.md
+22 -12
View File
@@ -142,22 +142,32 @@ All in the `default` project, on the `default` profile
## Backups
> **Migration in progress** (2026-08-09): the `ks2` destination below is
> being replaced by a local `backup` zpool on `sdb5` + off-site pull
> from nuc — plan and setup in [incus-copy.md](incus-copy.md). The
> cron blocks here describe what runs **today**; update on cutover.
Two **root** cron jobs (this repo's [`scripts/`](../scripts/) are the
copies that run here) — the primary safety net for the non-mirrored
`data` pool:
Full picture, schedule and restore procedures:
**[../backup-strategy.md](../backup-strategy.md)**. Drivers live in the
[scripts repo](https://git.lutran.fr/julien/scripts) and are deployed
to `/root/scripts`. Current **root** cron jobs — the safety net for the
non-mirrored `data` pool:
```cron
0 1 * * * /root/scripts/incus-copy.sh -d ks2 -m push
0 4 * * * /root/scripts/incus-backup.sh -d 164.132.173.57 -u rsyncbackup \
-i /root/.ssh/id_rsyncbackup -p 2233 \
-f /root/scripts/incus-backup.db -s data
0 1 * * * /root/scripts/incus-copy.sh -p backup -s backup >> /var/log/incus-copy.log 2>&1
0 5 * * * /root/scripts/restic-backup.sh >> /var/log/restic-backup.log 2>&1
0 14 * * 0 /root/scripts/restic-maintenance.sh >> /var/log/restic-maintenance.log 2>&1
```
- **01:00 `incus-copy.sh`** → replicas of every instance in project
`backup` on the sdb5 pool ([incus-copy.md](incus-copy.md),
[local-backup-cron.md](local-backup-cron.md)); the S3 instance leg
chains here once seeded.
- **05:00 `restic-backup.sh`** → auto-discovered DB dumps + the data
trees listed in
[`scripts/restic-paths`](https://git.lutran.fr/julien/scripts/src/branch/main/restic-paths),
into the `restic-data` bucket ([restic-backup.md](restic-backup.md)).
- **Sun 14:00 `restic-maintenance.sh`** → prune + integrity checks.
Historical (retired 2026-08): the `ks2` push and `incus-backup.sh`
rsync leg, and the plakar data leg
([plakar-s3-data.md](plakar-s3-data.md)).
- **01:00 — `incus-copy.sh` → `ks2`**: differential `incus copy … \
--refresh --refresh-exclude-older --mode push` of every *running*
instance to remote incus host `ks2` (a near-live replica). `ks2` must
ks4/local-backup-cron.md
+10 -15
View File
@@ -1,28 +1,23 @@
# local backup cron on ks4 (leg 1) — re-enabled
# local backup cron on ks4 (leg 1)
Done 2026-08-22. Context: **both** ks4 backup crons had been commented
out — no backup of any kind was running.
Enabled 2026-08-22 (both ks4 backup crons had been commented out — no
backup of any kind was running). Overview of all legs:
[../backup-strategy.md](../backup-strategy.md).
## What runs now (root crontab on ks4)
```cron
0 1 * * * /root/scripts/incus-copy.sh -p backup -s backup >> /var/log/incus-copy.log 2>&1
0 4 * * * /root/scripts/incus-backup.sh -d 164.132.173.57 -u rsyncbackup -i /root/.ssh/id_rsyncbackup -p 2233 -f /root/scripts/incus-backup.db -s data >> /var/log/incus-backup.log 2>&1
```
- `01:00` — **local leg**: `incus copy --refresh` of all instances
into project `backup` on the sdb5 `backup` zpool (replaces the old
push to ks2; see [incus-copy.md](incus-copy.md)).
- `04:00` — **stopgap**: `incus-backup.sh` rsync to ks2 kept alive
(manifest pruned of `spot`) until the plakar S3 leg replaces it —
then this line goes away as part of
[decommissioning ks2](../ks2/plan.md). ⚠️ keep
`scripts/incus-backup.db` in this repo in sync with
`/root/scripts/incus-backup.db` on ks4.
Both were also run once by hand (in a `screen`) on 2026-08-22 to
refresh the backups immediately. The old push-to-ks2 copy line stays
commented out — replica refreshes to ks2 are retired.
push to ks2; see [incus-copy.md](incus-copy.md)). The S3 instance
leg ([restic-backup.md](restic-backup.md) §6) chains onto this same
line once seeded — it reads the replicas this job just refreshed.
- The rsync stopgap to ks2 (`incus-backup.sh`, 04:00) and the plakar
data leg (04:30) are both **retired**; S3 backups are restic's job
at 05:00.
## Cleanup done the same day
ks4/plakar-incus-integration.md
+11 -1
View File
@@ -1,4 +1,14 @@
# plakar incus integration — design notes
# plakar incus integration — design notes (RETIRED)
> **Historical.** The importer works (smoke-tested against ks4's
> `backup` project) but production moved to restic, which needs no
> plugin: `incus file mount` exposes any instance — including stopped
> replicas — over the same per-instance sftp API
> ([restic-backup.md](restic-backup.md) §6). Code lives on at
> <https://github.com/jlutran/plakar-integration-incus>; the analysis
> below (per-file vs image, replica-vs-live consistency, the
> seek-bound wall) is what shaped the restic design.
Status: **importer scaffolded and compiling 2026-08-23** in
`perso/plakar/integration-incus` (walker modeled on the official sftp
ks4/plakar-s3-data.md
+13 -2
View File
@@ -1,4 +1,15 @@
# plakar data + DB backups on ks4 (leg 3)
# plakar data + DB backups on ks4 (leg 3) — SUPERSEDED
> **Historical.** This leg ran 2026-08-22 → 08-28 and was replaced by
> restic: see **[restic-backup.md](restic-backup.md)**. plakar's
> incremental cost scaled with tree size rather than churn
> ([PlakarKorp/plakar#2338](https://github.com/PlakarKorp/plakar/issues/2338)).
> The `plakar-data` kloset and the plakar install are deliberately kept
> until that issue concludes (the report offers debug runs against that
> repository); the `plakar-incus` bucket can be deleted. Kept here for
> the measurements and the design rationale, which the restic doc
> builds on.
Status: **live since 2026-08-24** — S3 kloset
(`lutran-ks4-plakar-data`) seeded (dumps + 9 fs sources), restore
@@ -19,7 +30,7 @@ leg 1's job. Direct-to-S3 keeps the incremental behavior — the
skip-unchanged baseline is the parent snapshot's VFS read from the
kloset (plakar caches repository metadata locally; v1.1.4's dirpack
prefetcher targets exactly this remote-backend walk). Driver script:
[`scripts/plakar-backup.sh`](../../scripts/plakar-backup.sh)
[`scripts/plakar-backup.sh`](https://git.lutran.fr/julien/scripts/src/branch/main/plakar-backup.sh)
(flock, incus DB dumps, iterates a sources file, prune + maintenance;
its `-k` takes any `plakar at` target, so `-k @s3` — a throwaway
local kloset, e.g. `-k /backup/plakar-test`, works for testing before
ks4/restic-backup.md
+23 -20
View File
@@ -1,10 +1,12 @@
# restic backups on ks4 — the S3 leg
Status: **decided 2026-08-25 — restic replaces plakar as the S3
backup tool** (evaluation first: parallel run + comparison, see §8).
Final architecture: **two tools** — `incus copy` for replication
(local sdb, nuc after FTTH, ks2 push as interim), **restic** for
everything S3.
Status: **live since 2026-08-28** — restic replaced plakar as the S3
backup tool (plakar crons off, its kloset kept for
[#2338](https://github.com/PlakarKorp/plakar/issues/2338); see §8).
Architecture: **two tools** — `incus copy` for replication (local sdb,
nuc after FTTH), **restic** for everything S3. Data leg (`restic-data`)
runs nightly at 05:00; the instance leg (`restic-incus`, §6) is written
and awaits its first seed.
Why: plakar's incremental cost scales with tree size, not churn
(~800 MiB parent metadata re-read from S3 + hours of CPU per night on
@@ -58,9 +60,10 @@ property).
### Memory (the real constraint on this 23/31 GiB-used box)
- Index RAM ≈ 200–300 B/blob → combined `restic-data` index
~1–1.5 GiB per operation (nextcloud ~1 M blobs + seafile ~2–4 M);
prune peaks 2–3×. Accepted with the merged layout (§1) — measure
peak RSS during the evaluation week (`/usr/bin/time -v`).
~1 GiB per operation; prune peaks 2–3×. The seafile side of that
estimate collapsed after its garbage collection (24.9 M → 0.5 M
objects, [seafile-gc.md](seafile-gc.md)). Measure peak RSS with
`/usr/bin/time -v` if the box ever feels tight.
- `GOGC=20` trades CPU for a smaller Go heap — enable if the seafile
prune ever pressures the box.
@@ -140,7 +143,7 @@ for r in restic-data restic-incus; do
done
```
## 4. Seed plan
## 4. Seed plan (executed 2026-08-25 → 08-28)
1. **ARC floor first** (helps every walk on the box):
```sh
@@ -252,18 +255,18 @@ restic -r s3:…/restic-data restore latest --target /backup/restore-test --incl
diff -r … && rm -rf /backup/restore-test
```
## 8. Evaluation protocol + plakar retirement
## 8. Cutover (done) + what is left
- **Parallel week**: plakar 04:30 cron keeps running untouched;
restic runs at 05:00 on the same data. Same nights, same churn →
direct comparison (wall, bytes read from store, bytes written, peak
RSS via `/usr/bin/time -v`) against plakar's measured table in
[plakar-s3-data.md](plakar-s3-data.md). Restore-speed compared on
the same 10 GiB dumps tree (plakar measured: 39 min).
- **Cutover** (after the comparison confirms + restore test passes):
drop the plakar 04:30 + Sunday-check crons; 01:00 becomes
`incus-copy ; restic instances phase` (or stays in the 05:00 run).
Update plan.md legs table.
- **Cutover 2026-08-28**: plakar's 04:30 and Sunday crons are off;
restic's 05:00 data leg is the S3 backup. The formal parallel-week
comparison was cut short — the evidence had already settled it
(plakar: ~800 MiB metadata re-read and 3–4 h per night on the
nextcloud tree, churn-independent; restic: seed at 60–115 MiB/s,
local-cache incrementals). plakar's measurements stay in
[plakar-s3-data.md](plakar-s3-data.md).
- **Left to do**: seed `restic-incus` (§6) and chain it onto the 01:00
line; restore test on the complete `restic-data`; then the ks2
decommission gates ([../ks2/plan.md](../ks2/plan.md)).
- **plakar retirement, one exception**: keep plakar installed and
keep the `plakar-data` kloset until
[#2338](https://github.com/PlakarKorp/plakar/issues/2338) concludes
ks4/seafile-gc.md
+11 -5
View File
@@ -21,8 +21,8 @@ current state → new library → delete old → GC `-r`, which works for
takes the dataset to ~600 k inodes, at which point per-file backup of
seafile becomes ordinary.
All crontab backup jobs on ks4 remain commented out until restart +
validation completes.
(The ks4 backup crons were paused during the GC window and are back
on since 2026-08-27.)
## Root cause of the 24.9 M-inode / 933 G dataset
@@ -73,9 +73,15 @@ setting (the existing per-library 30 d row stays). Users can still
override per library in the web UI if
`ENABLE_REPO_HISTORY_SETTING = True` in `seahub_settings.py`.
### 2. GC — offline, maintenance window
### 2. GC — maintenance window
CE's GC requires seafile stopped. On this pathological store expect
⚠️ Superseded by experience: with the **MySQL backend seafile runs
*online* GC** (the runs print "use online GC"), so stopping the
services is not required — the recurring cron below does not. The
2026-08-26 run stopped them anyway, out of caution. What follows is
that run's procedure.
CE's GC traditionally requires seafile stopped. On this pathological store expect
**hours** (it traverses fs/commits on a cold HDD); the dry-run sizes
the window. Inside the container:
@@ -115,7 +121,7 @@ du -sh .../seafile/rootfs/opt/seafile/seafile-data/storage/blocks # was 867 G
0 6 15 1,4,7,10 * incus exec seafile -- sh -c "cd /opt/seafile/seafile-server-latest && ./seaf-gc.sh && ./seaf-gc.sh -r && ./seaf-gc.sh -R" >> /var/log/seafile-gc.log 2>&1
```
At the post-cleanup scale (~120 k live files) the full sequence is
minutes, not hours. Keep it clear of the 01:00/04:30/05:00 backup
minutes, not hours. Keep it clear of the 01:00/05:00 backup
slots.
- 2026-08-27 follow-ups executed: 8 old libraries deleted by user →
trash emptied → `seaf-gc -r` purged their stores (18 m, blocks now