doc: plakar out of the current architecture, kept as reference

backup-strategy.md now leads with the three target legs (local incus
push to sdb, nuc pull over WireGuard, restic to S3 for DBs and
selected trees) and carries the plakar/ks2 story in a closing History
section. restic-backup.md, install.md, local-backup-cron.md and the
ks2 docs describe the predecessor only as 'the first S3
implementation', with one operational note that plakar is still
installed for issue #2338 reproduction.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Julien Lutran
2026-08-28 16:20:20 +02:00
co-authored by Claude Fable 5
parent b5aef914f5
commit d7ec780747
6 changed files with 78 additions and 64 deletions
+34 -15
View File
@@ -4,25 +4,29 @@ Audience: anyone with root on ks4 who needs to know **what is protected,
where, by which tool, and how to check it**. Details live in the linked where, by which tool, and how to check it**. Details live in the linked
docs; this page is the map. docs; this page is the map.
## Two tools, no more ## Two tools, three legs
| tool | job | why this one | | leg | mechanism | protects |
|---|---|---| |---|---|---|
| **`incus copy --refresh`** | replicate whole instances (ready-to-start copies) | ZFS-incremental, native to the platform, restores are `incus start` | | **local replication** | `incus copy --refresh` → pool `backup` on ks4's second disk (sdb) | instances, against losing the `data` pool |
| **restic** | off-site backups to S3 (files, database dumps, instance trees) | dedup + encryption, local metadata cache → a night costs only the churn ([restic-backup.md](ks4/restic-backup.md)) | | **remote replication** | nuc pulls the same replicas over WireGuard → pool `ks4backup` | instances, against losing ks4 or the site |
| **remote backup** | **restic** → S3 bucket `restic-data`: database dumps + selected filesystem trees | the data itself, versioned and encrypted, independent of every disk above |
plakar was evaluated first and replaced (its nightly cost scaled with Only two tools are involved: `incus copy` (ZFS-incremental, native to
tree size — [PlakarKorp/plakar#2338](https://github.com/PlakarKorp/plakar/issues/2338)); the platform — a restore is `incus start`) and restic (dedup +
the old backup server **ks2 is being retired** (decommission by encryption, local metadata cache, so a night costs only the churn —
Sep 30, 2026 — [ks2/plan.md](ks2/plan.md)): it still holds an ageing [ks4/restic-backup.md](ks4/restic-backup.md)).
copy of the instances until the nuc leg below takes over.
**Status 2026-08-28**: the local replica leg and the S3 data leg are The old backup server **ks2 is being retired** (decommission by
live. The **nuc pull leg is waiting for FTTH** (expected before end of Sep 30, 2026 — [ks2/plan.md](ks2/plan.md)); until FTTH enables the nuc
September); until it lands, the ks2 push is re-enabled as an interim leg it still receives an interim replica push.
off-site copy of the instances. Backing up instance *images* to S3 was
designed and shelved — S3 holds the data, the databases and the incus **Status 2026-08-28**: local replication and the S3 backup leg are
configuration, which is what a rebuild needs. live. The **nuc pull leg waits for FTTH** (expected before end of
September); until then the ks2 push stands in. Backing up whole
instance *images* to S3 was considered and left out — S3 holds the
data, the databases and the incus configuration, which is what a
rebuild needs.
## The map ## The map
@@ -120,3 +124,18 @@ New instances are picked up automatically by the replica and
instance legs (all instances, opt-out only). Databases are instance legs (all instances, opt-out only). Databases are
auto-discovered. Only **data trees** need one line in auto-discovered. Only **data trees** need one line in
[`restic-paths`](https://git.lutran.fr/julien/scripts/src/branch/main/restic-paths) — see [new-container.md](new-container.md). [`restic-paths`](https://git.lutran.fr/julien/scripts/src/branch/main/restic-paths) — see [new-container.md](new-container.md).
## History (reference only)
The road to the setup above, kept for the measurements rather than for
operations: an rsync-based leg to ks2
([ks2/plan.md](ks2/plan.md)) and a first S3 implementation with
**plakar** ([ks4/plakar-s3-data.md](ks4/plakar-s3-data.md), plus a
self-written incus connector,
[ks4/plakar-incus-integration.md](ks4/plakar-incus-integration.md)).
plakar was dropped because its nightly cost scaled with the size of
the tree rather than with the churn
([PlakarKorp/plakar#2338](https://github.com/PlakarKorp/plakar/issues/2338)).
It is still installed on ks4 and its `plakar-data` bucket still exists
— kept only so the upstream issue can be reproduced; nothing schedules
it any more.
+4 -4
View File
@@ -55,8 +55,8 @@ renewal date.
- [ ] Retire `scripts/incus-backup.sh` + `incus-backup.db` in the repo - [ ] Retire `scripts/incus-backup.sh` + `incus-backup.db` in the repo
(superseded by [`restic-backup.sh`](https://git.lutran.fr/julien/scripts/src/branch/main/restic-backup.sh)) — (superseded by [`restic-backup.sh`](https://git.lutran.fr/julien/scripts/src/branch/main/restic-backup.sh)) —
keep for reference, note the retirement in their headers keep for reference, note the retirement in their headers
- [ ] `login` container cleanup: the native postgres still holds the - [ ] `login` container cleanup: its **native** postgres still holds
pre-migration `outline` DB — `dropdb outline`, remove the the pre-migration `outline` database (live Outline uses the
package (noted in dockerised `outline-postgres-1`) — `dropdb outline` and remove
[plakar-s3-data.md](../ks4/plakar-s3-data.md) §3) the native postgres package
- [ ] Mark the ks2 line in `plan.md` done, date it - [ ] Mark the ks2 line in `plan.md` done, date it
+3 -3
View File
@@ -61,9 +61,9 @@ when nuc takes over.
2. ~~[local-backup-cron.md](../ks4/local-backup-cron.md)~~ — **done 2. ~~[local-backup-cron.md](../ks4/local-backup-cron.md)~~ — **done
2026-08-22**: leg 1 cron re-enabled, ks2 rsync kept as stopgap; 2026-08-22**: leg 1 cron re-enabled, ks2 rsync kept as stopgap;
`livetrail` verified present in the `backup` project `livetrail` verified present in the `backup` project
3. ~~plakar S3 leg~~**replaced by restic** 3. ~~first S3 leg~~**restic** is the S3 tool
([restic-backup.md](../ks4/restic-backup.md), live 2026-08-28; ([restic-backup.md](../ks4/restic-backup.md), live 2026-08-28; the
plakar's own doc kept as history) predecessor's doc is kept as reference)
4. ~~instance leg to S3~~**shelved 2026-08-28**: instances are 4. ~~instance leg to S3~~**shelved 2026-08-28**: instances are
protected by replication (sdb + nuc/ks2), their data and configs by protected by replication (sdb + nuc/ks2), their data and configs by
`restic-data` `restic-data`
+4 -3
View File
@@ -164,9 +164,10 @@ non-mirrored `data` pool:
into the `restic-data` bucket ([restic-backup.md](restic-backup.md)). into the `restic-data` bucket ([restic-backup.md](restic-backup.md)).
- **Sun 14:00 `restic-maintenance.sh`** → prune + integrity checks. - **Sun 14:00 `restic-maintenance.sh`** → prune + integrity checks.
Historical (retired 2026-08): the `ks2` push and `incus-backup.sh` A `ks2` interim replica push (`incus-copy.sh -d ks2 -m push`, 02:00)
rsync leg, and the plakar data leg runs until the nuc pull leg is seeded. Retired 2026-08: the
([plakar-s3-data.md](plakar-s3-data.md)). `incus-backup.sh` rsync leg and the first S3 implementation
([plakar-s3-data.md](plakar-s3-data.md) — reference only).
- **01:00 — `incus-copy.sh``ks2`**: differential `incus copy … \ - **01:00 — `incus-copy.sh``ks2`**: differential `incus copy … \
--refresh --refresh-exclude-older --mode push` of every *running* --refresh --refresh-exclude-older --mode push` of every *running*
+3 -3
View File
@@ -15,9 +15,9 @@ backup of any kind was running). Overview of all legs:
push to ks2; see [incus-copy.md](incus-copy.md)). An S3 leg for the push to ks2; see [incus-copy.md](incus-copy.md)). An S3 leg for the
instance filesystems was designed and shelved instance filesystems was designed and shelved
([restic-backup.md](restic-backup.md) §6). ([restic-backup.md](restic-backup.md) §6).
- The rsync stopgap to ks2 (`incus-backup.sh`, 04:00) and the plakar - The rsync stopgap to ks2 (`incus-backup.sh`, 04:00) and the first
data leg (04:30) are both **retired**; S3 backups are restic's job S3 implementation (04:30) are both **retired**; S3 backups are
at 05:00. restic's job at 05:00 ([restic-backup.md](restic-backup.md)).
## Cleanup done the same day ## Cleanup done the same day
+30 -36
View File
@@ -1,25 +1,18 @@
# restic backups on ks4 — the S3 leg # restic backups on ks4 — the S3 leg
Status: **live since 2026-08-28** — restic replaced plakar as the S3 Status: **live since 2026-08-28.** This is the S3 backup leg of the
backup tool (plakar crons off, its kloset kept for [strategy](../backup-strategy.md): database dumps + selected
[#2338](https://github.com/PlakarKorp/plakar/issues/2338); see §8). filesystem trees into the `restic-data` bucket, nightly at 05:00.
Architecture: **two tools**`incus copy` for replication (local sdb, Instances themselves are handled by replication (sdb, nuc after FTTH),
nuc after FTTH), **restic** for everything S3. Data leg (`restic-data`) not by this leg — an instance-filesystem leg (`restic-incus`, §6) was
runs nightly at 05:00. The instance leg (`restic-incus`, §6) is written and **shelved**; its script and empty repo stay in place
**shelved** (2026-08-28): instances are covered by replication, so should that change.
duplicating their filesystems into S3 buys little. The script and the
(empty) repo stay in place should that change.
Why: plakar's incremental cost scales with tree size, not churn Why restic: its comparison state lives in a **local cache**, so a
(~800 MiB parent metadata re-read from S3 + hours of CPU per night on night costs a local stat-walk plus the churn — the rsync cost model,
a 500k-file tree — measured, reported as with dedup, compression and encryption on top. (The predecessor and
[PlakarKorp/plakar#2338](https://github.com/PlakarKorp/plakar/issues/2338)). the measurements that motivated the switch are in
restic keeps its comparison state in a local cache: a night costs a [plakar-s3-data.md](plakar-s3-data.md) — reference only.)
local stat-walk plus the churn — the rsync cost model. Neither restic
nor rustic has plakar's source-plugin architecture, but none is
needed: instances are reached via `incus file mount` (FUSE over the
same per-instance sftp API our plakar integration used) — zero
patches to maintain (see §6).
## 1. Repositories — two buckets, two drivers (decided 2026-08-25) ## 1. Repositories — two buckets, two drivers (decided 2026-08-25)
@@ -28,7 +21,7 @@ patches to maintain (see §6).
| `restic-data` | SQL dumps + **all** fs sources (8 small + nextcloud + seafile) | `restic-backup.sh` | | `restic-data` | SQL dumps + **all** fs sources (8 small + nextcloud + seafile) | `restic-backup.sh` |
| `restic-incus` | *(shelved — empty repo, see §6)* the `backup`-project replicas, per-file via mount | `restic-incus-backup.sh` | | `restic-incus` | *(shelved — empty repo, see §6)* the `backup`-project replicas, per-file via mount | `restic-incus-backup.sh` |
Mirrors the plakar-era split (data leg / incus leg). Accepted Two repos so each can be pruned and checked independently. Accepted
trade-off: `restic-data` carries the giants' blobs (~35 M once trade-off: `restic-data` carries the giants' blobs (~35 M once
seafile is in), so **every operation on it loads a ~1.52 GiB seafile is in), so **every operation on it loads a ~1.52 GiB
index** — mitigated by running the whole nightly as a *single* index** — mitigated by running the whole nightly as a *single*
@@ -168,9 +161,9 @@ done
## 5. Nightly driver — `scripts/restic-backup.sh` ## 5. Nightly driver — `scripts/restic-backup.sh`
Same shape as the plakar drivers (flock, env, loud logging). Phases: Shape: flock, env file, loud logging. Phases:
1. **dumps** — lifted verbatim from `plakar-backup.sh`: incus DB 1. **dumps** — incus DB
dumps + auto-discovered MariaDB/PostgreSQL (native and docker) dumps + auto-discovered MariaDB/PostgreSQL (native and docker)
dumps into the dump dir (during the parallel week restic dumps dumps into the dump dir (during the parallel week restic dumps
into its own `/backup/dumps`; carry the archived `geo.sql` over into its own `/backup/dumps`; carry the archived `geo.sql` over
@@ -227,7 +220,7 @@ byte once a year, ~25 GiB read per week.
Neither restic nor rustic accepts source plugins, and forking the Neither restic nor rustic accepts source plugins, and forking the
backup tool is the wrong place to carry a patch. Instead, incus backup tool is the wrong place to carry a patch. Instead, incus
exposes any instance (including **stopped replicas**) as a FUSE mount exposes any instance (including **stopped replicas**) as a FUSE mount
over the same per-instance sftp API the plakar integration used: over the per-instance sftp API:
```sh ```sh
for inst in $(incus list --project backup -c n -f csv); do for inst in $(incus list --project backup -c n -f csv); do
@@ -252,8 +245,7 @@ opt-out via a variable — the no-manifest-drift rule survives the tool
change. Chained after the 01:00 incus-copy once validated, same change. Chained after the 01:00 incus-copy once validated, same
freshness reasoning as before. freshness reasoning as before.
Bonus over plakar: one shared repo dedups the Ubuntu base across all One shared repo would dedup the Ubuntu base across all replicas. Restore granularity: per-file; instance definitions ride
16 replicas. Restore granularity: per-file; instance definitions ride
the dump tree (`incus/<inst>.yaml` + `incus-global-db.sql`). the dump tree (`incus/<inst>.yaml` + `incus-global-db.sql`).
## 7. Restore test (gate) ## 7. Restore test (gate)
@@ -267,24 +259,26 @@ diff -r … && rm -rf /backup/restore-test
## 8. Cutover (done) + what is left ## 8. Cutover (done) + what is left
- **Cutover 2026-08-28**: plakar's 04:30 and Sunday crons are off; Legacy on the box: plakar is still installed and its `plakar-data`
restic's 05:00 data leg is the S3 backup. The formal parallel-week bucket still exists, kept only to reproduce
[#2338](https://github.com/PlakarKorp/plakar/issues/2338) if upstream
asks; no cron references it. The empty `plakar-incus` bucket can be
deleted.
- **Cutover 2026-08-28**: the previous S3 leg's crons are off and
restic's 05:00 run is the S3 backup. The formal parallel-week
comparison was cut short — the evidence had already settled it comparison was cut short — the evidence had already settled it
(plakar: ~800 MiB metadata re-read and 34 h per night on the (predecessor: ~800 MiB metadata re-read and 34 h per night on the
nextcloud tree, churn-independent; restic: seed at 60115 MiB/s, nextcloud tree, churn-independent; restic: seed at 60115 MiB/s,
local-cache incrementals). plakar's measurements stay in local-cache incrementals). Details in
[plakar-s3-data.md](plakar-s3-data.md). [plakar-s3-data.md](plakar-s3-data.md).
- **Left to do**: restore test on the complete `restic-data`; re-enable - **Left to do**: restore test on the complete `restic-data`; re-enable
the interim ks2 push until FTTH; then the ks2 decommission gates the interim ks2 push until FTTH; then the ks2 decommission gates
([../ks2/plan.md](../ks2/plan.md)). The instance leg (§6) is ([../ks2/plan.md](../ks2/plan.md)). The instance leg (§6) is
shelved, not pending. shelved, not pending.
- **plakar retirement, one exception**: keep plakar installed and - **Old tool**: see the legacy note above; the `integration-incus`
keep the `plakar-data` kloset until repo retires from production (optionally still published to
[#2338](https://github.com/PlakarKorp/plakar/issues/2338) concludes PlakarKorp/hub as a community contribution).
— the issue offers debug runs against that repository. Delete the
`plakar-incus` bucket immediately; `plakar-data` a few weeks later.
The `integration-incus` repo retires from production; optionally
still publish to PlakarKorp/hub as a community contribution.
- **Interim ks2 push** (until the nuc leg seeds after FTTH, expected - **Interim ks2 push** (until the nuc leg seeds after FTTH, expected
before end of September): re-enable the replica push so instances before end of September): re-enable the replica push so instances
keep an off-site copy meanwhile: keep an off-site copy meanwhile: