diff --git a/backup-strategy.md b/backup-strategy.md index 8c64474..94d3720 100644 --- a/backup-strategy.md +++ b/backup-strategy.md @@ -4,25 +4,29 @@ Audience: anyone with root on ks4 who needs to know **what is protected, where, by which tool, and how to check it**. Details live in the linked docs; this page is the map. -## Two tools, no more +## Two tools, three legs -| tool | job | why this one | +| leg | mechanism | protects | |---|---|---| -| **`incus copy --refresh`** | replicate whole instances (ready-to-start copies) | ZFS-incremental, native to the platform, restores are `incus start` | -| **restic** | off-site backups to S3 (files, database dumps, instance trees) | dedup + encryption, local metadata cache → a night costs only the churn ([restic-backup.md](ks4/restic-backup.md)) | +| **local replication** | `incus copy --refresh` → pool `backup` on ks4's second disk (sdb) | instances, against losing the `data` pool | +| **remote replication** | nuc pulls the same replicas over WireGuard → pool `ks4backup` | instances, against losing ks4 or the site | +| **remote backup** | **restic** → S3 bucket `restic-data`: database dumps + selected filesystem trees | the data itself, versioned and encrypted, independent of every disk above | -plakar was evaluated first and replaced (its nightly cost scaled with -tree size — [PlakarKorp/plakar#2338](https://github.com/PlakarKorp/plakar/issues/2338)); -the old backup server **ks2 is being retired** (decommission by -Sep 30, 2026 — [ks2/plan.md](ks2/plan.md)): it still holds an ageing -copy of the instances until the nuc leg below takes over. +Only two tools are involved: `incus copy` (ZFS-incremental, native to +the platform — a restore is `incus start`) and restic (dedup + +encryption, local metadata cache, so a night costs only the churn — +[ks4/restic-backup.md](ks4/restic-backup.md)). -**Status 2026-08-28**: the local replica leg and the S3 data leg are -live. The **nuc pull leg is waiting for FTTH** (expected before end of -September); until it lands, the ks2 push is re-enabled as an interim -off-site copy of the instances. Backing up instance *images* to S3 was -designed and shelved — S3 holds the data, the databases and the incus -configuration, which is what a rebuild needs. +The old backup server **ks2 is being retired** (decommission by +Sep 30, 2026 — [ks2/plan.md](ks2/plan.md)); until FTTH enables the nuc +leg it still receives an interim replica push. + +**Status 2026-08-28**: local replication and the S3 backup leg are +live. The **nuc pull leg waits for FTTH** (expected before end of +September); until then the ks2 push stands in. Backing up whole +instance *images* to S3 was considered and left out — S3 holds the +data, the databases and the incus configuration, which is what a +rebuild needs. ## The map @@ -120,3 +124,18 @@ New instances are picked up automatically by the replica and instance legs (all instances, opt-out only). Databases are auto-discovered. Only **data trees** need one line in [`restic-paths`](https://git.lutran.fr/julien/scripts/src/branch/main/restic-paths) — see [new-container.md](new-container.md). + +## History (reference only) + +The road to the setup above, kept for the measurements rather than for +operations: an rsync-based leg to ks2 +([ks2/plan.md](ks2/plan.md)) and a first S3 implementation with +**plakar** ([ks4/plakar-s3-data.md](ks4/plakar-s3-data.md), plus a +self-written incus connector, +[ks4/plakar-incus-integration.md](ks4/plakar-incus-integration.md)). +plakar was dropped because its nightly cost scaled with the size of +the tree rather than with the churn +([PlakarKorp/plakar#2338](https://github.com/PlakarKorp/plakar/issues/2338)). +It is still installed on ks4 and its `plakar-data` bucket still exists +— kept only so the upstream issue can be reproduced; nothing schedules +it any more. diff --git a/ks2/decommission.md b/ks2/decommission.md index 5a27ac9..d2018f7 100644 --- a/ks2/decommission.md +++ b/ks2/decommission.md @@ -55,8 +55,8 @@ renewal date. - [ ] Retire `scripts/incus-backup.sh` + `incus-backup.db` in the repo (superseded by [`restic-backup.sh`](https://git.lutran.fr/julien/scripts/src/branch/main/restic-backup.sh)) — keep for reference, note the retirement in their headers -- [ ] `login` container cleanup: the native postgres still holds the - pre-migration `outline` DB — `dropdb outline`, remove the - package (noted in - [plakar-s3-data.md](../ks4/plakar-s3-data.md) §3) +- [ ] `login` container cleanup: its **native** postgres still holds + the pre-migration `outline` database (live Outline uses the + dockerised `outline-postgres-1`) — `dropdb outline` and remove + the native postgres package - [ ] Mark the ks2 line in `plan.md` done, date it diff --git a/ks2/plan.md b/ks2/plan.md index 9e5d27a..60bcd7d 100644 --- a/ks2/plan.md +++ b/ks2/plan.md @@ -61,9 +61,9 @@ when nuc takes over. 2. ~~[local-backup-cron.md](../ks4/local-backup-cron.md)~~ — **done 2026-08-22**: leg 1 cron re-enabled, ks2 rsync kept as stopgap; `livetrail` verified present in the `backup` project -3. ~~plakar S3 leg~~ → **replaced by restic** - ([restic-backup.md](../ks4/restic-backup.md), live 2026-08-28; - plakar's own doc kept as history) +3. ~~first S3 leg~~ → **restic** is the S3 tool + ([restic-backup.md](../ks4/restic-backup.md), live 2026-08-28; the + predecessor's doc is kept as reference) 4. ~~instance leg to S3~~ — **shelved 2026-08-28**: instances are protected by replication (sdb + nuc/ks2), their data and configs by `restic-data` diff --git a/ks4/install.md b/ks4/install.md index b225ec5..10072ad 100644 --- a/ks4/install.md +++ b/ks4/install.md @@ -164,9 +164,10 @@ non-mirrored `data` pool: into the `restic-data` bucket ([restic-backup.md](restic-backup.md)). - **Sun 14:00 `restic-maintenance.sh`** → prune + integrity checks. -Historical (retired 2026-08): the `ks2` push and `incus-backup.sh` -rsync leg, and the plakar data leg -([plakar-s3-data.md](plakar-s3-data.md)). +A `ks2` interim replica push (`incus-copy.sh -d ks2 -m push`, 02:00) +runs until the nuc pull leg is seeded. Retired 2026-08: the +`incus-backup.sh` rsync leg and the first S3 implementation +([plakar-s3-data.md](plakar-s3-data.md) — reference only). - **01:00 — `incus-copy.sh` → `ks2`**: differential `incus copy … \ --refresh --refresh-exclude-older --mode push` of every *running* diff --git a/ks4/local-backup-cron.md b/ks4/local-backup-cron.md index e7fabe0..8a24018 100644 --- a/ks4/local-backup-cron.md +++ b/ks4/local-backup-cron.md @@ -15,9 +15,9 @@ backup of any kind was running). Overview of all legs: push to ks2; see [incus-copy.md](incus-copy.md)). An S3 leg for the instance filesystems was designed and shelved ([restic-backup.md](restic-backup.md) §6). -- The rsync stopgap to ks2 (`incus-backup.sh`, 04:00) and the plakar - data leg (04:30) are both **retired**; S3 backups are restic's job - at 05:00. +- The rsync stopgap to ks2 (`incus-backup.sh`, 04:00) and the first + S3 implementation (04:30) are both **retired**; S3 backups are + restic's job at 05:00 ([restic-backup.md](restic-backup.md)). ## Cleanup done the same day diff --git a/ks4/restic-backup.md b/ks4/restic-backup.md index 6afb049..188b5db 100644 --- a/ks4/restic-backup.md +++ b/ks4/restic-backup.md @@ -1,25 +1,18 @@ # restic backups on ks4 — the S3 leg -Status: **live since 2026-08-28** — restic replaced plakar as the S3 -backup tool (plakar crons off, its kloset kept for -[#2338](https://github.com/PlakarKorp/plakar/issues/2338); see §8). -Architecture: **two tools** — `incus copy` for replication (local sdb, -nuc after FTTH), **restic** for everything S3. Data leg (`restic-data`) -runs nightly at 05:00. The instance leg (`restic-incus`, §6) is -**shelved** (2026-08-28): instances are covered by replication, so -duplicating their filesystems into S3 buys little. The script and the -(empty) repo stay in place should that change. +Status: **live since 2026-08-28.** This is the S3 backup leg of the +[strategy](../backup-strategy.md): database dumps + selected +filesystem trees into the `restic-data` bucket, nightly at 05:00. +Instances themselves are handled by replication (sdb, nuc after FTTH), +not by this leg — an instance-filesystem leg (`restic-incus`, §6) was +written and **shelved**; its script and empty repo stay in place +should that change. -Why: plakar's incremental cost scales with tree size, not churn -(~800 MiB parent metadata re-read from S3 + hours of CPU per night on -a 500k-file tree — measured, reported as -[PlakarKorp/plakar#2338](https://github.com/PlakarKorp/plakar/issues/2338)). -restic keeps its comparison state in a local cache: a night costs a -local stat-walk plus the churn — the rsync cost model. Neither restic -nor rustic has plakar's source-plugin architecture, but none is -needed: instances are reached via `incus file mount` (FUSE over the -same per-instance sftp API our plakar integration used) — zero -patches to maintain (see §6). +Why restic: its comparison state lives in a **local cache**, so a +night costs a local stat-walk plus the churn — the rsync cost model, +with dedup, compression and encryption on top. (The predecessor and +the measurements that motivated the switch are in +[plakar-s3-data.md](plakar-s3-data.md) — reference only.) ## 1. Repositories — two buckets, two drivers (decided 2026-08-25) @@ -28,7 +21,7 @@ patches to maintain (see §6). | `restic-data` | SQL dumps + **all** fs sources (8 small + nextcloud + seafile) | `restic-backup.sh` | | `restic-incus` | *(shelved — empty repo, see §6)* the `backup`-project replicas, per-file via mount | `restic-incus-backup.sh` | -Mirrors the plakar-era split (data leg / incus leg). Accepted +Two repos so each can be pruned and checked independently. Accepted trade-off: `restic-data` carries the giants' blobs (~3–5 M once seafile is in), so **every operation on it loads a ~1.5–2 GiB index** — mitigated by running the whole nightly as a *single* @@ -168,9 +161,9 @@ done ## 5. Nightly driver — `scripts/restic-backup.sh` -Same shape as the plakar drivers (flock, env, loud logging). Phases: +Shape: flock, env file, loud logging. Phases: -1. **dumps** — lifted verbatim from `plakar-backup.sh`: incus DB +1. **dumps** — incus DB dumps + auto-discovered MariaDB/PostgreSQL (native and docker) dumps into the dump dir (during the parallel week restic dumps into its own `/backup/dumps`; carry the archived `geo.sql` over @@ -227,7 +220,7 @@ byte once a year, ~25 GiB read per week. Neither restic nor rustic accepts source plugins, and forking the backup tool is the wrong place to carry a patch. Instead, incus exposes any instance (including **stopped replicas**) as a FUSE mount -over the same per-instance sftp API the plakar integration used: +over the per-instance sftp API: ```sh for inst in $(incus list --project backup -c n -f csv); do @@ -252,8 +245,7 @@ opt-out via a variable — the no-manifest-drift rule survives the tool change. Chained after the 01:00 incus-copy once validated, same freshness reasoning as before. -Bonus over plakar: one shared repo dedups the Ubuntu base across all -16 replicas. Restore granularity: per-file; instance definitions ride +One shared repo would dedup the Ubuntu base across all replicas. Restore granularity: per-file; instance definitions ride the dump tree (`incus/.yaml` + `incus-global-db.sql`). ## 7. Restore test (gate) @@ -267,24 +259,26 @@ diff -r … && rm -rf /backup/restore-test ## 8. Cutover (done) + what is left -- **Cutover 2026-08-28**: plakar's 04:30 and Sunday crons are off; - restic's 05:00 data leg is the S3 backup. The formal parallel-week +Legacy on the box: plakar is still installed and its `plakar-data` +bucket still exists, kept only to reproduce +[#2338](https://github.com/PlakarKorp/plakar/issues/2338) if upstream +asks; no cron references it. The empty `plakar-incus` bucket can be +deleted. + +- **Cutover 2026-08-28**: the previous S3 leg's crons are off and + restic's 05:00 run is the S3 backup. The formal parallel-week comparison was cut short — the evidence had already settled it - (plakar: ~800 MiB metadata re-read and 3–4 h per night on the + (predecessor: ~800 MiB metadata re-read and 3–4 h per night on the nextcloud tree, churn-independent; restic: seed at 60–115 MiB/s, - local-cache incrementals). plakar's measurements stay in + local-cache incrementals). Details in [plakar-s3-data.md](plakar-s3-data.md). - **Left to do**: restore test on the complete `restic-data`; re-enable the interim ks2 push until FTTH; then the ks2 decommission gates ([../ks2/plan.md](../ks2/plan.md)). The instance leg (§6) is shelved, not pending. -- **plakar retirement, one exception**: keep plakar installed and - keep the `plakar-data` kloset until - [#2338](https://github.com/PlakarKorp/plakar/issues/2338) concludes - — the issue offers debug runs against that repository. Delete the - `plakar-incus` bucket immediately; `plakar-data` a few weeks later. - The `integration-incus` repo retires from production; optionally - still publish to PlakarKorp/hub as a community contribution. +- **Old tool**: see the legacy note above; the `integration-incus` + repo retires from production (optionally still published to + PlakarKorp/hub as a community contribution). - **Interim ks2 push** (until the nuc leg seeds after FTTH, expected before end of September): re-enable the replica push so instances keep an off-site copy meanwhile: