From b5aef914f57a10d738fcc6a2ae4d7f49d3146470 Mon Sep 17 00:00:00 2001 From: Julien Lutran Date: Fri, 28 Aug 2026 16:17:11 +0200 Subject: [PATCH] doc: instance leg to S3 shelved; ks2 push is the interim off-site leg MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Decided 2026-08-28: instances are protected by replication (sdb now, nuc after FTTH, ks2 push in between), their data/DBs/configs by restic-data — a third copy of instance filesystems in S3 was not worth the moving parts. backup-strategy.md now explains the replication-vs-backup split and the rebuild path when no replica survives; restic-backup.md §6 is marked SHELVED (script and empty repo kept). Co-Authored-By: Claude Fable 5 --- backup-strategy.md | 80 ++++++++++++++++++++++++---------------- ks2/plan.md | 17 +++++---- ks4/local-backup-cron.md | 6 +-- ks4/restic-backup.md | 29 ++++++++++----- new-container.md | 3 +- 5 files changed, 81 insertions(+), 54 deletions(-) diff --git a/backup-strategy.md b/backup-strategy.md index 802d465..8c64474 100644 --- a/backup-strategy.md +++ b/backup-strategy.md @@ -19,45 +19,58 @@ copy of the instances until the nuc leg below takes over. **Status 2026-08-28**: the local replica leg and the S3 data leg are live. The **nuc pull leg is waiting for FTTH** (expected before end of -September) and the **S3 instance leg is written but not yet seeded** — -until both land, instances have only the ks4-local replica off the -live pool. +September); until it lands, the ks2 push is re-enabled as an interim +off-site copy of the instances. Backing up instance *images* to S3 was +designed and shelved — S3 holds the data, the databases and the incus +configuration, which is what a rebuild needs. ## The map ``` - ks4 (OVH) off-site - ┌─────────────────────────────────────────────┐ - │ live instances pool `data` (sda) │ - │ nextcloud, seafile, mail, git, ... │ - │ │ │ - │ │ 01:00 incus copy --refresh │ nuc (home LAN, via WireGuard) - │ ▼ │ ┌──────────────────────────┐ - │ replicas (stopped) pool `backup` (sdb) │─────▶│ 05:00 incus copy (pull) │ - │ project `backup` │ │ pool `ks4backup` (USB) │ - │ │ │ └──────────────────────────┘ - │ │ 01:00+ restic (via incus file mount) - │ ▼ │ OVH Object Storage (S3, sbg) - │ ───────────────────────────────────────── │ ┌──────────────────────────┐ - │ 05:00 restic-backup.sh │─────▶│ bucket restic-data │ - │ • DB dumps (MariaDB/PostgreSQL, auto) │ │ dumps + all data trees │ - │ • data trees from /root/scripts/restic-paths │ │ - │ 01:00+ restic-incus-backup.sh │─────▶│ bucket restic-incus │ - │ • every replica, per file + config yaml │ │ instance filesystems │ - │ Sun 14:00 restic-maintenance.sh │ └──────────────────────────┘ - │ • prune + check (rotating full verify) │ - └─────────────────────────────────────────────┘ + ks4 (OVH) off-site + ┌────────────────────────────────────────────┐ + │ live instances pool `data` (sda) │ + │ nextcloud, seafile, mail, git, ... │ + │ │ │ + │ │ 01:00 incus copy --refresh │ nuc (home LAN, via WireGuard) + │ ▼ │ ┌───────────────────────────┐ + │ replicas (stopped) pool `backup` (sdb) │────▶│ 05:00 incus copy (pull) │ + │ project `backup` ready to start │ │ pool `ks4backup` (USB) │ + │ │ └───────────────────────────┘ + │ ──────────────────────────────────────── │ + │ │ OVH Object Storage (S3, sbg) + │ 05:00 restic-backup.sh │ ┌───────────────────────────┐ + │ • DB dumps (MariaDB/PostgreSQL, auto) │────▶│ bucket restic-data │ + │ • incus config + global DB dump │ │ everything needed to │ + │ • data trees (restic-paths) │ │ rebuild: dumps + data │ + │ │ └───────────────────────────┘ + │ Sun 14:00 restic-maintenance.sh │ + │ • prune + check (rotating full verify) │ + └────────────────────────────────────────────┘ ``` -Copies of any given byte: live → sdb replica (same box, other disk) -→ nuc replica (other site) → S3 (other site, other technology). -Databases additionally get application-consistent dumps nightly. +Two different kinds of protection, on purpose: + +- **instances** (the running systems) are protected by *replication* — + a ready-to-start copy on ks4's second disk and, after FTTH, on nuc. + Restoring one is `incus copy` + `incus start`. +- **the data inside them** (files, databases, incus configuration) is + protected by *backup* — encrypted, deduplicated, versioned on S3, + independent of ks4 and of the disks. Restoring means recreating the + container (its config is in the S3 dumps, its install steps are in + this repo) and pouring the data back. + +Backing up whole instance images to S3 as well was designed +([ks4/restic-backup.md](ks4/restic-backup.md) §6) and shelved: it +duplicates data already covered, and the replication legs already give +instances two homes. ## Schedule (root crontab on ks4) | when | what | log | |---|---|---| -| 01:00 daily | `incus-copy.sh -p backup -s backup` — refresh all replicas onto sdb, then `restic-incus-backup.sh` → S3 | `/var/log/incus-copy.log`, `/var/log/restic-incus.log` | +| 01:00 daily | `incus-copy.sh -p backup -s backup` — refresh all replicas onto sdb | `/var/log/incus-copy.log` | +| 02:00 daily | `incus-copy.sh -d ks2 -m push` — interim off-site replicas, until the nuc leg replaces it | `/var/log/incus-copy.log` | | 03:00 daily | instance snapshots (incus profile, 7-day expiry) — what keeps the refreshes incremental | `incus info ` | | 05:00 daily | `restic-backup.sh` — dumps + data trees → S3 | `/var/log/restic-backup.log` | | 05:00 daily (nuc) | nuc pulls all replicas over WireGuard | nuc: `/var/log/incus-copy-ks4.log` | @@ -71,7 +84,6 @@ Retention on S3: 14 daily, 8 weekly, 6 monthly snapshots. ```sh grep -aE "done \(rc=|FAILED" /var/log/incus-copy.log | tail -2 grep -aE "snapshot .* saved|done \(rc=|failed" /var/log/restic-backup.log | tail -3 -grep -aE "done \(rc=|failed" /var/log/restic-incus.log | tail -2 ``` `rc=0` everywhere = fine. Any `failed`/`FAILED` line names the culprit @@ -90,9 +102,13 @@ never silently. - **A database**: restore the `.sql` dump from `restic-data` (`/backup/dumps/mariadb//.sql`, `grants.sql` for users), load it with `incus exec -- mariadb < dump.sql`. -- **An instance from S3 only** (ks4 and both replica sites gone): - restore its tree + `.yaml` from `restic-incus`, recreate the - instance from the yaml/profile, push the tree back. +- **An instance when no replica survives** (worst case: ks4 and the + replica sites are gone): recreate the container from its page in + [ks4/](ks4/) (the doc *is* the install script), restore its data + trees and database dumps from `restic-data`, and use the incus + configuration captured nightly in the dump tree + (`/backup/dumps/incus/incus-global-db.sql`) to check devices, + profiles and addresses. Secrets you need for any of this: `/root/.restic-passphrase` and the S3 keys in `/root/.restic-env` — **both are in the password manager**; diff --git a/ks2/plan.md b/ks2/plan.md index c7657a7..9e5d27a 100644 --- a/ks2/plan.md +++ b/ks2/plan.md @@ -17,10 +17,10 @@ What remains on the box is **cold history**: instance replicas on pool `/backup/ns3061243` on pool `backup` (last refreshed 2026-08-28), snapshotted daily there (`zfs-auto-snapshot.sh`, 2-month expiry). -⚠️ Consequence while the nuc leg waits for FTTH: **instances have no -fresh off-site copy** — only the ks4-local sdb replicas plus ks2's -ageing ones. Either re-enable the interim push (below) or accept the -gap knowingly until FTTH. +⚠️ While the nuc leg waits for FTTH, instances have no *fresh* +off-site copy — the ks2 push is to be re-enabled as soon as the +initial restic sync finishes (decided 2026-08-28), and retired again +when nuc takes over. ## Inventory findings (2026-08-22) @@ -50,9 +50,9 @@ gap knowingly until FTTH. |---|---|---| | local, 2nd disk | `incus-copy.sh -p backup -s backup` → sdb5 pool | **live** (01:00) | | off-site, S3 (data+DB) | **restic** → bucket `restic-data` ([restic-backup.md](../ks4/restic-backup.md)) | **live** (05:00) | -| off-site, S3 (instances) | restic over `incus file mount` of the `backup`-project replicas | written, **not seeded** | +| off-site, S3 (instances) | restic over `incus file mount` of the `backup`-project replicas | **shelved** 2026-08-28 (replication covers instances) | | off-site, nuc | nuc pulls `ks4:*` → pool `ks4backup` over WG ([nuc-seed.md](nuc-seed.md)) | waiting FTTH (< Sep 30) | -| off-site, ks2 (optional interim) | `incus-copy.sh -d ks2 -m push` at 02:00 until the nuc leg seeds | **not enabled** — decide | +| off-site, ks2 (interim) | `incus-copy.sh -d ks2 -m push` at 02:00 until the nuc leg seeds | to re-enable once the restic seed finishes | ## Actions (backup work documented in [`../ks4/`](../ks4/), ks2-only tasks here) @@ -64,8 +64,9 @@ gap knowingly until FTTH. 3. ~~plakar S3 leg~~ → **replaced by restic** ([restic-backup.md](../ks4/restic-backup.md), live 2026-08-28; plakar's own doc kept as history) -4. instance leg — restic over `incus file mount` - ([restic-backup.md](../ks4/restic-backup.md) §6): **seed pending** +4. ~~instance leg to S3~~ — **shelved 2026-08-28**: instances are + protected by replication (sdb + nuc/ks2), their data and configs by + `restic-data` 5. [nuc-seed.md](nuc-seed.md) — **prepared**; after FTTH: seed nuc pull leg, verify all instances, test-restore one 6. [decommission.md](decommission.md) — **prepared**; cut flows, diff --git a/ks4/local-backup-cron.md b/ks4/local-backup-cron.md index b730ca7..e7fabe0 100644 --- a/ks4/local-backup-cron.md +++ b/ks4/local-backup-cron.md @@ -12,9 +12,9 @@ backup of any kind was running). Overview of all legs: - `01:00` — **local leg**: `incus copy --refresh` of all instances into project `backup` on the sdb5 `backup` zpool (replaces the old - push to ks2; see [incus-copy.md](incus-copy.md)). The S3 instance - leg ([restic-backup.md](restic-backup.md) §6) chains onto this same - line once seeded — it reads the replicas this job just refreshed. + push to ks2; see [incus-copy.md](incus-copy.md)). An S3 leg for the + instance filesystems was designed and shelved + ([restic-backup.md](restic-backup.md) §6). - The rsync stopgap to ks2 (`incus-backup.sh`, 04:00) and the plakar data leg (04:30) are both **retired**; S3 backups are restic's job at 05:00. diff --git a/ks4/restic-backup.md b/ks4/restic-backup.md index 384a96c..6afb049 100644 --- a/ks4/restic-backup.md +++ b/ks4/restic-backup.md @@ -5,8 +5,10 @@ backup tool (plakar crons off, its kloset kept for [#2338](https://github.com/PlakarKorp/plakar/issues/2338); see §8). Architecture: **two tools** — `incus copy` for replication (local sdb, nuc after FTTH), **restic** for everything S3. Data leg (`restic-data`) -runs nightly at 05:00; the instance leg (`restic-incus`, §6) is written -and awaits its first seed. +runs nightly at 05:00. The instance leg (`restic-incus`, §6) is +**shelved** (2026-08-28): instances are covered by replication, so +duplicating their filesystems into S3 buys little. The script and the +(empty) repo stay in place should that change. Why: plakar's incremental cost scales with tree size, not churn (~800 MiB parent metadata re-read from S3 + hours of CPU per night on @@ -24,7 +26,7 @@ patches to maintain (see §6). | bucket = repo | contents | driver script | |---|---|---| | `restic-data` | SQL dumps + **all** fs sources (8 small + nextcloud + seafile) | `restic-backup.sh` | -| `restic-incus` | the `backup`-project replicas, per-file via mount | `restic-incus-backup.sh` | +| `restic-incus` | *(shelved — empty repo, see §6)* the `backup`-project replicas, per-file via mount | `restic-incus-backup.sh` | Mirrors the plakar-era split (data leg / incus leg). Accepted trade-off: `restic-data` carries the giants' blobs (~3–5 M once @@ -183,8 +185,8 @@ Same shape as the plakar drivers (flock, env, loud logging). Phases: `--group-by host` because seed-era snapshots have different path sets than nightly ones and must age in one group). -The instances leg is `restic-incus-backup.sh` (§6), chained after the -01:00 incus-copy once validated. +The instances leg (`restic-incus-backup.sh`, §6) is shelved — nothing +chains onto the 01:00 line today. ```cron 0 5 * * * /root/scripts/restic-backup.sh >> /var/log/restic-backup.log 2>&1 @@ -212,7 +214,15 @@ Then `restic check` (structure, cheap) + `check --read-data-subset=1/52` rotating — full data verification of every byte once a year, ~25 GiB read per week. -## 6. Instances without a plugin — `incus file mount` +## 6. Instances without a plugin — `incus file mount` (SHELVED) + +> Decided 2026-08-28: **not deployed.** Instances are replicated (sdb, +> nuc after FTTH, ks2 in the meantime) and everything needed to +> rebuild one is in `restic-data`, so a third copy of their +> filesystems was judged not worth the nightly moving parts. The +> design and the script remain, ready if that changes. +> If the `restic-incus` bucket is ever deleted, drop it from `REPOS` +> in `restic-maintenance.sh` too. Neither restic nor rustic accepts source plugins, and forking the backup tool is the wrong place to carry a patch. Instead, incus @@ -264,9 +274,10 @@ diff -r … && rm -rf /backup/restore-test nextcloud tree, churn-independent; restic: seed at 60–115 MiB/s, local-cache incrementals). plakar's measurements stay in [plakar-s3-data.md](plakar-s3-data.md). -- **Left to do**: seed `restic-incus` (§6) and chain it onto the 01:00 - line; restore test on the complete `restic-data`; then the ks2 - decommission gates ([../ks2/plan.md](../ks2/plan.md)). +- **Left to do**: restore test on the complete `restic-data`; re-enable + the interim ks2 push until FTTH; then the ks2 decommission gates + ([../ks2/plan.md](../ks2/plan.md)). The instance leg (§6) is + shelved, not pending. - **plakar retirement, one exception**: keep plakar installed and keep the `plakar-data` kloset until [#2338](https://github.com/PlakarKorp/plakar/issues/2338) concludes diff --git a/new-container.md b/new-container.md index 747a7ca..9f649ad 100644 --- a/new-container.md +++ b/new-container.md @@ -67,8 +67,7 @@ them forces a full re-send of its whole disk every night. **Automatic, nothing to do:** - the 01:00 replica leg copies *every* instance to the `backup` pool - and to nuc — new instances included; -- the S3 instance leg backs up every replica the same way; + and off-site — new instances included; - **databases are auto-discovered**: any running container with MariaDB/MySQL or PostgreSQL gets every non-system database dumped nightly, plus users/grants.