doc: instance leg to S3 shelved; ks2 push is the interim off-site leg

Decided 2026-08-28: instances are protected by replication (sdb now,
nuc after FTTH, ks2 push in between), their data/DBs/configs by
restic-data — a third copy of instance filesystems in S3 was not
worth the moving parts. backup-strategy.md now explains the
replication-vs-backup split and the rebuild path when no replica
survives; restic-backup.md §6 is marked SHELVED (script and empty
repo kept).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

backup-strategy.md

@@ -19,45 +19,58 @@ copy of the instances until the nuc leg below takes over.
 
 **Status 2026-08-28**: the local replica leg and the S3 data leg are
 live. The **nuc pull leg is waiting for FTTH** (expected before end of
-September) and the **S3 instance leg is written but not yet seeded** —
+September); until it lands, the ks2 push is re-enabled as an interim
-until both land, instances have only the ks4-local replica off the
+off-site copy of the instances. Backing up instance *images* to S3 was
-live pool.
+designed and shelved — S3 holds the data, the databases and the incus
+configuration, which is what a rebuild needs.
 
 ## The map
 
 ```
  ks4 (OVH)                                           off-site
- ┌─────────────────────────────────────────────┐
+ ┌────────────────────────────────────────────┐
  │  live instances       pool `data` (sda)    │
  │  nextcloud, seafile, mail, git, ...        │
  │        │                                   │
  │        │ 01:00  incus copy --refresh       │     nuc (home LAN, via WireGuard)
- │        ▼                                    │      ┌──────────────────────────┐
+ │        ▼                                   │     ┌───────────────────────────┐
- │  replicas (stopped)    pool `backup` (sdb)  │─────▶│ 05:00 incus copy (pull)  │
+ │  replicas (stopped)   pool `backup` (sdb)  │────▶│ 05:00 incus copy (pull)   │
- │  project `backup`                           │      │ pool `ks4backup` (USB)   │
+ │  project `backup`     ready to start       │     │ pool `ks4backup` (USB)    │
- │        │                                    │      └──────────────────────────┘
+ │                                            │     └───────────────────────────┘
- │        │ 01:00+ restic (via incus file mount)
+ │  ────────────────────────────────────────  │
- │        ▼                                    │      OVH Object Storage (S3, sbg)
+ │                                            │     OVH Object Storage (S3, sbg)
- │  ─────────────────────────────────────────  │      ┌──────────────────────────┐
+ │  05:00 restic-backup.sh                    │     ┌───────────────────────────┐
- │  05:00 restic-backup.sh                     │─────▶│ bucket restic-data       │
+ │    • DB dumps (MariaDB/PostgreSQL, auto)   │────▶│ bucket restic-data        │
- │    • DB dumps (MariaDB/PostgreSQL, auto)    │      │  dumps + all data trees  │
+ │    • incus config + global DB dump         │     │  everything needed to     │
- │    • data trees from /root/scripts/restic-paths     │                          │
+ │    • data trees (restic-paths)             │     │  rebuild: dumps + data    │
- │  01:00+ restic-incus-backup.sh              │─────▶│ bucket restic-incus      │
+ │                                            │     └───────────────────────────┘
- │    • every replica, per file + config yaml  │      │  instance filesystems    │
+ │  Sun 14:00 restic-maintenance.sh           │
- │  Sun 14:00 restic-maintenance.sh            │      └──────────────────────────┘
  │    • prune + check (rotating full verify)  │
- └─────────────────────────────────────────────┘
+ └────────────────────────────────────────────┘
 ```
 
-Copies of any given byte: live → sdb replica (same box, other disk)
+Two different kinds of protection, on purpose:
-→ nuc replica (other site) → S3 (other site, other technology).
+
-Databases additionally get application-consistent dumps nightly.
+- **instances** (the running systems) are protected by *replication* —
+  a ready-to-start copy on ks4's second disk and, after FTTH, on nuc.
+  Restoring one is `incus copy` + `incus start`.
+- **the data inside them** (files, databases, incus configuration) is
+  protected by *backup* — encrypted, deduplicated, versioned on S3,
+  independent of ks4 and of the disks. Restoring means recreating the
+  container (its config is in the S3 dumps, its install steps are in
+  this repo) and pouring the data back.
+
+Backing up whole instance images to S3 as well was designed
+([ks4/restic-backup.md](ks4/restic-backup.md) §6) and shelved: it
+duplicates data already covered, and the replication legs already give
+instances two homes.
 
 ## Schedule (root crontab on ks4)
 
 | when | what | log |
 |---|---|---|
-| 01:00 daily | `incus-copy.sh -p backup -s backup` — refresh all replicas onto sdb, then `restic-incus-backup.sh` → S3 | `/var/log/incus-copy.log`, `/var/log/restic-incus.log` |
+| 01:00 daily | `incus-copy.sh -p backup -s backup` — refresh all replicas onto sdb | `/var/log/incus-copy.log` |
+| 02:00 daily | `incus-copy.sh -d ks2 -m push` — interim off-site replicas, until the nuc leg replaces it | `/var/log/incus-copy.log` |
 | 03:00 daily | instance snapshots (incus profile, 7-day expiry) — what keeps the refreshes incremental | `incus info <inst>` |
 | 05:00 daily | `restic-backup.sh` — dumps + data trees → S3 | `/var/log/restic-backup.log` |
 | 05:00 daily (nuc) | nuc pulls all replicas over WireGuard | nuc: `/var/log/incus-copy-ks4.log` |
@@ -71,7 +84,6 @@ Retention on S3: 14 daily, 8 weekly, 6 monthly snapshots.
 ```sh
 grep -aE "done \(rc=|FAILED" /var/log/incus-copy.log | tail -2
 grep -aE "snapshot .* saved|done \(rc=|failed" /var/log/restic-backup.log | tail -3
-grep -aE "done \(rc=|failed" /var/log/restic-incus.log | tail -2
 ```
 
 `rc=0` everywhere = fine. Any `failed`/`FAILED` line names the culprit
@@ -90,9 +102,13 @@ never silently.
 - **A database**: restore the `.sql` dump from `restic-data`
   (`/backup/dumps/mariadb/<inst>/<db>.sql`, `grants.sql` for users), load
   it with `incus exec <inst> -- mariadb < dump.sql`.
-- **An instance from S3 only** (ks4 and both replica sites gone):
+- **An instance when no replica survives** (worst case: ks4 and the
-  restore its tree + `<inst>.yaml` from `restic-incus`, recreate the
+  replica sites are gone): recreate the container from its page in
-  instance from the yaml/profile, push the tree back.
+  [ks4/](ks4/) (the doc *is* the install script), restore its data
+  trees and database dumps from `restic-data`, and use the incus
+  configuration captured nightly in the dump tree
+  (`/backup/dumps/incus/incus-global-db.sql`) to check devices,
+  profiles and addresses.
 
 Secrets you need for any of this: `/root/.restic-passphrase` and the S3
 keys in `/root/.restic-env` — **both are in the password manager**;

ks2/plan.md

@@ -17,10 +17,10 @@ What remains on the box is **cold history**: instance replicas on pool
 `/backup/ns3061243` on pool `backup` (last refreshed 2026-08-28),
 snapshotted daily there (`zfs-auto-snapshot.sh`, 2-month expiry).
 
-⚠️ Consequence while the nuc leg waits for FTTH: **instances have no
+⚠️ While the nuc leg waits for FTTH, instances have no *fresh*
-fresh off-site copy** — only the ks4-local sdb replicas plus ks2's
+off-site copy — the ks2 push is to be re-enabled as soon as the
-ageing ones. Either re-enable the interim push (below) or accept the
+initial restic sync finishes (decided 2026-08-28), and retired again
-gap knowingly until FTTH.
+when nuc takes over.
 
 ## Inventory findings (2026-08-22)
 
@@ -50,9 +50,9 @@ gap knowingly until FTTH.
 |---|---|---|
 | local, 2nd disk | `incus-copy.sh -p backup -s backup` → sdb5 pool | **live** (01:00) |
 | off-site, S3 (data+DB) | **restic** → bucket `restic-data` ([restic-backup.md](../ks4/restic-backup.md)) | **live** (05:00) |
-| off-site, S3 (instances) | restic over `incus file mount` of the `backup`-project replicas | written, **not seeded** |
+| off-site, S3 (instances) | restic over `incus file mount` of the `backup`-project replicas | **shelved** 2026-08-28 (replication covers instances) |
 | off-site, nuc | nuc pulls `ks4:*` → pool `ks4backup` over WG ([nuc-seed.md](nuc-seed.md)) | waiting FTTH (< Sep 30) |
-| off-site, ks2 (optional interim) | `incus-copy.sh -d ks2 -m push` at 02:00 until the nuc leg seeds | **not enabled** — decide |
+| off-site, ks2 (interim) | `incus-copy.sh -d ks2 -m push` at 02:00 until the nuc leg seeds | to re-enable once the restic seed finishes |
 
 ## Actions (backup work documented in [`../ks4/`](../ks4/), ks2-only tasks here)
 
@@ -64,8 +64,9 @@ gap knowingly until FTTH.
 3. ~~plakar S3 leg~~ → **replaced by restic**
    ([restic-backup.md](../ks4/restic-backup.md), live 2026-08-28;
    plakar's own doc kept as history)
-4. instance leg — restic over `incus file mount`
+4. ~~instance leg to S3~~ — **shelved 2026-08-28**: instances are
-   ([restic-backup.md](../ks4/restic-backup.md) §6): **seed pending**
+   protected by replication (sdb + nuc/ks2), their data and configs by
+   `restic-data`
 5. [nuc-seed.md](nuc-seed.md) — **prepared**; after FTTH: seed nuc
    pull leg, verify all instances, test-restore one
 6. [decommission.md](decommission.md) — **prepared**; cut flows,

ks4/local-backup-cron.md

@@ -12,9 +12,9 @@ backup of any kind was running). Overview of all legs:
 
 - `01:00` — **local leg**: `incus copy --refresh` of all instances
   into project `backup` on the sdb5 `backup` zpool (replaces the old
-  push to ks2; see [incus-copy.md](incus-copy.md)). The S3 instance
+  push to ks2; see [incus-copy.md](incus-copy.md)). An S3 leg for the
-  leg ([restic-backup.md](restic-backup.md) §6) chains onto this same
+  instance filesystems was designed and shelved
-  line once seeded — it reads the replicas this job just refreshed.
+  ([restic-backup.md](restic-backup.md) §6).
 - The rsync stopgap to ks2 (`incus-backup.sh`, 04:00) and the plakar
   data leg (04:30) are both **retired**; S3 backups are restic's job
   at 05:00.

ks4/restic-backup.md

@@ -5,8 +5,10 @@ backup tool (plakar crons off, its kloset kept for
 [#2338](https://github.com/PlakarKorp/plakar/issues/2338); see §8).
 Architecture: **two tools** — `incus copy` for replication (local sdb,
 nuc after FTTH), **restic** for everything S3. Data leg (`restic-data`)
-runs nightly at 05:00; the instance leg (`restic-incus`, §6) is written
+runs nightly at 05:00. The instance leg (`restic-incus`, §6) is
-and awaits its first seed.
+**shelved** (2026-08-28): instances are covered by replication, so
+duplicating their filesystems into S3 buys little. The script and the
+(empty) repo stay in place should that change.
 
 Why: plakar's incremental cost scales with tree size, not churn
 (~800 MiB parent metadata re-read from S3 + hours of CPU per night on
@@ -24,7 +26,7 @@ patches to maintain (see §6).
 | bucket = repo | contents | driver script |
 |---|---|---|
 | `restic-data` | SQL dumps + **all** fs sources (8 small + nextcloud + seafile) | `restic-backup.sh` |
-| `restic-incus` | the `backup`-project replicas, per-file via mount | `restic-incus-backup.sh` |
+| `restic-incus` | *(shelved — empty repo, see §6)* the `backup`-project replicas, per-file via mount | `restic-incus-backup.sh` |
 
 Mirrors the plakar-era split (data leg / incus leg). Accepted
 trade-off: `restic-data` carries the giants' blobs (~3–5 M once
@@ -183,8 +185,8 @@ Same shape as the plakar drivers (flock, env, loud logging). Phases:
    `--group-by host` because seed-era snapshots have different path
    sets than nightly ones and must age in one group).
 
-The instances leg is `restic-incus-backup.sh` (§6), chained after the
+The instances leg (`restic-incus-backup.sh`, §6) is shelved — nothing
-01:00 incus-copy once validated.
+chains onto the 01:00 line today.
 
 ```cron
 0 5 * * * /root/scripts/restic-backup.sh >> /var/log/restic-backup.log 2>&1
@@ -212,7 +214,15 @@ Then `restic check` (structure, cheap) + `check
 --read-data-subset=1/52` rotating — full data verification of every
 byte once a year, ~25 GiB read per week.
 
-## 6. Instances without a plugin — `incus file mount`
+## 6. Instances without a plugin — `incus file mount` (SHELVED)
+
+> Decided 2026-08-28: **not deployed.** Instances are replicated (sdb,
+> nuc after FTTH, ks2 in the meantime) and everything needed to
+> rebuild one is in `restic-data`, so a third copy of their
+> filesystems was judged not worth the nightly moving parts. The
+> design and the script remain, ready if that changes.
+> If the `restic-incus` bucket is ever deleted, drop it from `REPOS`
+> in `restic-maintenance.sh` too.
 
 Neither restic nor rustic accepts source plugins, and forking the
 backup tool is the wrong place to carry a patch. Instead, incus
@@ -264,9 +274,10 @@ diff -r … && rm -rf /backup/restore-test
   nextcloud tree, churn-independent; restic: seed at 60–115 MiB/s,
   local-cache incrementals). plakar's measurements stay in
   [plakar-s3-data.md](plakar-s3-data.md).
-- **Left to do**: seed `restic-incus` (§6) and chain it onto the 01:00
+- **Left to do**: restore test on the complete `restic-data`; re-enable
-  line; restore test on the complete `restic-data`; then the ks2
+  the interim ks2 push until FTTH; then the ks2 decommission gates
-  decommission gates ([../ks2/plan.md](../ks2/plan.md)).
+  ([../ks2/plan.md](../ks2/plan.md)). The instance leg (§6) is
+  shelved, not pending.
 - **plakar retirement, one exception**: keep plakar installed and
   keep the `plakar-data` kloset until
   [#2338](https://github.com/PlakarKorp/plakar/issues/2338) concludes

new-container.md

@@ -67,8 +67,7 @@ them forces a full re-send of its whole disk every night.
 **Automatic, nothing to do:**
 
 - the 01:00 replica leg copies *every* instance to the `backup` pool
-  and to nuc — new instances included;
+  and off-site — new instances included;
-- the S3 instance leg backs up every replica the same way;
 - **databases are auto-discovered**: any running container with
   MariaDB/MySQL or PostgreSQL gets every non-system database dumped
   nightly, plus users/grants.