doc: instance leg to S3 shelved; ks2 push is the interim off-site leg
Decided 2026-08-28: instances are protected by replication (sdb now, nuc after FTTH, ks2 push in between), their data/DBs/configs by restic-data — a third copy of instance filesystems in S3 was not worth the moving parts. backup-strategy.md now explains the replication-vs-backup split and the rebuild path when no replica survives; restic-backup.md §6 is marked SHELVED (script and empty repo kept). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Julien Lutran
co-authored by
Claude Fable 5
Claude Fable 5
parent
7d5d1d9b6b
commit
b5aef914f5
+48
-32
@@ -19,45 +19,58 @@ copy of the instances until the nuc leg below takes over.
|
||||
|
||||
**Status 2026-08-28**: the local replica leg and the S3 data leg are
|
||||
live. The **nuc pull leg is waiting for FTTH** (expected before end of
|
||||
September) and the **S3 instance leg is written but not yet seeded** —
|
||||
until both land, instances have only the ks4-local replica off the
|
||||
live pool.
|
||||
September); until it lands, the ks2 push is re-enabled as an interim
|
||||
off-site copy of the instances. Backing up instance *images* to S3 was
|
||||
designed and shelved — S3 holds the data, the databases and the incus
|
||||
configuration, which is what a rebuild needs.
|
||||
|
||||
## The map
|
||||
|
||||
```
|
||||
ks4 (OVH) off-site
|
||||
┌─────────────────────────────────────────────┐
|
||||
│ live instances pool `data` (sda) │
|
||||
│ nextcloud, seafile, mail, git, ... │
|
||||
│ │ │
|
||||
│ │ 01:00 incus copy --refresh │ nuc (home LAN, via WireGuard)
|
||||
│ ▼ │ ┌──────────────────────────┐
|
||||
│ replicas (stopped) pool `backup` (sdb) │─────▶│ 05:00 incus copy (pull) │
|
||||
│ project `backup` │ │ pool `ks4backup` (USB) │
|
||||
│ │ │ └──────────────────────────┘
|
||||
│ │ 01:00+ restic (via incus file mount)
|
||||
│ ▼ │ OVH Object Storage (S3, sbg)
|
||||
│ ───────────────────────────────────────── │ ┌──────────────────────────┐
|
||||
│ 05:00 restic-backup.sh │─────▶│ bucket restic-data │
|
||||
│ • DB dumps (MariaDB/PostgreSQL, auto) │ │ dumps + all data trees │
|
||||
│ • data trees from /root/scripts/restic-paths │ │
|
||||
│ 01:00+ restic-incus-backup.sh │─────▶│ bucket restic-incus │
|
||||
│ • every replica, per file + config yaml │ │ instance filesystems │
|
||||
│ Sun 14:00 restic-maintenance.sh │ └──────────────────────────┘
|
||||
│ • prune + check (rotating full verify) │
|
||||
└─────────────────────────────────────────────┘
|
||||
ks4 (OVH) off-site
|
||||
┌────────────────────────────────────────────┐
|
||||
│ live instances pool `data` (sda) │
|
||||
│ nextcloud, seafile, mail, git, ... │
|
||||
│ │ │
|
||||
│ │ 01:00 incus copy --refresh │ nuc (home LAN, via WireGuard)
|
||||
│ ▼ │ ┌───────────────────────────┐
|
||||
│ replicas (stopped) pool `backup` (sdb) │────▶│ 05:00 incus copy (pull) │
|
||||
│ project `backup` ready to start │ │ pool `ks4backup` (USB) │
|
||||
│ │ └───────────────────────────┘
|
||||
│ ──────────────────────────────────────── │
|
||||
│ │ OVH Object Storage (S3, sbg)
|
||||
│ 05:00 restic-backup.sh │ ┌───────────────────────────┐
|
||||
│ • DB dumps (MariaDB/PostgreSQL, auto) │────▶│ bucket restic-data │
|
||||
│ • incus config + global DB dump │ │ everything needed to │
|
||||
│ • data trees (restic-paths) │ │ rebuild: dumps + data │
|
||||
│ │ └───────────────────────────┘
|
||||
│ Sun 14:00 restic-maintenance.sh │
|
||||
│ • prune + check (rotating full verify) │
|
||||
└────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
Copies of any given byte: live → sdb replica (same box, other disk)
|
||||
→ nuc replica (other site) → S3 (other site, other technology).
|
||||
Databases additionally get application-consistent dumps nightly.
|
||||
Two different kinds of protection, on purpose:
|
||||
|
||||
- **instances** (the running systems) are protected by *replication* —
|
||||
a ready-to-start copy on ks4's second disk and, after FTTH, on nuc.
|
||||
Restoring one is `incus copy` + `incus start`.
|
||||
- **the data inside them** (files, databases, incus configuration) is
|
||||
protected by *backup* — encrypted, deduplicated, versioned on S3,
|
||||
independent of ks4 and of the disks. Restoring means recreating the
|
||||
container (its config is in the S3 dumps, its install steps are in
|
||||
this repo) and pouring the data back.
|
||||
|
||||
Backing up whole instance images to S3 as well was designed
|
||||
([ks4/restic-backup.md](ks4/restic-backup.md) §6) and shelved: it
|
||||
duplicates data already covered, and the replication legs already give
|
||||
instances two homes.
|
||||
|
||||
## Schedule (root crontab on ks4)
|
||||
|
||||
| when | what | log |
|
||||
|---|---|---|
|
||||
| 01:00 daily | `incus-copy.sh -p backup -s backup` — refresh all replicas onto sdb, then `restic-incus-backup.sh` → S3 | `/var/log/incus-copy.log`, `/var/log/restic-incus.log` |
|
||||
| 01:00 daily | `incus-copy.sh -p backup -s backup` — refresh all replicas onto sdb | `/var/log/incus-copy.log` |
|
||||
| 02:00 daily | `incus-copy.sh -d ks2 -m push` — interim off-site replicas, until the nuc leg replaces it | `/var/log/incus-copy.log` |
|
||||
| 03:00 daily | instance snapshots (incus profile, 7-day expiry) — what keeps the refreshes incremental | `incus info <inst>` |
|
||||
| 05:00 daily | `restic-backup.sh` — dumps + data trees → S3 | `/var/log/restic-backup.log` |
|
||||
| 05:00 daily (nuc) | nuc pulls all replicas over WireGuard | nuc: `/var/log/incus-copy-ks4.log` |
|
||||
@@ -71,7 +84,6 @@ Retention on S3: 14 daily, 8 weekly, 6 monthly snapshots.
|
||||
```sh
|
||||
grep -aE "done \(rc=|FAILED" /var/log/incus-copy.log | tail -2
|
||||
grep -aE "snapshot .* saved|done \(rc=|failed" /var/log/restic-backup.log | tail -3
|
||||
grep -aE "done \(rc=|failed" /var/log/restic-incus.log | tail -2
|
||||
```
|
||||
|
||||
`rc=0` everywhere = fine. Any `failed`/`FAILED` line names the culprit
|
||||
@@ -90,9 +102,13 @@ never silently.
|
||||
- **A database**: restore the `.sql` dump from `restic-data`
|
||||
(`/backup/dumps/mariadb/<inst>/<db>.sql`, `grants.sql` for users), load
|
||||
it with `incus exec <inst> -- mariadb < dump.sql`.
|
||||
- **An instance from S3 only** (ks4 and both replica sites gone):
|
||||
restore its tree + `<inst>.yaml` from `restic-incus`, recreate the
|
||||
instance from the yaml/profile, push the tree back.
|
||||
- **An instance when no replica survives** (worst case: ks4 and the
|
||||
replica sites are gone): recreate the container from its page in
|
||||
[ks4/](ks4/) (the doc *is* the install script), restore its data
|
||||
trees and database dumps from `restic-data`, and use the incus
|
||||
configuration captured nightly in the dump tree
|
||||
(`/backup/dumps/incus/incus-global-db.sql`) to check devices,
|
||||
profiles and addresses.
|
||||
|
||||
Secrets you need for any of this: `/root/.restic-passphrase` and the S3
|
||||
keys in `/root/.restic-env` — **both are in the password manager**;
|
||||
|
||||
Reference in New Issue
Block a user