doc: instance leg to S3 shelved; ks2 push is the interim off-site leg
Decided 2026-08-28: instances are protected by replication (sdb now, nuc after FTTH, ks2 push in between), their data/DBs/configs by restic-data — a third copy of instance filesystems in S3 was not worth the moving parts. backup-strategy.md now explains the replication-vs-backup split and the rebuild path when no replica survives; restic-backup.md §6 is marked SHELVED (script and empty repo kept). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Julien Lutran
co-authored by
Claude Fable 5
Claude Fable 5
parent
7d5d1d9b6b
commit
b5aef914f5
+48
-32
@@ -19,45 +19,58 @@ copy of the instances until the nuc leg below takes over.
|
||||
|
||||
**Status 2026-08-28**: the local replica leg and the S3 data leg are
|
||||
live. The **nuc pull leg is waiting for FTTH** (expected before end of
|
||||
September) and the **S3 instance leg is written but not yet seeded** —
|
||||
until both land, instances have only the ks4-local replica off the
|
||||
live pool.
|
||||
September); until it lands, the ks2 push is re-enabled as an interim
|
||||
off-site copy of the instances. Backing up instance *images* to S3 was
|
||||
designed and shelved — S3 holds the data, the databases and the incus
|
||||
configuration, which is what a rebuild needs.
|
||||
|
||||
## The map
|
||||
|
||||
```
|
||||
ks4 (OVH) off-site
|
||||
┌─────────────────────────────────────────────┐
|
||||
│ live instances pool `data` (sda) │
|
||||
│ nextcloud, seafile, mail, git, ... │
|
||||
│ │ │
|
||||
│ │ 01:00 incus copy --refresh │ nuc (home LAN, via WireGuard)
|
||||
│ ▼ │ ┌──────────────────────────┐
|
||||
│ replicas (stopped) pool `backup` (sdb) │─────▶│ 05:00 incus copy (pull) │
|
||||
│ project `backup` │ │ pool `ks4backup` (USB) │
|
||||
│ │ │ └──────────────────────────┘
|
||||
│ │ 01:00+ restic (via incus file mount)
|
||||
│ ▼ │ OVH Object Storage (S3, sbg)
|
||||
│ ───────────────────────────────────────── │ ┌──────────────────────────┐
|
||||
│ 05:00 restic-backup.sh │─────▶│ bucket restic-data │
|
||||
│ • DB dumps (MariaDB/PostgreSQL, auto) │ │ dumps + all data trees │
|
||||
│ • data trees from /root/scripts/restic-paths │ │
|
||||
│ 01:00+ restic-incus-backup.sh │─────▶│ bucket restic-incus │
|
||||
│ • every replica, per file + config yaml │ │ instance filesystems │
|
||||
│ Sun 14:00 restic-maintenance.sh │ └──────────────────────────┘
|
||||
│ • prune + check (rotating full verify) │
|
||||
└─────────────────────────────────────────────┘
|
||||
ks4 (OVH) off-site
|
||||
┌────────────────────────────────────────────┐
|
||||
│ live instances pool `data` (sda) │
|
||||
│ nextcloud, seafile, mail, git, ... │
|
||||
│ │ │
|
||||
│ │ 01:00 incus copy --refresh │ nuc (home LAN, via WireGuard)
|
||||
│ ▼ │ ┌───────────────────────────┐
|
||||
│ replicas (stopped) pool `backup` (sdb) │────▶│ 05:00 incus copy (pull) │
|
||||
│ project `backup` ready to start │ │ pool `ks4backup` (USB) │
|
||||
│ │ └───────────────────────────┘
|
||||
│ ──────────────────────────────────────── │
|
||||
│ │ OVH Object Storage (S3, sbg)
|
||||
│ 05:00 restic-backup.sh │ ┌───────────────────────────┐
|
||||
│ • DB dumps (MariaDB/PostgreSQL, auto) │────▶│ bucket restic-data │
|
||||
│ • incus config + global DB dump │ │ everything needed to │
|
||||
│ • data trees (restic-paths) │ │ rebuild: dumps + data │
|
||||
│ │ └───────────────────────────┘
|
||||
│ Sun 14:00 restic-maintenance.sh │
|
||||
│ • prune + check (rotating full verify) │
|
||||
└────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
Copies of any given byte: live → sdb replica (same box, other disk)
|
||||
→ nuc replica (other site) → S3 (other site, other technology).
|
||||
Databases additionally get application-consistent dumps nightly.
|
||||
Two different kinds of protection, on purpose:
|
||||
|
||||
- **instances** (the running systems) are protected by *replication* —
|
||||
a ready-to-start copy on ks4's second disk and, after FTTH, on nuc.
|
||||
Restoring one is `incus copy` + `incus start`.
|
||||
- **the data inside them** (files, databases, incus configuration) is
|
||||
protected by *backup* — encrypted, deduplicated, versioned on S3,
|
||||
independent of ks4 and of the disks. Restoring means recreating the
|
||||
container (its config is in the S3 dumps, its install steps are in
|
||||
this repo) and pouring the data back.
|
||||
|
||||
Backing up whole instance images to S3 as well was designed
|
||||
([ks4/restic-backup.md](ks4/restic-backup.md) §6) and shelved: it
|
||||
duplicates data already covered, and the replication legs already give
|
||||
instances two homes.
|
||||
|
||||
## Schedule (root crontab on ks4)
|
||||
|
||||
| when | what | log |
|
||||
|---|---|---|
|
||||
| 01:00 daily | `incus-copy.sh -p backup -s backup` — refresh all replicas onto sdb, then `restic-incus-backup.sh` → S3 | `/var/log/incus-copy.log`, `/var/log/restic-incus.log` |
|
||||
| 01:00 daily | `incus-copy.sh -p backup -s backup` — refresh all replicas onto sdb | `/var/log/incus-copy.log` |
|
||||
| 02:00 daily | `incus-copy.sh -d ks2 -m push` — interim off-site replicas, until the nuc leg replaces it | `/var/log/incus-copy.log` |
|
||||
| 03:00 daily | instance snapshots (incus profile, 7-day expiry) — what keeps the refreshes incremental | `incus info <inst>` |
|
||||
| 05:00 daily | `restic-backup.sh` — dumps + data trees → S3 | `/var/log/restic-backup.log` |
|
||||
| 05:00 daily (nuc) | nuc pulls all replicas over WireGuard | nuc: `/var/log/incus-copy-ks4.log` |
|
||||
@@ -71,7 +84,6 @@ Retention on S3: 14 daily, 8 weekly, 6 monthly snapshots.
|
||||
```sh
|
||||
grep -aE "done \(rc=|FAILED" /var/log/incus-copy.log | tail -2
|
||||
grep -aE "snapshot .* saved|done \(rc=|failed" /var/log/restic-backup.log | tail -3
|
||||
grep -aE "done \(rc=|failed" /var/log/restic-incus.log | tail -2
|
||||
```
|
||||
|
||||
`rc=0` everywhere = fine. Any `failed`/`FAILED` line names the culprit
|
||||
@@ -90,9 +102,13 @@ never silently.
|
||||
- **A database**: restore the `.sql` dump from `restic-data`
|
||||
(`/backup/dumps/mariadb/<inst>/<db>.sql`, `grants.sql` for users), load
|
||||
it with `incus exec <inst> -- mariadb < dump.sql`.
|
||||
- **An instance from S3 only** (ks4 and both replica sites gone):
|
||||
restore its tree + `<inst>.yaml` from `restic-incus`, recreate the
|
||||
instance from the yaml/profile, push the tree back.
|
||||
- **An instance when no replica survives** (worst case: ks4 and the
|
||||
replica sites are gone): recreate the container from its page in
|
||||
[ks4/](ks4/) (the doc *is* the install script), restore its data
|
||||
trees and database dumps from `restic-data`, and use the incus
|
||||
configuration captured nightly in the dump tree
|
||||
(`/backup/dumps/incus/incus-global-db.sql`) to check devices,
|
||||
profiles and addresses.
|
||||
|
||||
Secrets you need for any of this: `/root/.restic-passphrase` and the S3
|
||||
keys in `/root/.restic-env` — **both are in the password manager**;
|
||||
|
||||
+9
-8
@@ -17,10 +17,10 @@ What remains on the box is **cold history**: instance replicas on pool
|
||||
`/backup/ns3061243` on pool `backup` (last refreshed 2026-08-28),
|
||||
snapshotted daily there (`zfs-auto-snapshot.sh`, 2-month expiry).
|
||||
|
||||
⚠️ Consequence while the nuc leg waits for FTTH: **instances have no
|
||||
fresh off-site copy** — only the ks4-local sdb replicas plus ks2's
|
||||
ageing ones. Either re-enable the interim push (below) or accept the
|
||||
gap knowingly until FTTH.
|
||||
⚠️ While the nuc leg waits for FTTH, instances have no *fresh*
|
||||
off-site copy — the ks2 push is to be re-enabled as soon as the
|
||||
initial restic sync finishes (decided 2026-08-28), and retired again
|
||||
when nuc takes over.
|
||||
|
||||
## Inventory findings (2026-08-22)
|
||||
|
||||
@@ -50,9 +50,9 @@ gap knowingly until FTTH.
|
||||
|---|---|---|
|
||||
| local, 2nd disk | `incus-copy.sh -p backup -s backup` → sdb5 pool | **live** (01:00) |
|
||||
| off-site, S3 (data+DB) | **restic** → bucket `restic-data` ([restic-backup.md](../ks4/restic-backup.md)) | **live** (05:00) |
|
||||
| off-site, S3 (instances) | restic over `incus file mount` of the `backup`-project replicas | written, **not seeded** |
|
||||
| off-site, S3 (instances) | restic over `incus file mount` of the `backup`-project replicas | **shelved** 2026-08-28 (replication covers instances) |
|
||||
| off-site, nuc | nuc pulls `ks4:*` → pool `ks4backup` over WG ([nuc-seed.md](nuc-seed.md)) | waiting FTTH (< Sep 30) |
|
||||
| off-site, ks2 (optional interim) | `incus-copy.sh -d ks2 -m push` at 02:00 until the nuc leg seeds | **not enabled** — decide |
|
||||
| off-site, ks2 (interim) | `incus-copy.sh -d ks2 -m push` at 02:00 until the nuc leg seeds | to re-enable once the restic seed finishes |
|
||||
|
||||
## Actions (backup work documented in [`../ks4/`](../ks4/), ks2-only tasks here)
|
||||
|
||||
@@ -64,8 +64,9 @@ gap knowingly until FTTH.
|
||||
3. ~~plakar S3 leg~~ → **replaced by restic**
|
||||
([restic-backup.md](../ks4/restic-backup.md), live 2026-08-28;
|
||||
plakar's own doc kept as history)
|
||||
4. instance leg — restic over `incus file mount`
|
||||
([restic-backup.md](../ks4/restic-backup.md) §6): **seed pending**
|
||||
4. ~~instance leg to S3~~ — **shelved 2026-08-28**: instances are
|
||||
protected by replication (sdb + nuc/ks2), their data and configs by
|
||||
`restic-data`
|
||||
5. [nuc-seed.md](nuc-seed.md) — **prepared**; after FTTH: seed nuc
|
||||
pull leg, verify all instances, test-restore one
|
||||
6. [decommission.md](decommission.md) — **prepared**; cut flows,
|
||||
|
||||
@@ -12,9 +12,9 @@ backup of any kind was running). Overview of all legs:
|
||||
|
||||
- `01:00` — **local leg**: `incus copy --refresh` of all instances
|
||||
into project `backup` on the sdb5 `backup` zpool (replaces the old
|
||||
push to ks2; see [incus-copy.md](incus-copy.md)). The S3 instance
|
||||
leg ([restic-backup.md](restic-backup.md) §6) chains onto this same
|
||||
line once seeded — it reads the replicas this job just refreshed.
|
||||
push to ks2; see [incus-copy.md](incus-copy.md)). An S3 leg for the
|
||||
instance filesystems was designed and shelved
|
||||
([restic-backup.md](restic-backup.md) §6).
|
||||
- The rsync stopgap to ks2 (`incus-backup.sh`, 04:00) and the plakar
|
||||
data leg (04:30) are both **retired**; S3 backups are restic's job
|
||||
at 05:00.
|
||||
|
||||
+20
-9
@@ -5,8 +5,10 @@ backup tool (plakar crons off, its kloset kept for
|
||||
[#2338](https://github.com/PlakarKorp/plakar/issues/2338); see §8).
|
||||
Architecture: **two tools** — `incus copy` for replication (local sdb,
|
||||
nuc after FTTH), **restic** for everything S3. Data leg (`restic-data`)
|
||||
runs nightly at 05:00; the instance leg (`restic-incus`, §6) is written
|
||||
and awaits its first seed.
|
||||
runs nightly at 05:00. The instance leg (`restic-incus`, §6) is
|
||||
**shelved** (2026-08-28): instances are covered by replication, so
|
||||
duplicating their filesystems into S3 buys little. The script and the
|
||||
(empty) repo stay in place should that change.
|
||||
|
||||
Why: plakar's incremental cost scales with tree size, not churn
|
||||
(~800 MiB parent metadata re-read from S3 + hours of CPU per night on
|
||||
@@ -24,7 +26,7 @@ patches to maintain (see §6).
|
||||
| bucket = repo | contents | driver script |
|
||||
|---|---|---|
|
||||
| `restic-data` | SQL dumps + **all** fs sources (8 small + nextcloud + seafile) | `restic-backup.sh` |
|
||||
| `restic-incus` | the `backup`-project replicas, per-file via mount | `restic-incus-backup.sh` |
|
||||
| `restic-incus` | *(shelved — empty repo, see §6)* the `backup`-project replicas, per-file via mount | `restic-incus-backup.sh` |
|
||||
|
||||
Mirrors the plakar-era split (data leg / incus leg). Accepted
|
||||
trade-off: `restic-data` carries the giants' blobs (~3–5 M once
|
||||
@@ -183,8 +185,8 @@ Same shape as the plakar drivers (flock, env, loud logging). Phases:
|
||||
`--group-by host` because seed-era snapshots have different path
|
||||
sets than nightly ones and must age in one group).
|
||||
|
||||
The instances leg is `restic-incus-backup.sh` (§6), chained after the
|
||||
01:00 incus-copy once validated.
|
||||
The instances leg (`restic-incus-backup.sh`, §6) is shelved — nothing
|
||||
chains onto the 01:00 line today.
|
||||
|
||||
```cron
|
||||
0 5 * * * /root/scripts/restic-backup.sh >> /var/log/restic-backup.log 2>&1
|
||||
@@ -212,7 +214,15 @@ Then `restic check` (structure, cheap) + `check
|
||||
--read-data-subset=1/52` rotating — full data verification of every
|
||||
byte once a year, ~25 GiB read per week.
|
||||
|
||||
## 6. Instances without a plugin — `incus file mount`
|
||||
## 6. Instances without a plugin — `incus file mount` (SHELVED)
|
||||
|
||||
> Decided 2026-08-28: **not deployed.** Instances are replicated (sdb,
|
||||
> nuc after FTTH, ks2 in the meantime) and everything needed to
|
||||
> rebuild one is in `restic-data`, so a third copy of their
|
||||
> filesystems was judged not worth the nightly moving parts. The
|
||||
> design and the script remain, ready if that changes.
|
||||
> If the `restic-incus` bucket is ever deleted, drop it from `REPOS`
|
||||
> in `restic-maintenance.sh` too.
|
||||
|
||||
Neither restic nor rustic accepts source plugins, and forking the
|
||||
backup tool is the wrong place to carry a patch. Instead, incus
|
||||
@@ -264,9 +274,10 @@ diff -r … && rm -rf /backup/restore-test
|
||||
nextcloud tree, churn-independent; restic: seed at 60–115 MiB/s,
|
||||
local-cache incrementals). plakar's measurements stay in
|
||||
[plakar-s3-data.md](plakar-s3-data.md).
|
||||
- **Left to do**: seed `restic-incus` (§6) and chain it onto the 01:00
|
||||
line; restore test on the complete `restic-data`; then the ks2
|
||||
decommission gates ([../ks2/plan.md](../ks2/plan.md)).
|
||||
- **Left to do**: restore test on the complete `restic-data`; re-enable
|
||||
the interim ks2 push until FTTH; then the ks2 decommission gates
|
||||
([../ks2/plan.md](../ks2/plan.md)). The instance leg (§6) is
|
||||
shelved, not pending.
|
||||
- **plakar retirement, one exception**: keep plakar installed and
|
||||
keep the `plakar-data` kloset until
|
||||
[#2338](https://github.com/PlakarKorp/plakar/issues/2338) concludes
|
||||
|
||||
+1
-2
@@ -67,8 +67,7 @@ them forces a full re-send of its whole disk every night.
|
||||
**Automatic, nothing to do:**
|
||||
|
||||
- the 01:00 replica leg copies *every* instance to the `backup` pool
|
||||
and to nuc — new instances included;
|
||||
- the S3 instance leg backs up every replica the same way;
|
||||
and off-site — new instances included;
|
||||
- **databases are auto-discovered**: any running container with
|
||||
MariaDB/MySQL or PostgreSQL gets every non-system database dumped
|
||||
nightly, plus users/grants.
|
||||
|
||||
Reference in New Issue
Block a user