Julien LutranandClaude Fable 5
d7ec780747
backup-strategy.md now leads with the three target legs (local incus push to sdb, nuc pull over WireGuard, restic to S3 for DBs and selected trees) and carries the plakar/ks2 story in a closing History section. restic-backup.md, install.md, local-backup-cron.md and the ks2 docs describe the predecessor only as 'the first S3 implementation', with one operational note that plakar is still installed for issue #2338 reproduction. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
142 lines
7.8 KiB
Markdown
142 lines
7.8 KiB
Markdown
# ks4 backup strategy (2026-08)
|
|
|
|
Audience: anyone with root on ks4 who needs to know **what is protected,
|
|
where, by which tool, and how to check it**. Details live in the linked
|
|
docs; this page is the map.
|
|
|
|
## Two tools, three legs
|
|
|
|
| leg | mechanism | protects |
|
|
|---|---|---|
|
|
| **local replication** | `incus copy --refresh` → pool `backup` on ks4's second disk (sdb) | instances, against losing the `data` pool |
|
|
| **remote replication** | nuc pulls the same replicas over WireGuard → pool `ks4backup` | instances, against losing ks4 or the site |
|
|
| **remote backup** | **restic** → S3 bucket `restic-data`: database dumps + selected filesystem trees | the data itself, versioned and encrypted, independent of every disk above |
|
|
|
|
Only two tools are involved: `incus copy` (ZFS-incremental, native to
|
|
the platform — a restore is `incus start`) and restic (dedup +
|
|
encryption, local metadata cache, so a night costs only the churn —
|
|
[ks4/restic-backup.md](ks4/restic-backup.md)).
|
|
|
|
The old backup server **ks2 is being retired** (decommission by
|
|
Sep 30, 2026 — [ks2/plan.md](ks2/plan.md)); until FTTH enables the nuc
|
|
leg it still receives an interim replica push.
|
|
|
|
**Status 2026-08-28**: local replication and the S3 backup leg are
|
|
live. The **nuc pull leg waits for FTTH** (expected before end of
|
|
September); until then the ks2 push stands in. Backing up whole
|
|
instance *images* to S3 was considered and left out — S3 holds the
|
|
data, the databases and the incus configuration, which is what a
|
|
rebuild needs.
|
|
|
|
## The map
|
|
|
|
```
|
|
ks4 (OVH) off-site
|
|
┌────────────────────────────────────────────┐
|
|
│ live instances pool `data` (sda) │
|
|
│ nextcloud, seafile, mail, git, ... │
|
|
│ │ │
|
|
│ │ 01:00 incus copy --refresh │ nuc (home LAN, via WireGuard)
|
|
│ ▼ │ ┌───────────────────────────┐
|
|
│ replicas (stopped) pool `backup` (sdb) │────▶│ 05:00 incus copy (pull) │
|
|
│ project `backup` ready to start │ │ pool `ks4backup` (USB) │
|
|
│ │ └───────────────────────────┘
|
|
│ ──────────────────────────────────────── │
|
|
│ │ OVH Object Storage (S3, sbg)
|
|
│ 05:00 restic-backup.sh │ ┌───────────────────────────┐
|
|
│ • DB dumps (MariaDB/PostgreSQL, auto) │────▶│ bucket restic-data │
|
|
│ • incus config + global DB dump │ │ everything needed to │
|
|
│ • data trees (restic-paths) │ │ rebuild: dumps + data │
|
|
│ │ └───────────────────────────┘
|
|
│ Sun 14:00 restic-maintenance.sh │
|
|
│ • prune + check (rotating full verify) │
|
|
└────────────────────────────────────────────┘
|
|
```
|
|
|
|
Two different kinds of protection, on purpose:
|
|
|
|
- **instances** (the running systems) are protected by *replication* —
|
|
a ready-to-start copy on ks4's second disk and, after FTTH, on nuc.
|
|
Restoring one is `incus copy` + `incus start`.
|
|
- **the data inside them** (files, databases, incus configuration) is
|
|
protected by *backup* — encrypted, deduplicated, versioned on S3,
|
|
independent of ks4 and of the disks. Restoring means recreating the
|
|
container (its config is in the S3 dumps, its install steps are in
|
|
this repo) and pouring the data back.
|
|
|
|
Backing up whole instance images to S3 as well was designed
|
|
([ks4/restic-backup.md](ks4/restic-backup.md) §6) and shelved: it
|
|
duplicates data already covered, and the replication legs already give
|
|
instances two homes.
|
|
|
|
## Schedule (root crontab on ks4)
|
|
|
|
| when | what | log |
|
|
|---|---|---|
|
|
| 01:00 daily | `incus-copy.sh -p backup -s backup` — refresh all replicas onto sdb | `/var/log/incus-copy.log` |
|
|
| 02:00 daily | `incus-copy.sh -d ks2 -m push` — interim off-site replicas, until the nuc leg replaces it | `/var/log/incus-copy.log` |
|
|
| 03:00 daily | instance snapshots (incus profile, 7-day expiry) — what keeps the refreshes incremental | `incus info <inst>` |
|
|
| 05:00 daily | `restic-backup.sh` — dumps + data trees → S3 | `/var/log/restic-backup.log` |
|
|
| 05:00 daily (nuc) | nuc pulls all replicas over WireGuard | nuc: `/var/log/incus-copy-ks4.log` |
|
|
| Sun 14:00 | `restic-maintenance.sh` — prune (capped), structure check, 1/52 data verification | `/var/log/restic-maintenance.log` |
|
|
| 1st of month 06:00 / quarterly 15th | seafile GC dry-run report / real GC ([ks4/seafile-gc.md](ks4/seafile-gc.md)) | `/var/log/seafile-gc.log` |
|
|
|
|
Retention on S3: 14 daily, 8 weekly, 6 monthly snapshots.
|
|
|
|
## "Did last night work?" — the 30-second check
|
|
|
|
```sh
|
|
grep -aE "done \(rc=|FAILED" /var/log/incus-copy.log | tail -2
|
|
grep -aE "snapshot .* saved|done \(rc=|failed" /var/log/restic-backup.log | tail -3
|
|
```
|
|
|
|
`rc=0` everywhere = fine. Any `failed`/`FAILED` line names the culprit
|
|
(instance, database or path). All scripts exit non-zero on any error,
|
|
never silently.
|
|
|
|
## Restoring
|
|
|
|
- **A whole instance, fast (same box)**: `incus copy backup:<inst>`
|
|
style — copy the replica from project `backup` back into `default`
|
|
([ks4/incus-copy.md](ks4/incus-copy.md)); from nuc the same via the
|
|
remote.
|
|
- **A file or directory** (any date within retention):
|
|
`restic -r s3:s3.sbg.io.cloud.ovh.net/restic-data restore <snapshot> --target /backup/restore-x --include <path>`
|
|
(never restore into `/tmp` — it is RAM). Env: `. /root/.restic-env`.
|
|
- **A database**: restore the `.sql` dump from `restic-data`
|
|
(`/backup/dumps/mariadb/<inst>/<db>.sql`, `grants.sql` for users), load
|
|
it with `incus exec <inst> -- mariadb < dump.sql`.
|
|
- **An instance when no replica survives** (worst case: ks4 and the
|
|
replica sites are gone): recreate the container from its page in
|
|
[ks4/](ks4/) (the doc *is* the install script), restore its data
|
|
trees and database dumps from `restic-data`, and use the incus
|
|
configuration captured nightly in the dump tree
|
|
(`/backup/dumps/incus/incus-global-db.sql`) to check devices,
|
|
profiles and addresses.
|
|
|
|
Secrets you need for any of this: `/root/.restic-passphrase` and the S3
|
|
keys in `/root/.restic-env` — **both are in the password manager**;
|
|
without the passphrase the S3 buckets are unreadable.
|
|
|
|
## Adding something to the backups
|
|
|
|
New instances are picked up automatically by the replica and
|
|
instance legs (all instances, opt-out only). Databases are
|
|
auto-discovered. Only **data trees** need one line in
|
|
[`restic-paths`](https://git.lutran.fr/julien/scripts/src/branch/main/restic-paths) — see [new-container.md](new-container.md).
|
|
|
|
## History (reference only)
|
|
|
|
The road to the setup above, kept for the measurements rather than for
|
|
operations: an rsync-based leg to ks2
|
|
([ks2/plan.md](ks2/plan.md)) and a first S3 implementation with
|
|
**plakar** ([ks4/plakar-s3-data.md](ks4/plakar-s3-data.md), plus a
|
|
self-written incus connector,
|
|
[ks4/plakar-incus-integration.md](ks4/plakar-incus-integration.md)).
|
|
plakar was dropped because its nightly cost scaled with the size of
|
|
the tree rather than with the churn
|
|
([PlakarKorp/plakar#2338](https://github.com/PlakarKorp/plakar/issues/2338)).
|
|
It is still installed on ks4 and its `plakar-data` bucket still exists
|
|
— kept only so the upstream issue can be reproduced; nothing schedules
|
|
it any more.
|