Files

T

Julien LutranandClaude Fable 5 c18c8edc5b doc: add backup-strategy.md and new-container.md for non-specialist readers

backup-strategy.md: the two-tool architecture (incus copy + restic),
ASCII map of the legs (ks4 sdb / nuc pull / two S3 buckets), the
schedule table, a 30-second health check, and restore recipes for a
file, a database and a whole instance.

new-container.md: conventions for adding a service — one container
per service, no Docker (incus runs OCI natively, incus-compose for
stacks), gateway/proxy ingress, leave the snapshot schedule alone,
what the backups pick up automatically vs the one line to add to
restic-paths, DB-discovery requirements (incl. the .my.cnf trap), a
verification run, and the pitfalls (tmpfs /tmp, tiny-file swarms,
cron PATH).

README: start-here links, ks4 durability bullet and flow chart
updated to the restic era.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

2026-08-28 15:37:19 +02:00

5.8 KiB

Raw Blame History

ks4 backup strategy (2026-08)

Audience: anyone with root on ks4 who needs to know what is protected, where, by which tool, and how to check it. Details live in the linked docs; this page is the map.

Two tools, no more

tool	job	why this one
`incus copy --refresh`	replicate whole instances (ready-to-start copies)	ZFS-incremental, native to the platform, restores are `incus start`
restic	off-site backups to S3 (files, database dumps, instance trees)	dedup + encryption, local metadata cache → a night costs only the churn (restic-backup.md)

plakar was evaluated first and replaced (its nightly cost scaled with tree size — PlakarKorp/plakar#2338); the old backup server ks2 is retired (ks2/plan.md).

The map

 ks4 (OVH)                                            off-site
 ┌─────────────────────────────────────────────┐
 │  live instances        pool `data` (sda)    │
 │  nextcloud, seafile, mail, git, ...         │
 │        │                                    │
 │        │ 01:00  incus copy --refresh        │      nuc (home LAN, via WireGuard)
 │        ▼                                    │      ┌──────────────────────────┐
 │  replicas (stopped)    pool `backup` (sdb)  │─────▶│ 05:00 incus copy (pull)  │
 │  project `backup`                           │      │ pool `ks4backup` (USB)   │
 │        │                                    │      └──────────────────────────┘
 │        │ 01:00+ restic (via incus file mount)
 │        ▼                                    │      OVH Object Storage (S3, sbg)
 │  ─────────────────────────────────────────  │      ┌──────────────────────────┐
 │  05:00 restic-backup.sh                     │─────▶│ bucket restic-data       │
 │    • DB dumps (MariaDB/PostgreSQL, auto)    │      │  dumps + all data trees  │
 │    • data trees from /root/scripts/restic-paths     │                          │
 │  01:00+ restic-incus-backup.sh              │─────▶│ bucket restic-incus      │
 │    • every replica, per file + config yaml  │      │  instance filesystems    │
 │  Sun 14:00 restic-maintenance.sh            │      └──────────────────────────┘
 │    • prune + check (rotating full verify)   │
 └─────────────────────────────────────────────┘

Copies of any given byte: live → sdb replica (same box, other disk) → nuc replica (other site) → S3 (other site, other technology). Databases additionally get application-consistent dumps nightly.

Schedule (root crontab on ks4)

when	what	log
01:00 daily	`incus-copy.sh -p backup -s backup` — refresh all replicas onto sdb, then `restic-incus-backup.sh` → S3	`/var/log/incus-copy.log`, `/var/log/restic-incus.log`
03:00 daily	instance snapshots (incus profile, 7-day expiry) — what keeps the refreshes incremental	`incus info <inst>`
05:00 daily	`restic-backup.sh` — dumps + data trees → S3	`/var/log/restic-backup.log`
05:00 daily (nuc)	nuc pulls all replicas over WireGuard	nuc: `/var/log/incus-copy-ks4.log`
Sun 14:00	`restic-maintenance.sh` — prune (capped), structure check, 1/52 data verification	`/var/log/restic-maintenance.log`
1st of month 06:00 / quarterly 15th	seafile GC dry-run report / real GC (ks4/seafile-gc.md)	`/var/log/seafile-gc.log`

Retention on S3: 14 daily, 8 weekly, 6 monthly snapshots.

"Did last night work?" — the 30-second check

grep -aE "done \(rc=|FAILED" /var/log/incus-copy.log | tail -2
grep -aE "snapshot .* saved|done \(rc=|failed" /var/log/restic-backup.log | tail -3
grep -aE "done \(rc=|failed" /var/log/restic-incus.log | tail -2

rc=0 everywhere = fine. Any failed/FAILED line names the culprit (instance, database or path). All scripts exit non-zero on any error, never silently.

Restoring

A whole instance, fast (same box): incus copy backup:<inst> style — copy the replica from project backup back into default (ks4/incus-copy.md); from nuc the same via the remote.
A file or directory (any date within retention): restic -r s3:s3.sbg.io.cloud.ovh.net/restic-data restore <snapshot> --target /backup/restore-x --include <path> (never restore into /tmp — it is RAM). Env: . /root/.restic-env.
A database: restore the .sql dump from restic-data (/backup/dumps/mariadb/<inst>/<db>.sql, grants.sql for users), load it with incus exec <inst> -- mariadb < dump.sql.
An instance from S3 only (ks4 and both replica sites gone): restore its tree + <inst>.yaml from restic-incus, recreate the instance from the yaml/profile, push the tree back.

Secrets you need for any of this: /root/.restic-passphrase and the S3 keys in /root/.restic-env — both are in the password manager; without the passphrase the S3 buckets are unreadable.

Adding something to the backups

New instances are picked up automatically by the replica and instance legs (all instances, opt-out only). Databases are auto-discovered. Only data trees need one line in /root/scripts/restic-paths — see new-container.md.

5.8 KiB Raw Blame History