doc/backup-strategy.md

ks4 backup strategy (2026-08)

Audience: anyone with root on ks4 who needs to know what is protected, where, by which tool, and how to check it. Details live in the linked docs; this page is the map.

Two tools, three legs

leg	mechanism	protects
local replication	incus copy --refresh → pool backup on ks4's second disk (sdb)	instances, against losing the data pool
remote replication	nuc pulls the same replicas over WireGuard → pool ks4backup	instances, against losing ks4 or the site
remote backup	restic → S3 bucket restic-data: database dumps + selected filesystem trees	the data itself, versioned and encrypted, independent of every disk above

Only two tools are involved: incus copy (ZFS-incremental, native to the platform — a restore is incus start) and restic (dedup + encryption, local metadata cache, so a night costs only the churn — ks4/restic-backup.md).

The old backup server ks2 is being retired (decommission by Sep 30, 2026 — ks2/plan.md); until FTTH enables the nuc leg it still receives an interim replica push.

Status 2026-08-28: local replication and the S3 backup leg are live. The nuc pull leg waits for FTTH (expected before end of September); until then the ks2 push stands in. Backing up whole instance images to S3 was considered and left out — S3 holds the data, the databases and the incus configuration, which is what a rebuild needs.

The map

 ks4 (OVH)                                           off-site
 ┌────────────────────────────────────────────┐
 │  live instances       pool `data` (sda)    │
 │  nextcloud, seafile, mail, git, ...        │
 │        │                                   │
 │        │ 01:00  incus copy --refresh       │     nuc (home LAN, via WireGuard)
 │        ▼                                   │     ┌───────────────────────────┐
 │  replicas (stopped)   pool `backup` (sdb)  │────▶│ 05:00 incus copy (pull)   │
 │  project `backup`     ready to start       │     │ pool `ks4backup` (USB)    │
 │                                            │     └───────────────────────────┘
 │  ────────────────────────────────────────  │
 │                                            │     OVH Object Storage (S3, sbg)
 │  05:00 restic-backup.sh                    │     ┌───────────────────────────┐
 │    • DB dumps (MariaDB/PostgreSQL, auto)   │────▶│ bucket restic-data        │
 │    • incus config + global DB dump         │     │  everything needed to     │
 │    • data trees (restic-paths)             │     │  rebuild: dumps + data    │
 │                                            │     └───────────────────────────┘
 │  Sun 14:00 restic-maintenance.sh           │
 │    • prune + check (rotating full verify)  │
 └────────────────────────────────────────────┘

Two different kinds of protection, on purpose:

• instances (the running systems) are protected by replication — a ready-to-start copy on ks4's second disk and, after FTTH, on nuc. Restoring one is incus copy + incus start.
• the data inside them (files, databases, incus configuration) is protected by backup — encrypted, deduplicated, versioned on S3, independent of ks4 and of the disks. Restoring means recreating the container (its config is in the S3 dumps, its install steps are in this repo) and pouring the data back.

Backing up whole instance images to S3 as well was designed (ks4/restic-backup.md §6) and shelved: it duplicates data already covered, and the replication legs already give instances two homes.

Schedule (root crontab on ks4)

when	what	log
01:00 daily	incus-copy.sh -p backup -s backup — refresh all replicas onto sdb	/var/log/incus-copy.log
02:00 daily	incus-copy.sh -d ks2 -m push — interim off-site replicas, until the nuc leg replaces it	/var/log/incus-copy.log
03:00 daily	instance snapshots (incus profile, 7-day expiry) — what keeps the refreshes incremental	incus info <inst>
05:00 daily	restic-backup.sh — dumps + data trees → S3	/var/log/restic-backup.log
05:00 daily (nuc)	nuc pulls all replicas over WireGuard	nuc: /var/log/incus-copy-ks4.log
Sun 14:00	restic-maintenance.sh — prune (capped), structure check, 1/52 data verification	/var/log/restic-maintenance.log
1st of month 06:00 / quarterly 15th	seafile GC dry-run report / real GC (ks4/seafile-gc.md)	/var/log/seafile-gc.log

Retention on S3: 14 daily, 8 weekly, 6 monthly snapshots.

"Did last night work?" — the 30-second check

grep -aE "done \(rc=|FAILED" /var/log/incus-copy.log | tail -2
grep -aE "snapshot .* saved|done \(rc=|failed" /var/log/restic-backup.log | tail -3

rc=0 everywhere = fine. Any failed/FAILED line names the culprit (instance, database or path). All scripts exit non-zero on any error, never silently.

Restoring

• A whole instance, fast (same box): incus copy backup:<inst> style — copy the replica from project backup back into default (ks4/incus-copy.md); from nuc the same via the remote.
• A file or directory (any date within retention): restic -r s3:s3.sbg.io.cloud.ovh.net/restic-data restore <snapshot> --target /backup/restore-x --include <path> (never restore into /tmp — it is RAM). Env: . /root/.restic-env.
• A database: restore the .sql dump from restic-data (/backup/dumps/mariadb/<inst>/<db>.sql, grants.sql for users), load it with incus exec <inst> -- mariadb < dump.sql.
• An instance when no replica survives (worst case: ks4 and the replica sites are gone): recreate the container from its page in ks4/ (the doc is the install script), restore its data trees and database dumps from restic-data, and use the incus configuration captured nightly in the dump tree (/backup/dumps/incus/incus-global-db.sql) to check devices, profiles and addresses.

Secrets you need for any of this: /root/.restic-passphrase and the S3 keys in /root/.restic-env — both are in the password manager; without the passphrase the S3 buckets are unreadable.

Adding something to the backups

New instances are picked up automatically by the replica and instance legs (all instances, opt-out only). Databases are auto-discovered. Only data trees need one line in restic-paths — see new-container.md.

History (reference only)

The road to the setup above, kept for the measurements rather than for operations: an rsync-based leg to ks2 (ks2/plan.md) and a first S3 implementation with plakar (ks4/plakar-s3-data.md, plus a self-written incus connector, ks4/plakar-incus-integration.md). plakar was dropped because its nightly cost scaled with the size of the tree rather than with the churn (PlakarKorp/plakar#2338). It is still installed on ks4 and its plakar-data bucket still exists — kept only so the upstream issue can be reproduced; nothing schedules it any more.