diff --git a/README.md b/README.md index 531e06f..26fecca 100644 --- a/README.md +++ b/README.md @@ -2,6 +2,11 @@ Personal infrastructure notes. +**Start here**: [backup-strategy.md](backup-strategy.md) — what is +backed up, by which tool, on what schedule, and how to restore. +[new-container.md](new-container.md) — conventions for adding a +service to ks4 (and getting it backed up automatically). + - [`nuc/`](nuc/) — home lab on `nuc` (see below) - [`ks4/`](ks4/) — prod server `ks4` at OVH (see below) - [`ks2/`](ks2/) — legacy backup server being decommissioned @@ -61,15 +66,14 @@ Incus host at OVH — public-facing self-hosted services. outline, login (SSO), wireguard, + more — full table in [ks4/install.md](ks4/install.md). - ⚠️ The ZFS `data` pool is single-disk (not mirrored); durability - rests on nightly root cron jobs — 01:00 `incus copy --refresh` of - all instances to the local `backup` pool on sdb5 + rests on nightly cron jobs — 01:00 `incus copy --refresh` of all + instances to the local `backup` pool on sdb5, then the instance leg + to S3; 05:00 restic (DB dumps + data trees) to S3; nuc pulls the + replicas over WireGuard. Full picture and restore procedures: + **[backup-strategy.md](backup-strategy.md)** ([ks4/local-backup-cron.md](ks4/local-backup-cron.md), - [ks4/incus-copy.md](ks4/incus-copy.md)), 04:30 plakar backups - (DB dumps + data trees) to S3 - ([ks4/plakar-s3-data.md](ks4/plakar-s3-data.md)), plus during the - transition a 04:00 rsync stopgap to ks2 - ([ks2 decommission plan](ks2/plan.md)); a pull leg to nuc pool - `ks4backup` awaits FTTH. + [ks4/incus-copy.md](ks4/incus-copy.md), + [ks4/restic-backup.md](ks4/restic-backup.md)). ## Network flows (nuc <-> ks4) @@ -90,10 +94,11 @@ Incus host at OVH — public-facing self-hosted services. | | | | | 03:00 instance snapshots | | 03:00 instance snapshots | | 03:30 incus-copy: all instances | | 01:00 incus-copy: all instances | -| -> project backup, pool | | -> project backup, zpool | -| nucbackup (usb4t/backup/nuc) | | backup (sdb5) [planned] | -| 05:00 apt upgrade all containers | | 04:00 incus-backup.sh: DB dumps | -| | | -> /backup (sdb5) [planned] | +| -> project backup, pool | | -> project backup, zpool sdb5 | +| nucbackup (usb4t/backup/nuc) | | then restic instance leg -> S3 | +| 05:00 pull ks4:* -> ks4backup | | 05:00 restic: DB dumps + data | +| [pending FTTH] | | trees -> S3 (restic-data) | +| 05:30 apt upgrade all containers | | Sun 14:00 restic maintenance | +-----------------------------------+ +-------------------------------------+ phones/laptops: WG peers 10.8.0.2-3 reach 192.168.1.x through the same endpoint ``` @@ -106,6 +111,10 @@ fixed endpoint; ks4's incus API is never exposed to the internet. - One markdown file per instance in `nuc/`, containing the full install script (idempotent-ish, run as root on the host), first-run configuration, and troubleshooting notes. -- Containers are built from `images:ubuntu/24.04` and configured +- Containers are built from Debian/Ubuntu images and configured exclusively through `incus exec`/`incus config` so the doc is the single source of truth — rebuilding = re-running the script. +- Adding a service to ks4: follow + [new-container.md](new-container.md) (no Docker in containers — + incus runs OCI images natively and `incus-compose` handles stacks; + data paths go into `scripts/restic-paths`). diff --git a/backup-strategy.md b/backup-strategy.md new file mode 100644 index 0000000..3d9ff79 --- /dev/null +++ b/backup-strategy.md @@ -0,0 +1,98 @@ +# ks4 backup strategy (2026-08) + +Audience: anyone with root on ks4 who needs to know **what is protected, +where, by which tool, and how to check it**. Details live in the linked +docs; this page is the map. + +## Two tools, no more + +| tool | job | why this one | +|---|---|---| +| **`incus copy --refresh`** | replicate whole instances (ready-to-start copies) | ZFS-incremental, native to the platform, restores are `incus start` | +| **restic** | off-site backups to S3 (files, database dumps, instance trees) | dedup + encryption, local metadata cache → a night costs only the churn ([restic-backup.md](ks4/restic-backup.md)) | + +plakar was evaluated first and replaced (its nightly cost scaled with +tree size — [PlakarKorp/plakar#2338](https://github.com/PlakarKorp/plakar/issues/2338)); +the old backup server **ks2 is retired** ([ks2/plan.md](ks2/plan.md)). + +## The map + +``` + ks4 (OVH) off-site + ┌─────────────────────────────────────────────┐ + │ live instances pool `data` (sda) │ + │ nextcloud, seafile, mail, git, ... │ + │ │ │ + │ │ 01:00 incus copy --refresh │ nuc (home LAN, via WireGuard) + │ ▼ │ ┌──────────────────────────┐ + │ replicas (stopped) pool `backup` (sdb) │─────▶│ 05:00 incus copy (pull) │ + │ project `backup` │ │ pool `ks4backup` (USB) │ + │ │ │ └──────────────────────────┘ + │ │ 01:00+ restic (via incus file mount) + │ ▼ │ OVH Object Storage (S3, sbg) + │ ───────────────────────────────────────── │ ┌──────────────────────────┐ + │ 05:00 restic-backup.sh │─────▶│ bucket restic-data │ + │ • DB dumps (MariaDB/PostgreSQL, auto) │ │ dumps + all data trees │ + │ • data trees from /root/scripts/restic-paths │ │ + │ 01:00+ restic-incus-backup.sh │─────▶│ bucket restic-incus │ + │ • every replica, per file + config yaml │ │ instance filesystems │ + │ Sun 14:00 restic-maintenance.sh │ └──────────────────────────┘ + │ • prune + check (rotating full verify) │ + └─────────────────────────────────────────────┘ +``` + +Copies of any given byte: live → sdb replica (same box, other disk) +→ nuc replica (other site) → S3 (other site, other technology). +Databases additionally get application-consistent dumps nightly. + +## Schedule (root crontab on ks4) + +| when | what | log | +|---|---|---| +| 01:00 daily | `incus-copy.sh -p backup -s backup` — refresh all replicas onto sdb, then `restic-incus-backup.sh` → S3 | `/var/log/incus-copy.log`, `/var/log/restic-incus.log` | +| 03:00 daily | instance snapshots (incus profile, 7-day expiry) — what keeps the refreshes incremental | `incus info ` | +| 05:00 daily | `restic-backup.sh` — dumps + data trees → S3 | `/var/log/restic-backup.log` | +| 05:00 daily (nuc) | nuc pulls all replicas over WireGuard | nuc: `/var/log/incus-copy-ks4.log` | +| Sun 14:00 | `restic-maintenance.sh` — prune (capped), structure check, 1/52 data verification | `/var/log/restic-maintenance.log` | +| 1st of month 06:00 / quarterly 15th | seafile GC dry-run report / real GC ([ks4/seafile-gc.md](ks4/seafile-gc.md)) | `/var/log/seafile-gc.log` | + +Retention on S3: 14 daily, 8 weekly, 6 monthly snapshots. + +## "Did last night work?" — the 30-second check + +```sh +grep -aE "done \(rc=|FAILED" /var/log/incus-copy.log | tail -2 +grep -aE "snapshot .* saved|done \(rc=|failed" /var/log/restic-backup.log | tail -3 +grep -aE "done \(rc=|failed" /var/log/restic-incus.log | tail -2 +``` + +`rc=0` everywhere = fine. Any `failed`/`FAILED` line names the culprit +(instance, database or path). All scripts exit non-zero on any error, +never silently. + +## Restoring + +- **A whole instance, fast (same box)**: `incus copy backup:` + style — copy the replica from project `backup` back into `default` + ([ks4/incus-copy.md](ks4/incus-copy.md)); from nuc the same via the + remote. +- **A file or directory** (any date within retention): + `restic -r s3:s3.sbg.io.cloud.ovh.net/restic-data restore --target /backup/restore-x --include ` + (never restore into `/tmp` — it is RAM). Env: `. /root/.restic-env`. +- **A database**: restore the `.sql` dump from `restic-data` + (`/backup/dumps/mariadb//.sql`, `grants.sql` for users), load + it with `incus exec -- mariadb < dump.sql`. +- **An instance from S3 only** (ks4 and both replica sites gone): + restore its tree + `.yaml` from `restic-incus`, recreate the + instance from the yaml/profile, push the tree back. + +Secrets you need for any of this: `/root/.restic-passphrase` and the S3 +keys in `/root/.restic-env` — **both are in the password manager**; +without the passphrase the S3 buckets are unreadable. + +## Adding something to the backups + +New instances are picked up automatically by the replica and +instance legs (all instances, opt-out only). Databases are +auto-discovered. Only **data trees** need one line in +`/root/scripts/restic-paths` — see [new-container.md](new-container.md). diff --git a/new-container.md b/new-container.md new file mode 100644 index 0000000..7d425d2 --- /dev/null +++ b/new-container.md @@ -0,0 +1,144 @@ +# Creating a new service on ks4 — conventions + +Audience: you have root on ks4 and want to add a service. Follow this +and the service is automatically replicated, backed up and +restorable, without touching the backup machinery. +Companion page: [backup-strategy.md](backup-strategy.md). + +## 1. One service = one incus container + +```sh +incus launch images:debian/13 myservice +incus config device add myservice eth0 nic nictype=bridged parent=incusbr0 \ + ipv4.address=192.168.1.30 # pick a free IP, note it in the doc +incus exec myservice -- bash # install and configure from here +``` + +Rules of thumb: + +- **Debian 13 or Ubuntu 24.04 images**, unprivileged (the default). + Privileged only with a written reason (`nuc/jellyfin-client.md` is + the one example: it needs raw device access). +- **Configure through `incus exec` and write the commands down** in + `doc/ks4/.md` as you go. The doc *is* the rebuild + procedure — that is the whole convention: re-running it recreates + the service. +- The container is on the NAT bridge `incusbr0` (192.168.1.0/24). It + is **not** reachable from the internet by itself. +- **Web services**: do not open ports. Add an nginx vhost in the + `gateway` container that proxies to `http://192.168.1.30:PORT`; + gateway is the single HTTP/S entry point and handles TLS. + Non-HTTP services (mail, wireguard) use an incus `proxy` device — + copy an existing one as a model. + +## 2. No Docker inside the container + +Docker-in-incus works (`login`/`outline` predate this rule) but costs +a second layer of networking, storage and updates, and hides the +service's data behind docker volumes. + +Since incus can run **OCI images natively**, an upstream +`docker-compose.yml` becomes incus instances: + +```sh +incus remote add docker https://docker.io --protocol=oci # once per host +incus launch docker:library/redis myservice-redis +``` + +and multi-container stacks are described with +[`incus-compose`](https://github.com/lxc/incus-compose) (upstream, +compose-file syntax → incus instances). Each piece is then a normal +instance: same network, same snapshots, same backups, `incus exec` to +debug, no nested runtime. + +If you truly need Docker (upstream ships only a compose stack you +don't want to translate), say so in the service doc and note where +the volumes live — you will need them in step 4. + +## 3. Snapshots (do nothing, but know why) + +The `default` profile gives every instance `snapshots.schedule = +0 3 * * *`, `snapshots.expiry = 7d`. Leave it. Those snapshots are +what make the nightly replication incremental — an instance without +them forces a full re-send of its whole disk every night. + +## 4. Wire it into the backups + +**Automatic, nothing to do:** + +- the 01:00 replica leg copies *every* instance to the `backup` pool + and to nuc — new instances included; +- the S3 instance leg backs up every replica the same way; +- **databases are auto-discovered**: any running container with + MariaDB/MySQL or PostgreSQL gets every non-system database dumped + nightly, plus users/grants. + +**Manual, one line:** data that lives in the filesystem (uploads, +repositories, mail spools…) must be listed in +`/root/scripts/restic-paths`, one absolute host path per line: + +``` +/var/lib/incus/storage-pools/data/containers/myservice/rootfs/var/lib/myservice +``` + +(The container's `/x` is `…/containers//rootfs/x` on the host.) +No comments in that file — every line is read as a path. Regenerable +caches can be skipped by adding a pattern to +`/root/scripts/restic-exclude`. Both files live in the `scripts` git +repo: edit there, commit, `git pull` on ks4. + +For the database auto-discovery to work, keep the defaults: + +- MariaDB/MySQL: client **and** dump binaries installed, root access + over the local unix socket. **Do not leave a `database = …` line in + `/root/.my.cnf`** — it breaks `SHOW DATABASES` and the container is + then skipped (this happened once and went unnoticed for months). +- PostgreSQL: reachable as the `postgres` system user (peer auth). +- PostgreSQL in Docker: the container's image name must contain + "postgres" (official images do). + +A database you do *not* want dumped (huge, static, re-importable) +goes into `/root/scripts/plakar-db-exclude` as `/` — +every skip is logged, so the list cannot rot silently. + +## 5. Verify, once + +```sh +cd /root/scripts && git pull +/root/scripts/restic-backup.sh -s dumps # ~2 min: see your DB in the log +ls /backup/dumps/mariadb/myservice/ # dump present? +/root/scripts/restic-backup.sh # full run, then: +. /root/.restic-env +restic -r s3:s3.sbg.io.cloud.ovh.net/restic-data ls latest | grep myservice | head +``` + +Seeing your paths and dumps in that listing means the service is +protected. Do this the day you create the service, not the day you +need it. + +## 6. Pitfalls (learned the hard way) + +- **`/tmp` on ks4 is a 16 GiB tmpfs — it is RAM.** Never restore, dump + or stage data there; use `/backup/…`. +- **Avoid millions of tiny files.** A service that keeps every version + of everything (seafile did, 24.9 M objects) makes every backup and + every filesystem walk take days. If the upstream software has a + garbage-collection or retention setting, turn it on *and schedule + it* the day you deploy — see [ks4/seafile-gc.md](ks4/seafile-gc.md). +- **Keep data under one predictable path** per service; scattered data + means several `restic-paths` lines and things get forgotten. +- **Static IP + a note in the service doc**; two containers fighting + over one IP is a confusing outage. +- Cron on ks4 does not have `/usr/local/bin` in its `PATH` — use + absolute paths in anything you schedule. + +## 7. Checklist + +- [ ] container created from a Debian/Ubuntu image, static IP noted +- [ ] no Docker (or a written reason + volume paths) +- [ ] install/config steps written in `doc/ks4/.md` +- [ ] exposed through gateway (HTTP) or a proxy device (other) +- [ ] snapshots left at the profile default +- [ ] data paths added to `restic-paths` (committed to git) +- [ ] `restic-backup.sh` run once; dumps and paths verified in the repo +- [ ] service added to the table in [ks4/install.md](ks4/install.md)