From 79ad1b7179eb94ff7a2ae1ca6fbbf4b45a4c4850 Mon Sep 17 00:00:00 2001 From: Julien Lutran Date: Sun, 23 Aug 2026 08:56:28 +0200 Subject: [PATCH] Document the ks2 decommission plan and the new ks4 backup legs - ks2/: decommission plan with release gates (rental ends Sep 30), nuc-seed and final-decommission runbooks - ks4/local-backup-cron.md: leg-1 cron re-enabled, ks2 rsync stopgap, full-resend caveat when refreshes lapse past snapshots.expiry - ks4/plakar-s3-data.md: plakar direct-to-S3 leg (auto-discovered DB dumps incl. grants/globals, fs sources, exclude list, restore test) - ks4/plakar-incus-integration.md: design + scaffold status of the plakar incus importer - README.md: index ks2/, update the ks4 durability bullet Co-Authored-By: Claude Fable 5 --- README.md | 18 ++- ks2/decommission.md | 60 +++++++++ ks2/nuc-seed.md | 49 +++++++ ks2/plan.md | 78 +++++++++++ ks4/local-backup-cron.md | 56 ++++++++ ks4/plakar-incus-integration.md | 119 +++++++++++++++++ ks4/plakar-s3-data.md | 221 ++++++++++++++++++++++++++++++++ 7 files changed, 597 insertions(+), 4 deletions(-) create mode 100644 ks2/decommission.md create mode 100644 ks2/nuc-seed.md create mode 100644 ks2/plan.md create mode 100644 ks4/local-backup-cron.md create mode 100644 ks4/plakar-incus-integration.md create mode 100644 ks4/plakar-s3-data.md diff --git a/README.md b/README.md index 2afad70..531e06f 100644 --- a/README.md +++ b/README.md @@ -4,6 +4,11 @@ Personal infrastructure notes. - [`nuc/`](nuc/) — home lab on `nuc` (see below) - [`ks4/`](ks4/) — prod server `ks4` at OVH (see below) +- [`ks2/`](ks2/) — legacy backup server being decommissioned + (rental ends Sep 30, 2026): [plan](ks2/plan.md) +- [`archer-c7/`](archer-c7/) — home router (TP-Link Archer C7 v5, + OpenWrt, `192.168.0.2`, WAN = phone USB tether): + [upgrade to 25.12](archer-c7/upgrade-openwrt-25.12.md) ## Homelab: nuc @@ -56,10 +61,15 @@ Incus host at OVH — public-facing self-hosted services. outline, login (SSO), wireguard, + more — full table in [ks4/install.md](ks4/install.md). - ⚠️ The ZFS `data` pool is single-disk (not mirrored); durability - rests on two nightly root cron jobs — `incus copy --refresh` to remote - host `ks2` (moving to nuc pool `ks4backup`, see - [ks4/incus-copy.md](ks4/incus-copy.md)), and an rsync backup - (`scripts/incus-backup.sh`) to `164.132.173.57:/backup/ks4`. + rests on nightly root cron jobs — 01:00 `incus copy --refresh` of + all instances to the local `backup` pool on sdb5 + ([ks4/local-backup-cron.md](ks4/local-backup-cron.md), + [ks4/incus-copy.md](ks4/incus-copy.md)), 04:30 plakar backups + (DB dumps + data trees) to S3 + ([ks4/plakar-s3-data.md](ks4/plakar-s3-data.md)), plus during the + transition a 04:00 rsync stopgap to ks2 + ([ks2 decommission plan](ks2/plan.md)); a pull leg to nuc pool + `ks4backup` awaits FTTH. ## Network flows (nuc <-> ks4) diff --git a/ks2/decommission.md b/ks2/decommission.md new file mode 100644 index 0000000..31625c3 --- /dev/null +++ b/ks2/decommission.md @@ -0,0 +1,60 @@ +# ks2 decommission — final checklist + +Status: **prepared — execute only when every gate in +[plan.md](plan.md) is ticked.** Rental ends **Sep 30, 2026**; leave a +few days of margin. + +## Preconditions (the plan's release gates) + +- [ ] plakar S3 leg running nightly for ≥ a week, `done (rc=0)`, + restore test passed ([plakar-s3-data.md](../ks4/plakar-s3-data.md) §7) +- [ ] nuc pull leg seeded and one instance test-restored + ([nuc-seed.md](nuc-seed.md)) +- [ ] local leg (sdb5) cron green in `/var/log/incus-copy.log` + +## 1. Cut the last flows to ks2 (root on ks4) + +```sh +crontab -e # delete the 04:00 incus-backup.sh line (164.132.173.57) +incus remote list # if a "ks2" remote is still defined: +incus remote remove ks2 +# drop the rsyncbackup SSH key if it exists only for ks2: +ls /root/.ssh/id_rsyncbackup* && rm -i /root/.ssh/id_rsyncbackup* +``` + +## 2. Last look at ks2 before wiping (root on ks2) + +```sh +# anything in the rsync tree newer than the plakar cutover? +find /backup/ns3061243 -newermt '2026-08-23' | head +# pre-2023 dirs (catc, mythoughts, qcm, scripts): instances still +# exist stopped on ks4 — spot-check nothing unique remains: +ls /backup/ns3061243/{catc,mythoughts,qcm,scripts} +``` + +Archive anything unique to the plakar dump dir on ks4 (it rides the +nightly `dumps` snapshot to S3 for free once it stops changing). + +## 3. Wipe and terminate + +```sh +# on ks2 — destroy the data (both pools hold copies of personal data): +zpool destroy backup && zpool destroy data +# optional: shred the OS disks via OVH rescue mode / reinstall +``` + +Then in the OVH manager: cancel the ks2 (Kimsufi) server before the +renewal date. + +## 4. Post-mortem cleanup + +- [ ] Update [doc/README.md](../README.md): remove ks2 from the ks4 + durability bullet and the index line for `ks2/`s "being + decommissioned" wording (keep the folder as history) +- [ ] Retire `scripts/incus-backup.sh` + `incus-backup.db` in the repo + (superseded by `scripts/plakar-backup.sh`) — keep for reference, + note the retirement in their headers +- [ ] `login` container cleanup parked in + [plakar-s3-data.md](../ks4/plakar-s3-data.md): `dropdb outline`, + remove native postgres +- [ ] Mark the ks2 line in `plan.md` done, date it diff --git a/ks2/nuc-seed.md b/ks2/nuc-seed.md new file mode 100644 index 0000000..f084978 --- /dev/null +++ b/ks2/nuc-seed.md @@ -0,0 +1,49 @@ +# nuc pull leg — seed after FTTH + +Status: **prepared, waiting on the FTTH link.** Everything is already +configured on nuc (see the main [README](../README.md)): incus remote +`ks4` over the WireGuard tunnel (`wg-ks4`, 10.8.0.20 → 10.8.0.1), +target pool `ks4backup` on the USB ZFS pool (`usb4t/backup/ks4`) — +only the seed itself waited on bandwidth. + +## Seed (root on nuc) + +```sh +# sanity: remote reachable through the tunnel +incus list ks4: | head + +# full pull of every ks4 instance into pool ks4backup (screen/tmux — +# first pass moves ~1.7 T through the WG tunnel) +/root/scripts/incus-copy.sh -r ks4 -s ks4backup 2>&1 | tee -a /var/log/incus-copy-ks4.log +``` + +Notes: + +- First pass is a full send per instance; later refreshes are + ZFS-incremental **as long as they run at least every + `snapshots.expiry` (7 d on ks4)** — same caveat as + [ks4's local leg](../ks4/local-backup-cron.md). +- Replicas arrive stopped with `boot.autostart=false` (script does + this) — they must never come up on the LAN with ks4's proxy devices. + +## Cron (after the seed) + +Add to nuc's root crontab, offset from the 03:30 local nucbackup copy +and ks4's own 01:00/04:30 jobs: + +```cron +0 5 * * * /root/scripts/incus-copy.sh -r ks4 -s ks4backup >> /var/log/incus-copy-ks4.log 2>&1 +``` + +## Verification (release gate for ks2) + +```sh +incus list --project backup 2>/dev/null || incus list | grep -c . # all ks4 instances present +# test-restore one instance: copy a replica to the default pool, +# start it isolated, check the service answers, then delete it +incus copy solar solar-restoretest -s default +incus start solar-restoretest && incus exec solar-restoretest -- systemctl is-system-running +incus delete -f solar-restoretest +``` + +Once verified, tick the nuc gate in the [ks2 plan](plan.md). diff --git a/ks2/plan.md b/ks2/plan.md new file mode 100644 index 0000000..9315101 --- /dev/null +++ b/ks2/plan.md @@ -0,0 +1,78 @@ +# ks2 decommission — summary plan + +Goal: release `ks2.lutran.fr` (`164.132.173.57`, SSH :2233, host +`ns3247221`) once ks4 has a real 3-2-1 backup without it. +**Deadline: rental ends Sep 30, 2026.** + +## What ks2 does today (updated 2026-08-22) + +One remaining role, fed nightly by a root cron **on ks4**: + +- `04:00` `incus-backup.sh … -d 164.132.173.57` — DB dumps + selected + paths rsync'd to ks2 `/backup/ns3061243` on pool `backup` (1.59 T, + 88 % full), snapshotted daily on ks2 (`zfs-auto-snapshot.sh`, + 2-month expiry). **Kept as a stopgap** until the plakar S3 leg + replaces it. + +Retired: the `01:00` replica push to ks2 pool `data` — replaced by +the local leg on ks4's sdb5 pool +([local-backup-cron.md](../ks4/local-backup-cron.md)); `biwiki` and `spot` +deleted from ks2. + +## Inventory findings (2026-08-22) + +- `biwiki` exists only on ks2, `spot` only on ks2 + ks4's `backup` + project — **decision: both abandoned**, safe to delete from backups. +- `incus-backup.db` is stale (lists `spot`, misses `livetrail`, + `outline`, `login`, …) — forgotten-manifest drift is exactly what + the plakar-based replacement must eliminate (back up *all* + instances by default, opt-out instead of opt-in). +- `livetrail` (running on ks4) is **absent from ks4's local `backup` + project** — the local leg was run manually once and never again; + automating its cron fixes this. +- Both ks4 backup crons were **commented out — no ks4 backup ran at + all** (hence the stale ks2 rsync data: dirs `Jul 23`, incus dumps + `Aug 9`). Fixed 2026-08-22 — see + [local-backup-cron.md](../ks4/local-backup-cron.md). +- `/backup/ns3061243` also holds pre-2023 dirs (`catc`, `mythoughts`, + `qcm`) — those instances still exist stopped on ks4, so nothing + unique expected there; spot-check before wiping. +- Losing ks2 also loses its 2 months of rsync-snapshot history — + acceptable once plakar has built equivalent retention. + +## Target architecture (3-2-1 for ks4) + +| Leg | Mechanism | Status | +|---|---|---| +| local, 2nd disk | `incus-copy.sh -p backup -s backup` → sdb5 pool | works, **cron missing** | +| off-site, nuc | nuc pulls `ks4:*` → pool `ks4backup` over WG | waiting FTTH seed | +| off-site, S3 (data+DB) | plakar on ks4 → S3 (replaces `incus-backup.sh`) | to do | +| off-site, S3 (instances) | plakar **incus integration** (to develop) → S3 | to do | + +## Actions (backup work documented in [`../ks4/`](../ks4/), ks2-only tasks here) + +1. ~~salvage biwiki/spot~~ — decided 2026-08-22: both abandoned, + **deleted** from ks2 and ks4's `backup` project +2. ~~[local-backup-cron.md](../ks4/local-backup-cron.md)~~ — **done + 2026-08-22**: leg 1 cron re-enabled, ks2 rsync kept as stopgap; + `livetrail` verified present in the `backup` project +3. [plakar-s3-data.md](../ks4/plakar-s3-data.md) — **seeding S3 + 2026-08-22** (direct-to-S3 kloset `lutran-ks4-plakar-data`, driver + script `scripts/plakar-backup.sh`, pipeline validated locally); + next: restore test + cron +4. [plakar-incus-integration.md](../ks4/plakar-incus-integration.md) — + design seeded 2026-08-22: per-file importer over the incus sftp + API, instances → S3 +5. [nuc-seed.md](nuc-seed.md) — **prepared**; after FTTH: seed nuc + pull leg, verify all instances, test-restore one +6. [decommission.md](decommission.md) — **prepared**; cut flows, + final diff of `/backup/ns3061243`, wipe pools, terminate at OVH + +## Release gates — ks2 can be dropped only when + +- [x] biwiki + spot consciously abandoned (2026-08-22) +- [ ] local leg cron running ≥ a few days, all instances present +- [ ] nuc leg fully seeded **and** one instance test-restored +- [ ] plakar S3 data/DB backups running daily **and** one DB + one + file tree test-restored +- [ ] ks4 crons pointing at ks2 disabled diff --git a/ks4/local-backup-cron.md b/ks4/local-backup-cron.md new file mode 100644 index 0000000..972a3ad --- /dev/null +++ b/ks4/local-backup-cron.md @@ -0,0 +1,56 @@ +# local backup cron on ks4 (leg 1) — re-enabled + +Done 2026-08-22. Context: **both** ks4 backup crons had been commented +out — no backup of any kind was running. + +## What runs now (root crontab on ks4) + +```cron +0 1 * * * /root/scripts/incus-copy.sh -p backup -s backup >> /var/log/incus-copy.log 2>&1 +0 4 * * * /root/scripts/incus-backup.sh -d 164.132.173.57 -u rsyncbackup -i /root/.ssh/id_rsyncbackup -p 2233 -f /root/scripts/incus-backup.db -s data >> /var/log/incus-backup.log 2>&1 +``` + +- `01:00` — **local leg**: `incus copy --refresh` of all instances + into project `backup` on the sdb5 `backup` zpool (replaces the old + push to ks2; see [incus-copy.md](incus-copy.md)). +- `04:00` — **stopgap**: `incus-backup.sh` rsync to ks2 kept alive + (manifest pruned of `spot`) until the plakar S3 leg replaces it — + then this line goes away as part of + [decommissioning ks2](../ks2/plan.md). ⚠️ keep + `scripts/incus-backup.db` in this repo in sync with + `/root/scripts/incus-backup.db` on ks4. + +Both were also run once by hand (in a `screen`) on 2026-08-22 to +refresh the backups immediately. The old push-to-ks2 copy line stays +commented out — replica refreshes to ks2 are retired. + +## Cleanup done the same day + +- ks2: `incus delete biwiki spot` (abandoned instances — freed space + on the 95 %-full `data` pool) +- ks4: `incus delete spot --project backup` + +## Verification + +```bash +# all instances present, incl. livetrail (first copy is a full send) +sudo incus list --project backup -c ns -f csv +# cron runs land here (manual screen runs don't): +sudo tail /var/log/incus-copy.log /var/log/incus-backup.log +# rsync leg freshness, on ks2: +ls -lat /backup/ns3061243/ | head +``` + +Notes: + +- `flock` in both scripts makes overlapping cron/manual runs abort + safely instead of racing. +- Replicas get `boot.autostart=false` set by the script — they must + never fight the live instances for their static IPs. +- Keep `snapshots.schedule` set on the sources so refreshes stay + ZFS-incremental (a refresh without common snapshots falls back to + rsync). Corollary: the cron must run at least every + `snapshots.expiry` (7 d) — a longer gap expires every common + snapshot and the next refresh silently degrades to a **full + re-send** (observed 2026-08-22: seafile re-sent 933 G after the + weeks-long gap while the crons were commented out). diff --git a/ks4/plakar-incus-integration.md b/ks4/plakar-incus-integration.md new file mode 100644 index 0000000..f7d6818 --- /dev/null +++ b/ks4/plakar-incus-integration.md @@ -0,0 +1,119 @@ +# plakar incus integration — design notes + +Status: **importer scaffolded and compiling 2026-08-23** in +`perso/plakar/integration-incus` (walker modeled on the official sftp +integration; enumerate project → per-instance sftp walk → per-file +records + synthetic `.incus.yaml`; allow/deny lists; VM sftp failures +reported as record errors, not fatal). See its README for +build/install/TODO. Untested against a live incus yet. + +Goal: plakar source connector for Incus so instance backups land in a +plakar kloset on S3 (leg 4 of the ks4 3-2-1 plan — see +[plan.md](../ks2/plan.md)). + +## Decision 1 — per-file importer, not tarball streaming + +The proxmox integration streams one full vzdump archive per VM +(`FLAG_STREAM`). For ks4 that model reads ~1.7 TB nightly (seafile +933 G, nextcloud 725 G): the incus backup/export API produces **full +tarballs only** — `shared/api/instance_backup.go` has just +`instance_only` and `optimized_storage`, no incremental option. + +Instead the importer emits **one record per file** with real stat info +and a lazy reader. kloset's engine then does the incremental work for +us: `snapshot/backup.go` (`checkVFSCache`) reuses the cached object +when a path's stat matches the previous run and **never opens the +file**. Nightly cost after the initial seed = stat-walk + changed +files only; S3 receives only new chunks. Bonus: per-file browse and +restore of any snapshot. + +File access is native incus REST: `GET /1.0/instances/{name}/sftp` +(a real SFTP session, also available per storage volume). Works over +the local unix socket or remote :8443, on stopped containers (the +daemon mounts the volume on demand), no ZFS assumptions. + +## Decision 2 — do not reuse `incus copy --refresh` / migration API + +Considered and rejected. `copy --refresh` is `POST /1.0/instances` +with `source.refresh=true` (`shared/api/instance.go:373`) and is +**not** zfs-only — transports are negotiated per storage driver +(`MigrationTypes()`: zfs/btrfs/ceph native streams, universal RSYNC +fallback, `driver_common.go:230`). But: + +- the receiver must be another incus daemon speaking the migration + websocket protocol (`internal/migration/migrate.proto`); +- the optimized payload is an opaque `zfs send` stream — storing + those in plakar means chain-of-increments restore onto a real + zpool, no per-file browse, no pruning, one corrupt link breaks the + chain; +- the RSYNC transport needs a materialized previous copy on the + receiver — a full-size local mirror, i.e. leg 1 rebuilt inside a + plugin; +- gotcha: `driver_zfs.go:801` — refresh **without** snapshot copying + forces RSYNC even zfs↔zfs (keep `snapshots.schedule` on sources). + +`copy --refresh` stays what legs 1–2 use (host↔host replication); +plakar is content-addressed archiving to dumb storage. Complementary, +not competing. + +## Decision 3 — read the `backup` project replicas, not live instances + +Reading a live rootfs is fuzzy. Leg 1 (`incus-copy.sh -p backup -s +backup`, 01:00) already produces quiescent, crash-consistent stopped +replicas — the importer targets those (`?project=backup`), scheduled +after the copy completes. DB consistency remains the mariadb-dump +job's responsibility (plakar `mysql` importer / dump step). + +## Decision 4 — enumerate everything by default + +`incus-backup.db` manifest drift (stale `spot`, missing `livetrail`, +`outline`, `login`…) is the failure mode to kill: the importer backs +up **all instances in the project by default**, opt-out via config, +never opt-in. + +## Other design points + +- Location scheme `incus://///...` — + stable pathnames keep the kloset VFS cache effective. +- Synthetic records per instance: config + profiles + (`GET /1.0/instances/{name}`, expanded) so restore can recreate the + instance before pushing files back. +- Optional later `mode=image`: full tarball via the backups API for + occasional exact-image restores (`incus import`); plakar dedups + unchanged chunks between runs. Not the nightly path. +- Restore path (exporter): create instance from stored config → push + file tree back via sftp. Weaker than `incus import` but granular. + +## Packaging & deployment + +Plugins are standalone executables (gRPC over stdio) — no rebuild of +plakar, fully compatible with the APT-installed binary on ks4: + +```sh +cd plakar/integration-incus && make # build importer/exporter +plakar pkg create manifest.yaml v0.1.0 # → incus_v0.1.0_linux_amd64.ptar +# on ks4, as root (plugins are per-user, cron runs as root): +plakar pkg add ./incus_v0.1.0_linux_amd64.ptar +plakar source add ks4-incus incus://... # then list it in /root/scripts/plakar-sources +``` + +Installing a *local* `.ptar` file needs no `plakar login` (the plugin +registry is only for fetching by name). Model the `manifest.yaml` on +integration-proxmox (`tier: third-party`, `api_version: v1.1.0`). +Compatibility contract = the go-kloset-sdk wire protocol: pin the SDK +to the installed plakar line (v1.1.x), rebuild the ptar on a plakar +major upgrade; a mismatch surfaces at backup time, so run a manual +`plakar backup @ks4-incus` after plakar upgrades. + +## Open questions + +- uid/gid view through the instance sftp endpoint (idmap handling for + unprivileged containers) — verify what stat returns and what restore + must remap. +- xattrs / ACLs / device nodes over sftp — coverage and fidelity. +- VMs: file access needs a running incus-agent — out of scope for ks4 + (containers only); document the limitation. +- Scheduling/locking: chain after the 01:00 incus-copy (flock on the + same lock, or a wrapper script). +- S3 hardening: scoped credentials (no delete), versioning/object + lock, plakar client-side encryption passphrase storage. diff --git a/ks4/plakar-s3-data.md b/ks4/plakar-s3-data.md new file mode 100644 index 0000000..458dd96 --- /dev/null +++ b/ks4/plakar-s3-data.md @@ -0,0 +1,221 @@ +# plakar data + DB backups on ks4 (leg 3) + +Status: **pipeline validated 2026-08-22** against a local test kloset +(`/backup/plakar-test`): full dump auto-discovery incl. grants/globals, +exclude mechanism, fs sources, prune + maintenance, rc=0; second run +confirmed incremental behavior (21 GiB dumps snapshot → 60 MiB +written). **S3 credentials ready — next: store add, create, seed.** +Replaces `incus-backup.sh` (the 04:00 rsync to ks2 stays alive as a +stopgap until this leg is verified — see +[local-backup-cron.md](local-backup-cron.md)). + +Architecture: plakar runs **on ks4** (decision recorded in +[plan.md](../ks2/plan.md)) and backs up **directly into an S3 kloset** +(`plakar at @s3 backup @src`). A local staging kloset on sdb5 was +considered and rejected: it would double-store ~1.5 T that the leg-1 +replicas already hold on the same disk, and local restore is already +leg 1's job. Direct-to-S3 keeps the incremental behavior — the +skip-unchanged baseline is the parent snapshot's VFS read from the +kloset (plakar caches repository metadata locally; v1.1.4's dirpack +prefetcher targets exactly this remote-backend walk). Driver script: +[`scripts/plakar-backup.sh`](../../scripts/plakar-backup.sh) +(flock, incus DB dumps, iterates a sources file, prune + maintenance; +its `-k` takes any `plakar at` target, so `-k @s3` — a throwaway +local kloset, e.g. `-k /backup/plakar-test`, works for testing before +credentials arrive and can be deleted after). + +## 1. Install plakar via the official APT repo (root on ks4) + +Repo verified reachable 2026-08-22 (suite `stable`, amd64/arm64); +upgrades then ride the normal `apt upgrade` flow. + +```sh +curl -fsSL https://plakar.io/dist/keys/community-v1.1.0.gpg \ + | gpg --dearmor -o /usr/share/keyrings/plakar.gpg +echo "deb [signed-by=/usr/share/keyrings/plakar.gpg] https://plakar.io/dist/repos/deb/ stable main" \ + > /etc/apt/sources.list.d/plakar.list +apt update && apt install plakar +plakar version # v1.1.4 at time of writing +``` + +The mysql integration is a plugin; official plugins install from the +plugin server after `plakar login`, or build locally (needs make + Go): + +```sh +plakar pkg add mysql # or: clone integrations, cd mysql && make install +``` + +## 2. Kloset + passphrase + +```sh +head -c 32 /dev/urandom | base64 > /root/.plakar-passphrase +chmod 600 /root/.plakar-passphrase +# ⚠️ copy the passphrase to the password manager NOW — without it +# the S3 kloset is unreadable garbage. +export PLAKAR_PASSPHRASE=$(cat /root/.plakar-passphrase) + +# when S3 credentials are available: +plakar store add s3 s3:/// access_key= secret_access_key= use_tls=true +plakar at @s3 create +# meanwhile, a throwaway kloset lets everything below be tested now: +plakar at /backup/plakar-test create +``` + +## 3. Database dumps — `incus exec`, no DB users needed + +The plakar mysql integration was considered and rejected: it connects +over TCP, which would have required a bind-address change plus a +dedicated backup user in every DB container. Instead the driver +script keeps the old `incus-backup.sh` method — `incus exec -- +mariadb-dump --single-transaction ...` (container-local root over the +unix socket) — with one upgrade: **databases are auto-discovered**, +so there is no manifest to go stale (the failure mode that bit +`incus-backup.db`). Three probes per *running* container: + +- native MariaDB/MySQL (either binary naming — `mail` has a + hand-installed `mariadb-dump` + mysql-named client; every + non-system DB) → `/backup/plakar-dumps/mariadb//.sql`, + plus a replayable users/grants dump (`SHOW GRANTS` per user, incl. + password hashes) → `grants.sql` — so the S3 dump path is + self-sufficient without the replicas, mirroring pg's `globals.sql` +- native PostgreSQL (`pg_dump` per DB + `pg_dumpall --globals-only` + for roles, peer auth as `postgres`) → + `/backup/plakar-dumps/postgres//` +- PostgreSQL **inside docker** (covers `outline`/`login`): docker + containers whose image name contains "postgres", dump user from + `$POSTGRES_USER` → `/backup/plakar-dumps/postgres///` + (a pg bundled in an image *not* named `*postgres*` would be missed — + check the nightly log's `dump` lines after adding such a service) + +Stopped containers (e.g. `databap`) are skipped — +their static state is covered by the leg-1 replicas. + +Opt-out: `/root/scripts/plakar-db-exclude` (optional) lists dump +paths to skip, one per line, exactly as the `dump` log lines print +them (`/` or `//`). Every skip is logged +(`SKIP `), so unlike the old manifest the list can't rot +silently. Applied 2026-08-22: `livetrail/geo` (10 GiB static map +data, 3 h dump). Its last dump deliberately stays in the dump dir: +unchanged → skipped by the VFS cache, so every nightly `dumps` +snapshot keeps carrying it to S3 at zero cost. To refresh after a +re-import: drop the exclude line for one night. + +Dumps are staged **plain, not gzipped**: plakar chunks and compresses +anyway, and CDC dedup across near-identical consecutive dumps only +works on uncompressed input. The dump tree (incus's own DBs + +MariaDB) is backed up as one snapshot tagged `dumps`. The nightly log +lists every `dump /` line — a database disappearing from the +log is visible. (The installed mysql plugin is now unused; harmless +to keep.) + +Beyond databases, `outline`/`login` may hold non-DB state (uploaded +files, docker volumes) — decide per instance whether an fs source is +needed in §4. + +Findings from the first dry-run (2026-08-22): + +- `livetrail`'s `/root/.my.cnf` had a stale `database = spot` default + that broke discovery — fixed 2026-08-22 (its DB is `livetrail`). +- `login` still runs a **native** postgres holding the pre-migration + `outline` DB (live Outline uses `outline-postgres-1`). Dumped + harmlessly for now; cleanup once the S3 leg is verified: + `dropdb outline` + remove the native postgres package. + +## 4. Filesystem sources + +Derived from the old `incus-backup.db` manifest (minus `spot`; DBs +are handled by §3's auto-discovery). FS paths are read from the live +containers' rootfs on the host — crash-consistent, same fidelity as +the old rsync (the per-file +[incus integration](plakar-incus-integration.md) supersedes this +later). Verify the pool mount prefix first: +`ls /var/lib/incus/storage-pools/data/containers/mail/rootfs/`. + +`outline`/`login` non-DB state identified 2026-08-22 via +`docker inspect` (Outline's local file storage volume; authentik's +data + custom templates — `/opt/authentik` also catches the compose +file). ⚠️ Still to review: whether `livetrail` holds state. + +Register sources (root on ks4): + +```sh +R=/var/lib/incus/storage-pools/data/containers +plakar source add bitwarden-fs $R/bitwarden/rootfs/opt/bitwarden +plakar source add gateway-fs $R/gateway/rootfs/var/www +plakar source add git-fs $R/git/rootfs/home/git/projects +plakar source add login-fs $R/login/rootfs/opt/authentik +plakar source add mail-vmail $R/mail/rootfs/var/vmail +plakar source add mail-www $R/mail/rootfs/var/www +plakar source add nextcloud-fs $R/nextcloud/rootfs/nextcloud +plakar source add outline-fs $R/outline/rootfs/var/lib/docker/volumes/outline_storage-data/_data +plakar source add seafile-fs $R/seafile/rootfs/opt/seafile +plakar source add solar-fs $R/solar/rootfs/var/www/html/solar +``` + +Then list the names (one per line) in `/root/scripts/plakar-sources` — +the script iterates that file. + +## 5. Script + cron + +```sh +# copy scripts/plakar-backup.sh from this repo to /root/scripts/, then: +chmod +x /root/scripts/plakar-backup.sh +/root/scripts/plakar-backup.sh -k /backup/plakar-test # dry-run the pipeline now +/root/scripts/plakar-backup.sh -k @s3 # first real run (once creds exist) +plakar at @s3 ls # inspect snapshots +``` + +Cron (04:30, after the 04:00 stopgap so they don't hammer the same +DBs simultaneously; move to 04:00 once the rsync stopgap is dropped). +Inline `backup -check` was removed 2026-08-22: against S3 it re-reads +every written chunk (~2.6 MB/s latency-bound; gateway-fs: 7 min +backup, 23 min check — the giants would add 10–20 h nightly). +Verification is a weekly deep check of the latest snapshots instead: + +```cron +30 4 * * * /root/scripts/plakar-backup.sh -k @s3 >> /var/log/plakar-backup.log 2>&1 +0 12 * * 0 PLAKAR_PASSPHRASE=$(cat /root/.plakar-passphrase) plakar -quiet at @s3 check -latest >> /var/log/plakar-check.log 2>&1 +``` + +Once validated against @s3, delete the throwaway kloset: +`rm -rf /backup/plakar-test`. + +## 6. S3 hardening + +Scoped credentials (no delete permission if the provider allows it), +bucket versioning or object lock, and prefer an OVH region — ks4 → +OVH Object Storage stays on the OVH backbone, so backup and restore +speed is LAN-like. + +## 7. Restore test (release gate) + +Two restores from `@s3`, exercising both data types: + +```sh +export PLAKAR_PASSPHRASE=$(cat /root/.plakar-passphrase) +plakar at @s3 ls # expect 11 snapshots: dumps + 10 fs sources + +# (a) file tree — restore solar-fs and diff against the live tree +plakar at @s3 restore -to /tmp/restore-test +diff -r /tmp/restore-test/var/lib/incus/storage-pools/data/containers/solar/rootfs/var/www/html/solar \ + /var/lib/incus/storage-pools/data/containers/solar/rootfs/var/www/html/solar +rm -rf /tmp/restore-test + +# (b) database — restore a dump and load it into a scratch container +plakar at @s3 restore -to /tmp/restore-db +incus launch images:ubuntu/24.04 dbtest +incus exec dbtest -- apt-get install -y mariadb-server +incus file push /tmp/restore-db/backup/plakar-dumps/mariadb/freshrss/freshrss.sql dbtest/root/ +incus exec dbtest -- sh -c 'mariadb < /root/freshrss.sql' # dump embeds CREATE DATABASE +incus exec dbtest -- mariadb -e 'SELECT COUNT(*) FROM information_schema.tables WHERE table_schema="freshrss"' +incus delete -f dbtest && rm -rf /tmp/restore-db +``` + +Pass = diff empty (modulo files changed since the snapshot) and the +scratch DB loads with a sane table count. + +## Retirement + +Once this leg is verified (and once more from @s3): remove the 04:00 +`incus-backup.sh` cron to ks2 — precondition for +[decommissioning ks2](../ks2/plan.md).