Document the ks2 decommission plan and the new ks4 backup legs
- ks2/: decommission plan with release gates (rental ends Sep 30), nuc-seed and final-decommission runbooks - ks4/local-backup-cron.md: leg-1 cron re-enabled, ks2 rsync stopgap, full-resend caveat when refreshes lapse past snapshots.expiry - ks4/plakar-s3-data.md: plakar direct-to-S3 leg (auto-discovered DB dumps incl. grants/globals, fs sources, exclude list, restore test) - ks4/plakar-incus-integration.md: design + scaffold status of the plakar incus importer - README.md: index ks2/, update the ks4 durability bullet Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Julien Lutran
co-authored by
Claude Fable 5
Claude Fable 5
parent
4b30b43c7f
commit
79ad1b7179
@@ -0,0 +1,221 @@
|
||||
# plakar data + DB backups on ks4 (leg 3)
|
||||
|
||||
Status: **pipeline validated 2026-08-22** against a local test kloset
|
||||
(`/backup/plakar-test`): full dump auto-discovery incl. grants/globals,
|
||||
exclude mechanism, fs sources, prune + maintenance, rc=0; second run
|
||||
confirmed incremental behavior (21 GiB dumps snapshot → 60 MiB
|
||||
written). **S3 credentials ready — next: store add, create, seed.**
|
||||
Replaces `incus-backup.sh` (the 04:00 rsync to ks2 stays alive as a
|
||||
stopgap until this leg is verified — see
|
||||
[local-backup-cron.md](local-backup-cron.md)).
|
||||
|
||||
Architecture: plakar runs **on ks4** (decision recorded in
|
||||
[plan.md](../ks2/plan.md)) and backs up **directly into an S3 kloset**
|
||||
(`plakar at @s3 backup @src`). A local staging kloset on sdb5 was
|
||||
considered and rejected: it would double-store ~1.5 T that the leg-1
|
||||
replicas already hold on the same disk, and local restore is already
|
||||
leg 1's job. Direct-to-S3 keeps the incremental behavior — the
|
||||
skip-unchanged baseline is the parent snapshot's VFS read from the
|
||||
kloset (plakar caches repository metadata locally; v1.1.4's dirpack
|
||||
prefetcher targets exactly this remote-backend walk). Driver script:
|
||||
[`scripts/plakar-backup.sh`](../../scripts/plakar-backup.sh)
|
||||
(flock, incus DB dumps, iterates a sources file, prune + maintenance;
|
||||
its `-k` takes any `plakar at` target, so `-k @s3` — a throwaway
|
||||
local kloset, e.g. `-k /backup/plakar-test`, works for testing before
|
||||
credentials arrive and can be deleted after).
|
||||
|
||||
## 1. Install plakar via the official APT repo (root on ks4)
|
||||
|
||||
Repo verified reachable 2026-08-22 (suite `stable`, amd64/arm64);
|
||||
upgrades then ride the normal `apt upgrade` flow.
|
||||
|
||||
```sh
|
||||
curl -fsSL https://plakar.io/dist/keys/community-v1.1.0.gpg \
|
||||
| gpg --dearmor -o /usr/share/keyrings/plakar.gpg
|
||||
echo "deb [signed-by=/usr/share/keyrings/plakar.gpg] https://plakar.io/dist/repos/deb/ stable main" \
|
||||
> /etc/apt/sources.list.d/plakar.list
|
||||
apt update && apt install plakar
|
||||
plakar version # v1.1.4 at time of writing
|
||||
```
|
||||
|
||||
The mysql integration is a plugin; official plugins install from the
|
||||
plugin server after `plakar login`, or build locally (needs make + Go):
|
||||
|
||||
```sh
|
||||
plakar pkg add mysql # or: clone integrations, cd mysql && make install
|
||||
```
|
||||
|
||||
## 2. Kloset + passphrase
|
||||
|
||||
```sh
|
||||
head -c 32 /dev/urandom | base64 > /root/.plakar-passphrase
|
||||
chmod 600 /root/.plakar-passphrase
|
||||
# ⚠️ copy the passphrase to the password manager NOW — without it
|
||||
# the S3 kloset is unreadable garbage.
|
||||
export PLAKAR_PASSPHRASE=$(cat /root/.plakar-passphrase)
|
||||
|
||||
# when S3 credentials are available:
|
||||
plakar store add s3 s3://<endpoint>/<bucket> access_key=<AK> secret_access_key=<SK> use_tls=true
|
||||
plakar at @s3 create
|
||||
# meanwhile, a throwaway kloset lets everything below be tested now:
|
||||
plakar at /backup/plakar-test create
|
||||
```
|
||||
|
||||
## 3. Database dumps — `incus exec`, no DB users needed
|
||||
|
||||
The plakar mysql integration was considered and rejected: it connects
|
||||
over TCP, which would have required a bind-address change plus a
|
||||
dedicated backup user in every DB container. Instead the driver
|
||||
script keeps the old `incus-backup.sh` method — `incus exec <ct> --
|
||||
mariadb-dump --single-transaction ...` (container-local root over the
|
||||
unix socket) — with one upgrade: **databases are auto-discovered**,
|
||||
so there is no manifest to go stale (the failure mode that bit
|
||||
`incus-backup.db`). Three probes per *running* container:
|
||||
|
||||
- native MariaDB/MySQL (either binary naming — `mail` has a
|
||||
hand-installed `mariadb-dump` + mysql-named client; every
|
||||
non-system DB) → `/backup/plakar-dumps/mariadb/<ct>/<db>.sql`,
|
||||
plus a replayable users/grants dump (`SHOW GRANTS` per user, incl.
|
||||
password hashes) → `grants.sql` — so the S3 dump path is
|
||||
self-sufficient without the replicas, mirroring pg's `globals.sql`
|
||||
- native PostgreSQL (`pg_dump` per DB + `pg_dumpall --globals-only`
|
||||
for roles, peer auth as `postgres`) →
|
||||
`/backup/plakar-dumps/postgres/<ct>/`
|
||||
- PostgreSQL **inside docker** (covers `outline`/`login`): docker
|
||||
containers whose image name contains "postgres", dump user from
|
||||
`$POSTGRES_USER` → `/backup/plakar-dumps/postgres/<ct>/<docker-name>/`
|
||||
(a pg bundled in an image *not* named `*postgres*` would be missed —
|
||||
check the nightly log's `dump` lines after adding such a service)
|
||||
|
||||
Stopped containers (e.g. `databap`) are skipped —
|
||||
their static state is covered by the leg-1 replicas.
|
||||
|
||||
Opt-out: `/root/scripts/plakar-db-exclude` (optional) lists dump
|
||||
paths to skip, one per line, exactly as the `dump` log lines print
|
||||
them (`<ct>/<db>` or `<ct>/<docker-name>/<db>`). Every skip is logged
|
||||
(`SKIP <path>`), so unlike the old manifest the list can't rot
|
||||
silently. Applied 2026-08-22: `livetrail/geo` (10 GiB static map
|
||||
data, 3 h dump). Its last dump deliberately stays in the dump dir:
|
||||
unchanged → skipped by the VFS cache, so every nightly `dumps`
|
||||
snapshot keeps carrying it to S3 at zero cost. To refresh after a
|
||||
re-import: drop the exclude line for one night.
|
||||
|
||||
Dumps are staged **plain, not gzipped**: plakar chunks and compresses
|
||||
anyway, and CDC dedup across near-identical consecutive dumps only
|
||||
works on uncompressed input. The dump tree (incus's own DBs +
|
||||
MariaDB) is backed up as one snapshot tagged `dumps`. The nightly log
|
||||
lists every `dump <ct>/<db>` line — a database disappearing from the
|
||||
log is visible. (The installed mysql plugin is now unused; harmless
|
||||
to keep.)
|
||||
|
||||
Beyond databases, `outline`/`login` may hold non-DB state (uploaded
|
||||
files, docker volumes) — decide per instance whether an fs source is
|
||||
needed in §4.
|
||||
|
||||
Findings from the first dry-run (2026-08-22):
|
||||
|
||||
- `livetrail`'s `/root/.my.cnf` had a stale `database = spot` default
|
||||
that broke discovery — fixed 2026-08-22 (its DB is `livetrail`).
|
||||
- `login` still runs a **native** postgres holding the pre-migration
|
||||
`outline` DB (live Outline uses `outline-postgres-1`). Dumped
|
||||
harmlessly for now; cleanup once the S3 leg is verified:
|
||||
`dropdb outline` + remove the native postgres package.
|
||||
|
||||
## 4. Filesystem sources
|
||||
|
||||
Derived from the old `incus-backup.db` manifest (minus `spot`; DBs
|
||||
are handled by §3's auto-discovery). FS paths are read from the live
|
||||
containers' rootfs on the host — crash-consistent, same fidelity as
|
||||
the old rsync (the per-file
|
||||
[incus integration](plakar-incus-integration.md) supersedes this
|
||||
later). Verify the pool mount prefix first:
|
||||
`ls /var/lib/incus/storage-pools/data/containers/mail/rootfs/`.
|
||||
|
||||
`outline`/`login` non-DB state identified 2026-08-22 via
|
||||
`docker inspect` (Outline's local file storage volume; authentik's
|
||||
data + custom templates — `/opt/authentik` also catches the compose
|
||||
file). ⚠️ Still to review: whether `livetrail` holds state.
|
||||
|
||||
Register sources (root on ks4):
|
||||
|
||||
```sh
|
||||
R=/var/lib/incus/storage-pools/data/containers
|
||||
plakar source add bitwarden-fs $R/bitwarden/rootfs/opt/bitwarden
|
||||
plakar source add gateway-fs $R/gateway/rootfs/var/www
|
||||
plakar source add git-fs $R/git/rootfs/home/git/projects
|
||||
plakar source add login-fs $R/login/rootfs/opt/authentik
|
||||
plakar source add mail-vmail $R/mail/rootfs/var/vmail
|
||||
plakar source add mail-www $R/mail/rootfs/var/www
|
||||
plakar source add nextcloud-fs $R/nextcloud/rootfs/nextcloud
|
||||
plakar source add outline-fs $R/outline/rootfs/var/lib/docker/volumes/outline_storage-data/_data
|
||||
plakar source add seafile-fs $R/seafile/rootfs/opt/seafile
|
||||
plakar source add solar-fs $R/solar/rootfs/var/www/html/solar
|
||||
```
|
||||
|
||||
Then list the names (one per line) in `/root/scripts/plakar-sources` —
|
||||
the script iterates that file.
|
||||
|
||||
## 5. Script + cron
|
||||
|
||||
```sh
|
||||
# copy scripts/plakar-backup.sh from this repo to /root/scripts/, then:
|
||||
chmod +x /root/scripts/plakar-backup.sh
|
||||
/root/scripts/plakar-backup.sh -k /backup/plakar-test # dry-run the pipeline now
|
||||
/root/scripts/plakar-backup.sh -k @s3 # first real run (once creds exist)
|
||||
plakar at @s3 ls # inspect snapshots
|
||||
```
|
||||
|
||||
Cron (04:30, after the 04:00 stopgap so they don't hammer the same
|
||||
DBs simultaneously; move to 04:00 once the rsync stopgap is dropped).
|
||||
Inline `backup -check` was removed 2026-08-22: against S3 it re-reads
|
||||
every written chunk (~2.6 MB/s latency-bound; gateway-fs: 7 min
|
||||
backup, 23 min check — the giants would add 10–20 h nightly).
|
||||
Verification is a weekly deep check of the latest snapshots instead:
|
||||
|
||||
```cron
|
||||
30 4 * * * /root/scripts/plakar-backup.sh -k @s3 >> /var/log/plakar-backup.log 2>&1
|
||||
0 12 * * 0 PLAKAR_PASSPHRASE=$(cat /root/.plakar-passphrase) plakar -quiet at @s3 check -latest >> /var/log/plakar-check.log 2>&1
|
||||
```
|
||||
|
||||
Once validated against @s3, delete the throwaway kloset:
|
||||
`rm -rf /backup/plakar-test`.
|
||||
|
||||
## 6. S3 hardening
|
||||
|
||||
Scoped credentials (no delete permission if the provider allows it),
|
||||
bucket versioning or object lock, and prefer an OVH region — ks4 →
|
||||
OVH Object Storage stays on the OVH backbone, so backup and restore
|
||||
speed is LAN-like.
|
||||
|
||||
## 7. Restore test (release gate)
|
||||
|
||||
Two restores from `@s3`, exercising both data types:
|
||||
|
||||
```sh
|
||||
export PLAKAR_PASSPHRASE=$(cat /root/.plakar-passphrase)
|
||||
plakar at @s3 ls # expect 11 snapshots: dumps + 10 fs sources
|
||||
|
||||
# (a) file tree — restore solar-fs and diff against the live tree
|
||||
plakar at @s3 restore -to /tmp/restore-test <solar-fs-snap-id>
|
||||
diff -r /tmp/restore-test/var/lib/incus/storage-pools/data/containers/solar/rootfs/var/www/html/solar \
|
||||
/var/lib/incus/storage-pools/data/containers/solar/rootfs/var/www/html/solar
|
||||
rm -rf /tmp/restore-test
|
||||
|
||||
# (b) database — restore a dump and load it into a scratch container
|
||||
plakar at @s3 restore -to /tmp/restore-db <dumps-snap-id>
|
||||
incus launch images:ubuntu/24.04 dbtest
|
||||
incus exec dbtest -- apt-get install -y mariadb-server
|
||||
incus file push /tmp/restore-db/backup/plakar-dumps/mariadb/freshrss/freshrss.sql dbtest/root/
|
||||
incus exec dbtest -- sh -c 'mariadb < /root/freshrss.sql' # dump embeds CREATE DATABASE
|
||||
incus exec dbtest -- mariadb -e 'SELECT COUNT(*) FROM information_schema.tables WHERE table_schema="freshrss"'
|
||||
incus delete -f dbtest && rm -rf /tmp/restore-db
|
||||
```
|
||||
|
||||
Pass = diff empty (modulo files changed since the snapshot) and the
|
||||
scratch DB loads with a sane table count.
|
||||
|
||||
## Retirement
|
||||
|
||||
Once this leg is verified (and once more from @s3): remove the 04:00
|
||||
`incus-backup.sh` cron to ks2 — precondition for
|
||||
[decommissioning ks2](../ks2/plan.md).
|
||||
Reference in New Issue
Block a user