julien/doc
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Document the ks2 decommission plan and the new ks4 backup legs

Browse Source 
- ks2/: decommission plan with release gates (rental ends Sep 30),
  nuc-seed and final-decommission runbooks
- ks4/local-backup-cron.md: leg-1 cron re-enabled, ks2 rsync stopgap,
  full-resend caveat when refreshes lapse past snapshots.expiry
- ks4/plakar-s3-data.md: plakar direct-to-S3 leg (auto-discovered DB
  dumps incl. grants/globals, fs sources, exclude list, restore test)
- ks4/plakar-incus-integration.md: design + scaffold status of the
  plakar incus importer
- README.md: index ks2/, update the ks4 durability bullet

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Julien Lutran
2026-08-23 08:56:28 +02:00
co-authored by Claude Fable 5
parent 4b30b43c7f
commit 79ad1b7179
7 changed files with 597 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files
ks4/local-backup-cron.md
+56
View File
@@ -0,0 +1,56 @@
# local backup cron on ks4 (leg 1) — re-enabled
Done 2026-08-22. Context: **both** ks4 backup crons had been commented
out — no backup of any kind was running.
## What runs now (root crontab on ks4)
```cron
0 1 * * * /root/scripts/incus-copy.sh -p backup -s backup >> /var/log/incus-copy.log 2>&1
0 4 * * * /root/scripts/incus-backup.sh -d 164.132.173.57 -u rsyncbackup -i /root/.ssh/id_rsyncbackup -p 2233 -f /root/scripts/incus-backup.db -s data >> /var/log/incus-backup.log 2>&1
```
- `01:00` — **local leg**: `incus copy --refresh` of all instances
into project `backup` on the sdb5 `backup` zpool (replaces the old
push to ks2; see [incus-copy.md](incus-copy.md)).
- `04:00` — **stopgap**: `incus-backup.sh` rsync to ks2 kept alive
(manifest pruned of `spot`) until the plakar S3 leg replaces it —
then this line goes away as part of
[decommissioning ks2](../ks2/plan.md). ⚠️ keep
`scripts/incus-backup.db` in this repo in sync with
`/root/scripts/incus-backup.db` on ks4.
Both were also run once by hand (in a `screen`) on 2026-08-22 to
refresh the backups immediately. The old push-to-ks2 copy line stays
commented out — replica refreshes to ks2 are retired.
## Cleanup done the same day
- ks2: `incus delete biwiki spot` (abandoned instances — freed space
on the 95 %-full `data` pool)
- ks4: `incus delete spot --project backup`
## Verification
```bash
# all instances present, incl. livetrail (first copy is a full send)
sudo incus list --project backup -c ns -f csv
# cron runs land here (manual screen runs don't):
sudo tail /var/log/incus-copy.log /var/log/incus-backup.log
# rsync leg freshness, on ks2:
ls -lat /backup/ns3061243/ | head
```
Notes:
- `flock` in both scripts makes overlapping cron/manual runs abort
safely instead of racing.
- Replicas get `boot.autostart=false` set by the script — they must
never fight the live instances for their static IPs.
- Keep `snapshots.schedule` set on the sources so refreshes stay
ZFS-incremental (a refresh without common snapshots falls back to
rsync). Corollary: the cron must run at least every
`snapshots.expiry` (7 d) — a longer gap expires every common
snapshot and the next refresh silently degrades to a **full
re-send** (observed 2026-08-22: seafile re-sent 933 G after the
weeks-long gap while the crons were commented out).
ks4/plakar-incus-integration.md
+119
View File
@@ -0,0 +1,119 @@
# plakar incus integration — design notes
Status: **importer scaffolded and compiling 2026-08-23** in
`perso/plakar/integration-incus` (walker modeled on the official sftp
integration; enumerate project → per-instance sftp walk → per-file
records + synthetic `.incus.yaml`; allow/deny lists; VM sftp failures
reported as record errors, not fatal). See its README for
build/install/TODO. Untested against a live incus yet.
Goal: plakar source connector for Incus so instance backups land in a
plakar kloset on S3 (leg 4 of the ks4 3-2-1 plan — see
[plan.md](../ks2/plan.md)).
## Decision 1 — per-file importer, not tarball streaming
The proxmox integration streams one full vzdump archive per VM
(`FLAG_STREAM`). For ks4 that model reads ~1.7 TB nightly (seafile
933 G, nextcloud 725 G): the incus backup/export API produces **full
tarballs only** — `shared/api/instance_backup.go` has just
`instance_only` and `optimized_storage`, no incremental option.
Instead the importer emits **one record per file** with real stat info
and a lazy reader. kloset's engine then does the incremental work for
us: `snapshot/backup.go` (`checkVFSCache`) reuses the cached object
when a path's stat matches the previous run and **never opens the
file**. Nightly cost after the initial seed = stat-walk + changed
files only; S3 receives only new chunks. Bonus: per-file browse and
restore of any snapshot.
File access is native incus REST: `GET /1.0/instances/{name}/sftp`
(a real SFTP session, also available per storage volume). Works over
the local unix socket or remote :8443, on stopped containers (the
daemon mounts the volume on demand), no ZFS assumptions.
## Decision 2 — do not reuse `incus copy --refresh` / migration API
Considered and rejected. `copy --refresh` is `POST /1.0/instances`
with `source.refresh=true` (`shared/api/instance.go:373`) and is
**not** zfs-only — transports are negotiated per storage driver
(`MigrationTypes()`: zfs/btrfs/ceph native streams, universal RSYNC
fallback, `driver_common.go:230`). But:
- the receiver must be another incus daemon speaking the migration
websocket protocol (`internal/migration/migrate.proto`);
- the optimized payload is an opaque `zfs send` stream — storing
those in plakar means chain-of-increments restore onto a real
zpool, no per-file browse, no pruning, one corrupt link breaks the
chain;
- the RSYNC transport needs a materialized previous copy on the
receiver — a full-size local mirror, i.e. leg 1 rebuilt inside a
plugin;
- gotcha: `driver_zfs.go:801` — refresh **without** snapshot copying
forces RSYNC even zfs↔zfs (keep `snapshots.schedule` on sources).
`copy --refresh` stays what legs 1–2 use (host↔host replication);
plakar is content-addressed archiving to dumb storage. Complementary,
not competing.
## Decision 3 — read the `backup` project replicas, not live instances
Reading a live rootfs is fuzzy. Leg 1 (`incus-copy.sh -p backup -s
backup`, 01:00) already produces quiescent, crash-consistent stopped
replicas — the importer targets those (`?project=backup`), scheduled
after the copy completes. DB consistency remains the mariadb-dump
job's responsibility (plakar `mysql` importer / dump step).
## Decision 4 — enumerate everything by default
`incus-backup.db` manifest drift (stale `spot`, missing `livetrail`,
`outline`, `login`…) is the failure mode to kill: the importer backs
up **all instances in the project by default**, opt-out via config,
never opt-in.
## Other design points
- Location scheme `incus://<remote>/<project>/<instance>/...` —
stable pathnames keep the kloset VFS cache effective.
- Synthetic records per instance: config + profiles
(`GET /1.0/instances/{name}`, expanded) so restore can recreate the
instance before pushing files back.
- Optional later `mode=image`: full tarball via the backups API for
occasional exact-image restores (`incus import`); plakar dedups
unchanged chunks between runs. Not the nightly path.
- Restore path (exporter): create instance from stored config → push
file tree back via sftp. Weaker than `incus import` but granular.
## Packaging & deployment
Plugins are standalone executables (gRPC over stdio) — no rebuild of
plakar, fully compatible with the APT-installed binary on ks4:
```sh
cd plakar/integration-incus && make # build importer/exporter
plakar pkg create manifest.yaml v0.1.0 # → incus_v0.1.0_linux_amd64.ptar
# on ks4, as root (plugins are per-user, cron runs as root):
plakar pkg add ./incus_v0.1.0_linux_amd64.ptar
plakar source add ks4-incus incus://... # then list it in /root/scripts/plakar-sources
```
Installing a *local* `.ptar` file needs no `plakar login` (the plugin
registry is only for fetching by name). Model the `manifest.yaml` on
integration-proxmox (`tier: third-party`, `api_version: v1.1.0`).
Compatibility contract = the go-kloset-sdk wire protocol: pin the SDK
to the installed plakar line (v1.1.x), rebuild the ptar on a plakar
major upgrade; a mismatch surfaces at backup time, so run a manual
`plakar backup @ks4-incus` after plakar upgrades.
## Open questions
- uid/gid view through the instance sftp endpoint (idmap handling for
unprivileged containers) — verify what stat returns and what restore
must remap.
- xattrs / ACLs / device nodes over sftp — coverage and fidelity.
- VMs: file access needs a running incus-agent — out of scope for ks4
(containers only); document the limitation.
- Scheduling/locking: chain after the 01:00 incus-copy (flock on the
same lock, or a wrapper script).
- S3 hardening: scoped credentials (no delete), versioning/object
lock, plakar client-side encryption passphrase storage.
ks4/plakar-s3-data.md
+221
View File
@@ -0,0 +1,221 @@
# plakar data + DB backups on ks4 (leg 3)
Status: **pipeline validated 2026-08-22** against a local test kloset
(`/backup/plakar-test`): full dump auto-discovery incl. grants/globals,
exclude mechanism, fs sources, prune + maintenance, rc=0; second run
confirmed incremental behavior (21 GiB dumps snapshot → 60 MiB
written). **S3 credentials ready — next: store add, create, seed.**
Replaces `incus-backup.sh` (the 04:00 rsync to ks2 stays alive as a
stopgap until this leg is verified — see
[local-backup-cron.md](local-backup-cron.md)).
Architecture: plakar runs **on ks4** (decision recorded in
[plan.md](../ks2/plan.md)) and backs up **directly into an S3 kloset**
(`plakar at @s3 backup @src`). A local staging kloset on sdb5 was
considered and rejected: it would double-store ~1.5 T that the leg-1
replicas already hold on the same disk, and local restore is already
leg 1's job. Direct-to-S3 keeps the incremental behavior — the
skip-unchanged baseline is the parent snapshot's VFS read from the
kloset (plakar caches repository metadata locally; v1.1.4's dirpack
prefetcher targets exactly this remote-backend walk). Driver script:
[`scripts/plakar-backup.sh`](../../scripts/plakar-backup.sh)
(flock, incus DB dumps, iterates a sources file, prune + maintenance;
its `-k` takes any `plakar at` target, so `-k @s3` — a throwaway
local kloset, e.g. `-k /backup/plakar-test`, works for testing before
credentials arrive and can be deleted after).
## 1. Install plakar via the official APT repo (root on ks4)
Repo verified reachable 2026-08-22 (suite `stable`, amd64/arm64);
upgrades then ride the normal `apt upgrade` flow.
```sh
curl -fsSL https://plakar.io/dist/keys/community-v1.1.0.gpg \
| gpg --dearmor -o /usr/share/keyrings/plakar.gpg
echo "deb [signed-by=/usr/share/keyrings/plakar.gpg] https://plakar.io/dist/repos/deb/ stable main" \
> /etc/apt/sources.list.d/plakar.list
apt update && apt install plakar
plakar version # v1.1.4 at time of writing
```
The mysql integration is a plugin; official plugins install from the
plugin server after `plakar login`, or build locally (needs make + Go):
```sh
plakar pkg add mysql # or: clone integrations, cd mysql && make install
```
## 2. Kloset + passphrase
```sh
head -c 32 /dev/urandom | base64 > /root/.plakar-passphrase
chmod 600 /root/.plakar-passphrase
# ⚠️ copy the passphrase to the password manager NOW — without it
# the S3 kloset is unreadable garbage.
export PLAKAR_PASSPHRASE=$(cat /root/.plakar-passphrase)
# when S3 credentials are available:
plakar store add s3 s3://<endpoint>/<bucket> access_key=<AK> secret_access_key=<SK> use_tls=true
plakar at @s3 create
# meanwhile, a throwaway kloset lets everything below be tested now:
plakar at /backup/plakar-test create
```
## 3. Database dumps — `incus exec`, no DB users needed
The plakar mysql integration was considered and rejected: it connects
over TCP, which would have required a bind-address change plus a
dedicated backup user in every DB container. Instead the driver
script keeps the old `incus-backup.sh` method — `incus exec <ct> --
mariadb-dump --single-transaction ...` (container-local root over the
unix socket) — with one upgrade: **databases are auto-discovered**,
so there is no manifest to go stale (the failure mode that bit
`incus-backup.db`). Three probes per *running* container:
- native MariaDB/MySQL (either binary naming — `mail` has a
hand-installed `mariadb-dump` + mysql-named client; every
non-system DB) → `/backup/plakar-dumps/mariadb/<ct>/<db>.sql`,
plus a replayable users/grants dump (`SHOW GRANTS` per user, incl.
password hashes) → `grants.sql` — so the S3 dump path is
self-sufficient without the replicas, mirroring pg's `globals.sql`
- native PostgreSQL (`pg_dump` per DB + `pg_dumpall --globals-only`
for roles, peer auth as `postgres`) →
`/backup/plakar-dumps/postgres/<ct>/`
- PostgreSQL **inside docker** (covers `outline`/`login`): docker
containers whose image name contains "postgres", dump user from
`$POSTGRES_USER` → `/backup/plakar-dumps/postgres/<ct>/<docker-name>/`
(a pg bundled in an image *not* named `*postgres*` would be missed —
check the nightly log's `dump` lines after adding such a service)
Stopped containers (e.g. `databap`) are skipped —
their static state is covered by the leg-1 replicas.
Opt-out: `/root/scripts/plakar-db-exclude` (optional) lists dump
paths to skip, one per line, exactly as the `dump` log lines print
them (`<ct>/<db>` or `<ct>/<docker-name>/<db>`). Every skip is logged
(`SKIP <path>`), so unlike the old manifest the list can't rot
silently. Applied 2026-08-22: `livetrail/geo` (10 GiB static map
data, 3 h dump). Its last dump deliberately stays in the dump dir:
unchanged → skipped by the VFS cache, so every nightly `dumps`
snapshot keeps carrying it to S3 at zero cost. To refresh after a
re-import: drop the exclude line for one night.
Dumps are staged **plain, not gzipped**: plakar chunks and compresses
anyway, and CDC dedup across near-identical consecutive dumps only
works on uncompressed input. The dump tree (incus's own DBs +
MariaDB) is backed up as one snapshot tagged `dumps`. The nightly log
lists every `dump <ct>/<db>` line — a database disappearing from the
log is visible. (The installed mysql plugin is now unused; harmless
to keep.)
Beyond databases, `outline`/`login` may hold non-DB state (uploaded
files, docker volumes) — decide per instance whether an fs source is
needed in §4.
Findings from the first dry-run (2026-08-22):
- `livetrail`'s `/root/.my.cnf` had a stale `database = spot` default
that broke discovery — fixed 2026-08-22 (its DB is `livetrail`).
- `login` still runs a **native** postgres holding the pre-migration
`outline` DB (live Outline uses `outline-postgres-1`). Dumped
harmlessly for now; cleanup once the S3 leg is verified:
`dropdb outline` + remove the native postgres package.
## 4. Filesystem sources
Derived from the old `incus-backup.db` manifest (minus `spot`; DBs
are handled by §3's auto-discovery). FS paths are read from the live
containers' rootfs on the host — crash-consistent, same fidelity as
the old rsync (the per-file
[incus integration](plakar-incus-integration.md) supersedes this
later). Verify the pool mount prefix first:
`ls /var/lib/incus/storage-pools/data/containers/mail/rootfs/`.
`outline`/`login` non-DB state identified 2026-08-22 via
`docker inspect` (Outline's local file storage volume; authentik's
data + custom templates — `/opt/authentik` also catches the compose
file). ⚠️ Still to review: whether `livetrail` holds state.
Register sources (root on ks4):
```sh
R=/var/lib/incus/storage-pools/data/containers
plakar source add bitwarden-fs $R/bitwarden/rootfs/opt/bitwarden
plakar source add gateway-fs $R/gateway/rootfs/var/www
plakar source add git-fs $R/git/rootfs/home/git/projects
plakar source add login-fs $R/login/rootfs/opt/authentik
plakar source add mail-vmail $R/mail/rootfs/var/vmail
plakar source add mail-www $R/mail/rootfs/var/www
plakar source add nextcloud-fs $R/nextcloud/rootfs/nextcloud
plakar source add outline-fs $R/outline/rootfs/var/lib/docker/volumes/outline_storage-data/_data
plakar source add seafile-fs $R/seafile/rootfs/opt/seafile
plakar source add solar-fs $R/solar/rootfs/var/www/html/solar
```
Then list the names (one per line) in `/root/scripts/plakar-sources` —
the script iterates that file.
## 5. Script + cron
```sh
# copy scripts/plakar-backup.sh from this repo to /root/scripts/, then:
chmod +x /root/scripts/plakar-backup.sh
/root/scripts/plakar-backup.sh -k /backup/plakar-test # dry-run the pipeline now
/root/scripts/plakar-backup.sh -k @s3 # first real run (once creds exist)
plakar at @s3 ls # inspect snapshots
```
Cron (04:30, after the 04:00 stopgap so they don't hammer the same
DBs simultaneously; move to 04:00 once the rsync stopgap is dropped).
Inline `backup -check` was removed 2026-08-22: against S3 it re-reads
every written chunk (~2.6 MB/s latency-bound; gateway-fs: 7 min
backup, 23 min check — the giants would add 10–20 h nightly).
Verification is a weekly deep check of the latest snapshots instead:
```cron
30 4 * * * /root/scripts/plakar-backup.sh -k @s3 >> /var/log/plakar-backup.log 2>&1
0 12 * * 0 PLAKAR_PASSPHRASE=$(cat /root/.plakar-passphrase) plakar -quiet at @s3 check -latest >> /var/log/plakar-check.log 2>&1
```
Once validated against @s3, delete the throwaway kloset:
`rm -rf /backup/plakar-test`.
## 6. S3 hardening
Scoped credentials (no delete permission if the provider allows it),
bucket versioning or object lock, and prefer an OVH region — ks4 →
OVH Object Storage stays on the OVH backbone, so backup and restore
speed is LAN-like.
## 7. Restore test (release gate)
Two restores from `@s3`, exercising both data types:
```sh
export PLAKAR_PASSPHRASE=$(cat /root/.plakar-passphrase)
plakar at @s3 ls # expect 11 snapshots: dumps + 10 fs sources
# (a) file tree — restore solar-fs and diff against the live tree
plakar at @s3 restore -to /tmp/restore-test <solar-fs-snap-id>
diff -r /tmp/restore-test/var/lib/incus/storage-pools/data/containers/solar/rootfs/var/www/html/solar \
/var/lib/incus/storage-pools/data/containers/solar/rootfs/var/www/html/solar
rm -rf /tmp/restore-test
# (b) database — restore a dump and load it into a scratch container
plakar at @s3 restore -to /tmp/restore-db <dumps-snap-id>
incus launch images:ubuntu/24.04 dbtest
incus exec dbtest -- apt-get install -y mariadb-server
incus file push /tmp/restore-db/backup/plakar-dumps/mariadb/freshrss/freshrss.sql dbtest/root/
incus exec dbtest -- sh -c 'mariadb < /root/freshrss.sql' # dump embeds CREATE DATABASE
incus exec dbtest -- mariadb -e 'SELECT COUNT(*) FROM information_schema.tables WHERE table_schema="freshrss"'
incus delete -f dbtest && rm -rf /tmp/restore-db
```
Pass = diff empty (modulo files changed since the snapshot) and the
scratch DB loads with a sane table count.
## Retirement
Once this leg is verified (and once more from @s3): remove the 04:00
`incus-backup.sh` cron to ks2 — precondition for
[decommissioning ks2](../ks2/plan.md).