# plakar data + DB backups on ks4 (leg 3) Status: **pipeline validated 2026-08-22** against a local test kloset (`/backup/plakar-test`): full dump auto-discovery incl. grants/globals, exclude mechanism, fs sources, prune + maintenance, rc=0; second run confirmed incremental behavior (21 GiB dumps snapshot → 60 MiB written). **S3 credentials ready — next: store add, create, seed.** Replaces `incus-backup.sh` (the 04:00 rsync to ks2 stays alive as a stopgap until this leg is verified — see [local-backup-cron.md](local-backup-cron.md)). Architecture: plakar runs **on ks4** (decision recorded in [plan.md](../ks2/plan.md)) and backs up **directly into an S3 kloset** (`plakar at @s3 backup @src`). A local staging kloset on sdb5 was considered and rejected: it would double-store ~1.5 T that the leg-1 replicas already hold on the same disk, and local restore is already leg 1's job. Direct-to-S3 keeps the incremental behavior — the skip-unchanged baseline is the parent snapshot's VFS read from the kloset (plakar caches repository metadata locally; v1.1.4's dirpack prefetcher targets exactly this remote-backend walk). Driver script: [`scripts/plakar-backup.sh`](../../scripts/plakar-backup.sh) (flock, incus DB dumps, iterates a sources file, prune + maintenance; its `-k` takes any `plakar at` target, so `-k @s3` — a throwaway local kloset, e.g. `-k /backup/plakar-test`, works for testing before credentials arrive and can be deleted after). ## 1. Install plakar via the official APT repo (root on ks4) Repo verified reachable 2026-08-22 (suite `stable`, amd64/arm64); upgrades then ride the normal `apt upgrade` flow. ```sh curl -fsSL https://plakar.io/dist/keys/community-v1.1.0.gpg \ | gpg --dearmor -o /usr/share/keyrings/plakar.gpg echo "deb [signed-by=/usr/share/keyrings/plakar.gpg] https://plakar.io/dist/repos/deb/ stable main" \ > /etc/apt/sources.list.d/plakar.list apt update && apt install plakar plakar version # v1.1.4 at time of writing ``` The mysql integration is a plugin; official plugins install from the plugin server after `plakar login`, or build locally (needs make + Go): ```sh plakar pkg add mysql # or: clone integrations, cd mysql && make install ``` ## 2. Kloset + passphrase ```sh head -c 32 /dev/urandom | base64 > /root/.plakar-passphrase chmod 600 /root/.plakar-passphrase # ⚠️ copy the passphrase to the password manager NOW — without it # the S3 kloset is unreadable garbage. export PLAKAR_PASSPHRASE=$(cat /root/.plakar-passphrase) # when S3 credentials are available: plakar store add s3 s3:/// access_key= secret_access_key= use_tls=true plakar at @s3 create # meanwhile, a throwaway kloset lets everything below be tested now: plakar at /backup/plakar-test create ``` ## 3. Database dumps — `incus exec`, no DB users needed The plakar mysql integration was considered and rejected: it connects over TCP, which would have required a bind-address change plus a dedicated backup user in every DB container. Instead the driver script keeps the old `incus-backup.sh` method — `incus exec -- mariadb-dump --single-transaction ...` (container-local root over the unix socket) — with one upgrade: **databases are auto-discovered**, so there is no manifest to go stale (the failure mode that bit `incus-backup.db`). Three probes per *running* container: - native MariaDB/MySQL (either binary naming — `mail` has a hand-installed `mariadb-dump` + mysql-named client; every non-system DB) → `/backup/plakar-dumps/mariadb//.sql`, plus a replayable users/grants dump (`SHOW GRANTS` per user, incl. password hashes) → `grants.sql` — so the S3 dump path is self-sufficient without the replicas, mirroring pg's `globals.sql` - native PostgreSQL (`pg_dump` per DB + `pg_dumpall --globals-only` for roles, peer auth as `postgres`) → `/backup/plakar-dumps/postgres//` - PostgreSQL **inside docker** (covers `outline`/`login`): docker containers whose image name contains "postgres", dump user from `$POSTGRES_USER` → `/backup/plakar-dumps/postgres///` (a pg bundled in an image *not* named `*postgres*` would be missed — check the nightly log's `dump` lines after adding such a service) Stopped containers (e.g. `databap`) are skipped — their static state is covered by the leg-1 replicas. Opt-out: `/root/scripts/plakar-db-exclude` (optional) lists dump paths to skip, one per line, exactly as the `dump` log lines print them (`/` or `//`). Every skip is logged (`SKIP `), so unlike the old manifest the list can't rot silently. Applied 2026-08-22: `livetrail/geo` (10 GiB static map data, 3 h dump). Its last dump deliberately stays in the dump dir: unchanged → skipped by the VFS cache, so every nightly `dumps` snapshot keeps carrying it to S3 at zero cost. To refresh after a re-import: drop the exclude line for one night. Dumps are staged **plain, not gzipped**: plakar chunks and compresses anyway, and CDC dedup across near-identical consecutive dumps only works on uncompressed input. The dump tree (incus's own DBs + MariaDB) is backed up as one snapshot tagged `dumps`. The nightly log lists every `dump /` line — a database disappearing from the log is visible. (The installed mysql plugin is now unused; harmless to keep.) Beyond databases, `outline`/`login` may hold non-DB state (uploaded files, docker volumes) — decide per instance whether an fs source is needed in §4. Findings from the first dry-run (2026-08-22): - `livetrail`'s `/root/.my.cnf` had a stale `database = spot` default that broke discovery — fixed 2026-08-22 (its DB is `livetrail`). - `login` still runs a **native** postgres holding the pre-migration `outline` DB (live Outline uses `outline-postgres-1`). Dumped harmlessly for now; cleanup once the S3 leg is verified: `dropdb outline` + remove the native postgres package. ## 4. Filesystem sources Derived from the old `incus-backup.db` manifest (minus `spot`; DBs are handled by §3's auto-discovery). FS paths are read from the live containers' rootfs on the host — crash-consistent, same fidelity as the old rsync (the per-file [incus integration](plakar-incus-integration.md) supersedes this later). Verify the pool mount prefix first: `ls /var/lib/incus/storage-pools/data/containers/mail/rootfs/`. `outline`/`login` non-DB state identified 2026-08-22 via `docker inspect` (Outline's local file storage volume; authentik's data + custom templates — `/opt/authentik` also catches the compose file). ⚠️ Still to review: whether `livetrail` holds state. Register sources (root on ks4): ```sh R=/var/lib/incus/storage-pools/data/containers plakar source add bitwarden-fs $R/bitwarden/rootfs/opt/bitwarden plakar source add gateway-fs $R/gateway/rootfs/var/www plakar source add git-fs $R/git/rootfs/home/git/projects plakar source add login-fs $R/login/rootfs/opt/authentik plakar source add mail-vmail $R/mail/rootfs/var/vmail plakar source add mail-www $R/mail/rootfs/var/www plakar source add nextcloud-fs $R/nextcloud/rootfs/nextcloud plakar source add outline-fs $R/outline/rootfs/var/lib/docker/volumes/outline_storage-data/_data plakar source add seafile-fs $R/seafile/rootfs/opt/seafile plakar source add solar-fs $R/solar/rootfs/var/www/html/solar ``` Then list the names (one per line) in `/root/scripts/plakar-sources` — the script iterates that file. ## 5. Script + cron ```sh # copy scripts/plakar-backup.sh from this repo to /root/scripts/, then: chmod +x /root/scripts/plakar-backup.sh /root/scripts/plakar-backup.sh -k /backup/plakar-test # dry-run the pipeline now /root/scripts/plakar-backup.sh -k @s3 # first real run (once creds exist) plakar at @s3 ls # inspect snapshots ``` Cron (04:30, after the 04:00 stopgap so they don't hammer the same DBs simultaneously; move to 04:00 once the rsync stopgap is dropped). Inline `backup -check` was removed 2026-08-22: against S3 it re-reads every written chunk (~2.6 MB/s latency-bound; gateway-fs: 7 min backup, 23 min check — the giants would add 10–20 h nightly). Verification is a weekly deep check of the latest snapshots instead: ```cron 30 4 * * * /root/scripts/plakar-backup.sh -k @s3 >> /var/log/plakar-backup.log 2>&1 0 12 * * 0 PLAKAR_PASSPHRASE=$(cat /root/.plakar-passphrase) plakar -quiet at @s3 check -latest >> /var/log/plakar-check.log 2>&1 ``` Once validated against @s3, delete the throwaway kloset: `rm -rf /backup/plakar-test`. ## 6. S3 hardening Scoped credentials (no delete permission if the provider allows it), bucket versioning or object lock, and prefer an OVH region — ks4 → OVH Object Storage stays on the OVH backbone, so backup and restore speed is LAN-like. ## 7. Restore test (release gate) Two restores from `@s3`, exercising both data types: ```sh export PLAKAR_PASSPHRASE=$(cat /root/.plakar-passphrase) plakar at @s3 ls # expect 11 snapshots: dumps + 10 fs sources # (a) file tree — restore solar-fs and diff against the live tree plakar at @s3 restore -to /tmp/restore-test diff -r /tmp/restore-test/var/lib/incus/storage-pools/data/containers/solar/rootfs/var/www/html/solar \ /var/lib/incus/storage-pools/data/containers/solar/rootfs/var/www/html/solar rm -rf /tmp/restore-test # (b) database — restore a dump and load it into a scratch container plakar at @s3 restore -to /tmp/restore-db incus launch images:ubuntu/24.04 dbtest incus exec dbtest -- apt-get install -y mariadb-server incus file push /tmp/restore-db/backup/plakar-dumps/mariadb/freshrss/freshrss.sql dbtest/root/ incus exec dbtest -- sh -c 'mariadb < /root/freshrss.sql' # dump embeds CREATE DATABASE incus exec dbtest -- mariadb -e 'SELECT COUNT(*) FROM information_schema.tables WHERE table_schema="freshrss"' incus delete -f dbtest && rm -rf /tmp/restore-db ``` Pass = diff empty (modulo files changed since the snapshot) and the scratch DB loads with a sane table count. ## Retirement Once this leg is verified (and once more from @s3): remove the 04:00 `incus-backup.sh` cron to ks2 — precondition for [decommissioning ks2](../ks2/plan.md).