doc/ks4/plakar-s3-data.md

plakar data + DB backups on ks4 (leg 3) — SUPERSEDED

Historical. This leg ran 2026-08-22 → 08-28 and was replaced by restic: see restic-backup.md. plakar's incremental cost scaled with tree size rather than churn (PlakarKorp/plakar#2338). The plakar-data kloset and the plakar install are deliberately kept until that issue concludes (the report offers debug runs against that repository); the plakar-incus bucket can be deleted. Kept here for the measurements and the design rationale, which the restic doc builds on.

Status: live since 2026-08-24 — S3 kloset (lutran-ks4-plakar-data) seeded (dumps + 9 fs sources), restore test passed (file tree + 10 GiB dumps snapshot), 04:30 nightly cron + Sunday check -latest installed. Replaces incus-backup.sh (its 04:00 ks2 rsync cron is disabled). Open items: seafile-fs excluded (see §3 warning), nextcloud-fs costs 3–4 h/night (cost scales with tree size, not churn — reported upstream 2026-08-25: PlakarKorp/plakar#2338; giants to move to a restic leg meanwhile).

Architecture: plakar runs on ks4 (decision recorded in plan.md) and backs up directly into an S3 kloset (plakar at @s3 backup @src). A local staging kloset on sdb5 was considered and rejected: it would double-store ~1.5 T that the leg-1 replicas already hold on the same disk, and local restore is already leg 1's job. Direct-to-S3 keeps the incremental behavior — the skip-unchanged baseline is the parent snapshot's VFS read from the kloset (plakar caches repository metadata locally; v1.1.4's dirpack prefetcher targets exactly this remote-backend walk). Driver script: scripts/plakar-backup.sh (flock, incus DB dumps, iterates a sources file, prune + maintenance; its -k takes any plakar at target, so -k @s3 — a throwaway local kloset, e.g. -k /backup/plakar-test, works for testing before credentials arrive and can be deleted after).

1. Install plakar via the official APT repo (root on ks4)

Repo verified reachable 2026-08-22 (suite stable, amd64/arm64); upgrades then ride the normal apt upgrade flow.

curl -fsSL https://plakar.io/dist/keys/community-v1.1.0.gpg \
    | gpg --dearmor -o /usr/share/keyrings/plakar.gpg
echo "deb [signed-by=/usr/share/keyrings/plakar.gpg] https://plakar.io/dist/repos/deb/ stable main" \
    > /etc/apt/sources.list.d/plakar.list
apt update && apt install plakar
plakar version           # v1.1.4 at time of writing

The mysql integration is a plugin; official plugins install from the plugin server after plakar login, or build locally (needs make + Go):

plakar pkg add mysql        # or: clone integrations, cd mysql && make install

2. Kloset + passphrase

head -c 32 /dev/urandom | base64 > /root/.plakar-passphrase
chmod 600 /root/.plakar-passphrase
# ⚠️ copy the passphrase to the password manager NOW — without it
# the S3 kloset is unreadable garbage.
export PLAKAR_PASSPHRASE=$(cat /root/.plakar-passphrase)

# when S3 credentials are available:
plakar store add s3 s3://<endpoint>/<bucket> access_key=<AK> secret_access_key=<SK> use_tls=true
plakar at @s3 create
# meanwhile, a throwaway kloset lets everything below be tested now:
plakar at /backup/plakar-test create

3. Database dumps — incus exec, no DB users needed

The plakar mysql integration was considered and rejected: it connects over TCP, which would have required a bind-address change plus a dedicated backup user in every DB container. Instead the driver script keeps the old incus-backup.sh method — incus exec <ct> -- mariadb-dump --single-transaction ... (container-local root over the unix socket) — with one upgrade: databases are auto-discovered, so there is no manifest to go stale (the failure mode that bit incus-backup.db). Three probes per running container:

• native MariaDB/MySQL (either binary naming — mail has a hand-installed mariadb-dump + mysql-named client; every non-system DB) → /backup/plakar-dumps/mariadb/<ct>/<db>.sql, plus a replayable users/grants dump (SHOW GRANTS per user, incl. password hashes) → grants.sql — so the S3 dump path is self-sufficient without the replicas, mirroring pg's globals.sql
• native PostgreSQL (pg_dump per DB + pg_dumpall --globals-only for roles, peer auth as postgres) → /backup/plakar-dumps/postgres/<ct>/
• PostgreSQL inside docker (covers outline/login): docker containers whose image name contains "postgres", dump user from $POSTGRES_USER → /backup/plakar-dumps/postgres/<ct>/<docker-name>/ (a pg bundled in an image not named *postgres* would be missed — check the nightly log's dump lines after adding such a service)

Stopped containers (e.g. databap) are skipped — their static state is covered by the leg-1 replicas.

Opt-out: /root/scripts/plakar-db-exclude (optional) lists dump paths to skip, one per line, exactly as the dump log lines print them (<ct>/<db> or <ct>/<docker-name>/<db>). Every skip is logged (SKIP <path>), so unlike the old manifest the list can't rot silently. Applied 2026-08-22: livetrail/geo (10 GiB static map data, 3 h dump). Its last dump deliberately stays in the dump dir: unchanged → skipped by the VFS cache, so every nightly dumps snapshot keeps carrying it to S3 at zero cost. To refresh after a re-import: drop the exclude line for one night.

Dumps are staged plain, not gzipped: plakar chunks and compresses anyway, and CDC dedup across near-identical consecutive dumps only works on uncompressed input. The dump tree (incus's own DBs + MariaDB) is backed up as one snapshot tagged dumps. The nightly log lists every dump <ct>/<db> line — a database disappearing from the log is visible. (The installed mysql plugin is now unused; harmless to keep.)

Beyond databases, outline/login may hold non-DB state (uploaded files, docker volumes) — decide per instance whether an fs source is needed in §4.

⚠️ seafile-fs excluded from the data leg (2026-08-24). Its S3 seed ran 12 h with no visible upload and was killed. Successive theories (EINTR livelock — GODEBUG=asyncpreemptoff=1 changed nothing; kernel reclaim stall — PSI counters too low; ARC-starved lstats — disproved by find walking nextcloud's whole tree in 8m20 cold / 19 s warm) all fell to evidence; cause unknown. Next diagnostic: bisect by seeding seafile-fs into a local scratch kloset (plakar at /backup/plakar-scratch create + timed backup) — completes fine → S3 store path implicated; crawls → importer-side on that tree shape (millions of tiny block files); then the maintainer-suggested -trace=all (writes to stdout). Until resolved, seafile's fresh off-site coverage waits on the incus leg reading the sdb replica.

Findings from the first dry-run (2026-08-22):

• livetrail's /root/.my.cnf had a stale database = spot default that broke discovery — fixed 2026-08-22 (its DB is livetrail).
• login still runs a native postgres holding the pre-migration outline DB (live Outline uses outline-postgres-1). Dumped harmlessly for now; cleanup once the S3 leg is verified: dropdb outline + remove the native postgres package.

4. Filesystem sources

Derived from the old incus-backup.db manifest (minus spot; DBs are handled by §3's auto-discovery). FS paths are read from the live containers' rootfs on the host — crash-consistent, same fidelity as the old rsync (the per-file incus integration supersedes this later). Verify the pool mount prefix first: ls /var/lib/incus/storage-pools/data/containers/mail/rootfs/.

outline/login non-DB state identified 2026-08-22 via docker inspect (Outline's local file storage volume; authentik's data + custom templates — /opt/authentik also catches the compose file). ⚠️ Still to review: whether livetrail holds state.

Register sources (root on ks4):

R=/var/lib/incus/storage-pools/data/containers
plakar source add bitwarden-fs  $R/bitwarden/rootfs/opt/bitwarden
plakar source add gateway-fs    $R/gateway/rootfs/var/www
plakar source add git-fs        $R/git/rootfs/home/git/projects
plakar source add login-fs      $R/login/rootfs/opt/authentik
plakar source add mail-vmail    $R/mail/rootfs/var/vmail
plakar source add mail-www      $R/mail/rootfs/var/www
plakar source add nextcloud-fs  $R/nextcloud/rootfs/nextcloud
plakar source add outline-fs    $R/outline/rootfs/var/lib/docker/volumes/outline_storage-data/_data
plakar source add seafile-fs    $R/seafile/rootfs/opt/seafile
plakar source add solar-fs      $R/solar/rootfs/var/www/html/solar

Then list the names (one per line) in /root/scripts/plakar-sources — the script iterates that file.

5. Script + cron

# copy scripts/plakar-backup.sh from this repo to /root/scripts/, then:
chmod +x /root/scripts/plakar-backup.sh
/root/scripts/plakar-backup.sh -k /backup/plakar-test   # dry-run the pipeline now
/root/scripts/plakar-backup.sh -k @s3                   # first real run (once creds exist)
plakar at @s3 ls                                        # inspect snapshots

Cron (04:30, after the 04:00 stopgap so they don't hammer the same DBs simultaneously; move to 04:00 once the rsync stopgap is dropped). Inline backup -check was removed 2026-08-22: against S3 it re-reads every written chunk (~2.6 MB/s latency-bound; gateway-fs: 7 min backup, 23 min check — the giants would add 10–20 h nightly). Verification is a weekly deep check of the latest snapshots instead:

30 4 * * * /root/scripts/plakar-backup.sh -k @s3 >> /var/log/plakar-backup.log 2>&1
0 12 * * 0 PLAKAR_PASSPHRASE=$(cat /root/.plakar-passphrase) plakar -quiet at @s3 check -latest >> /var/log/plakar-check.log 2>&1

Once validated against @s3, delete the throwaway kloset: rm -rf /backup/plakar-test.

6. S3 hardening

Scoped credentials (no delete permission if the provider allows it), bucket versioning or object lock, and prefer an OVH region — ks4 → OVH Object Storage stays on the OVH backbone, so backup and restore speed is LAN-like.

7. Restore test (release gate)

Two restores from @s3, exercising both data types:

export PLAKAR_PASSPHRASE=$(cat /root/.plakar-passphrase)
plakar at @s3 ls          # expect 11 snapshots: dumps + 10 fs sources

# (a) file tree — restore solar-fs and diff against the live tree
plakar at @s3 restore -to /backup/restore-test <solar-fs-snap-id>
diff -r /backup/restore-test/var/lib/incus/storage-pools/data/containers/solar/rootfs/var/www/html/solar \
        /var/lib/incus/storage-pools/data/containers/solar/rootfs/var/www/html/solar
rm -rf /backup/restore-test

# (b) database — restore a dump and load it into a scratch container
plakar at @s3 restore -to /backup/restore-db <dumps-snap-id>
incus launch images:ubuntu/24.04 dbtest
incus exec dbtest -- apt-get install -y mariadb-server
incus file push /backup/restore-db/backup/plakar-dumps/mariadb/freshrss/freshrss.sql dbtest/root/
incus exec dbtest -- sh -c 'mariadb < /root/freshrss.sql'   # dump embeds CREATE DATABASE
incus exec dbtest -- mariadb -e 'SELECT COUNT(*) FROM information_schema.tables WHERE table_schema="freshrss"'
incus delete -f dbtest && rm -rf /backup/restore-db

Pass = diff empty (modulo files changed since the snapshot) and the scratch DB loads with a sane table count.

Retirement

Once this leg is verified (and once more from @s3): remove the 04:00 incus-backup.sh cron to ks2 — precondition for decommissioning ks2.