doc/ks4/plakar-s3-data.md

# plakar data + DB backups on ks4 (leg 3)

Status: **pipeline validated 2026-08-22** against a local test kloset
(`/backup/plakar-test`): full dump auto-discovery incl. grants/globals,
exclude mechanism, fs sources, prune + maintenance, rc=0; second run
confirmed incremental behavior (21 GiB dumps snapshot → 60 MiB
written). **S3 credentials ready — next: store add, create, seed.**
Replaces `incus-backup.sh` (the 04:00 rsync to ks2 stays alive as a
stopgap until this leg is verified — see
[local-backup-cron.md](local-backup-cron.md)).

Architecture: plakar runs **on ks4** (decision recorded in
[plan.md](../ks2/plan.md)) and backs up **directly into an S3 kloset**
(`plakar at @s3 backup @src`). A local staging kloset on sdb5 was
considered and rejected: it would double-store ~1.5 T that the leg-1
replicas already hold on the same disk, and local restore is already
leg 1's job. Direct-to-S3 keeps the incremental behavior — the
skip-unchanged baseline is the parent snapshot's VFS read from the
kloset (plakar caches repository metadata locally; v1.1.4's dirpack
prefetcher targets exactly this remote-backend walk). Driver script:
[`scripts/plakar-backup.sh`](../../scripts/plakar-backup.sh)
(flock, incus DB dumps, iterates a sources file, prune + maintenance;
its `-k` takes any `plakar at` target, so `-k @s3` — a throwaway
local kloset, e.g. `-k /backup/plakar-test`, works for testing before
credentials arrive and can be deleted after).

## 1. Install plakar via the official APT repo (root on ks4)

Repo verified reachable 2026-08-22 (suite `stable`, amd64/arm64);
upgrades then ride the normal `apt upgrade` flow.

```sh
curl -fsSL https://plakar.io/dist/keys/community-v1.1.0.gpg \
    | gpg --dearmor -o /usr/share/keyrings/plakar.gpg
echo "deb [signed-by=/usr/share/keyrings/plakar.gpg] https://plakar.io/dist/repos/deb/ stable main" \
    > /etc/apt/sources.list.d/plakar.list
apt update && apt install plakar
plakar version           # v1.1.4 at time of writing
```

The mysql integration is a plugin; official plugins install from the
plugin server after `plakar login`, or build locally (needs make + Go):

```sh
plakar pkg add mysql        # or: clone integrations, cd mysql && make install
```

## 2. Kloset + passphrase

```sh
head -c 32 /dev/urandom | base64 > /root/.plakar-passphrase
chmod 600 /root/.plakar-passphrase
# ⚠️ copy the passphrase to the password manager NOW — without it
# the S3 kloset is unreadable garbage.
export PLAKAR_PASSPHRASE=$(cat /root/.plakar-passphrase)

# when S3 credentials are available:
plakar store add s3 s3://<endpoint>/<bucket> access_key=<AK> secret_access_key=<SK> use_tls=true
plakar at @s3 create
# meanwhile, a throwaway kloset lets everything below be tested now:
plakar at /backup/plakar-test create
```

## 3. Database dumps — `incus exec`, no DB users needed

The plakar mysql integration was considered and rejected: it connects
over TCP, which would have required a bind-address change plus a
dedicated backup user in every DB container. Instead the driver
script keeps the old `incus-backup.sh` method — `incus exec <ct> --
mariadb-dump --single-transaction ...` (container-local root over the
unix socket) — with one upgrade: **databases are auto-discovered**,
so there is no manifest to go stale (the failure mode that bit
`incus-backup.db`). Three probes per *running* container:

- native MariaDB/MySQL (either binary naming — `mail` has a
  hand-installed `mariadb-dump` + mysql-named client; every
  non-system DB) → `/backup/plakar-dumps/mariadb/<ct>/<db>.sql`,
  plus a replayable users/grants dump (`SHOW GRANTS` per user, incl.
  password hashes) → `grants.sql` — so the S3 dump path is
  self-sufficient without the replicas, mirroring pg's `globals.sql`
- native PostgreSQL (`pg_dump` per DB + `pg_dumpall --globals-only`
  for roles, peer auth as `postgres`) →
  `/backup/plakar-dumps/postgres/<ct>/`
- PostgreSQL **inside docker** (covers `outline`/`login`): docker
  containers whose image name contains "postgres", dump user from
  `$POSTGRES_USER` → `/backup/plakar-dumps/postgres/<ct>/<docker-name>/`
  (a pg bundled in an image *not* named `*postgres*` would be missed —
  check the nightly log's `dump` lines after adding such a service)

Stopped containers (e.g. `databap`) are skipped —
their static state is covered by the leg-1 replicas.

Opt-out: `/root/scripts/plakar-db-exclude` (optional) lists dump
paths to skip, one per line, exactly as the `dump` log lines print
them (`<ct>/<db>` or `<ct>/<docker-name>/<db>`). Every skip is logged
(`SKIP <path>`), so unlike the old manifest the list can't rot
silently. Applied 2026-08-22: `livetrail/geo` (10 GiB static map
data, 3 h dump). Its last dump deliberately stays in the dump dir:
unchanged → skipped by the VFS cache, so every nightly `dumps`
snapshot keeps carrying it to S3 at zero cost. To refresh after a
re-import: drop the exclude line for one night.

Dumps are staged **plain, not gzipped**: plakar chunks and compresses
anyway, and CDC dedup across near-identical consecutive dumps only
works on uncompressed input. The dump tree (incus's own DBs +
MariaDB) is backed up as one snapshot tagged `dumps`. The nightly log
lists every `dump <ct>/<db>` line — a database disappearing from the
log is visible. (The installed mysql plugin is now unused; harmless
to keep.)

Beyond databases, `outline`/`login` may hold non-DB state (uploaded
files, docker volumes) — decide per instance whether an fs source is
needed in §4.

⚠️ **seafile-fs excluded from the data leg (2026-08-24).** The S3
seed wedged for 12 h inside a single `lstat()` in seafile's block
store (millions of tiny files on the contended live pool). SIGQUIT
goroutine dump: main goroutine looping in `os.ignoringEINTR` around
`fstatat` — a Go preemption/EINTR **livelock** (100 % CPU, no
progress; dmesg clean, so not a kernel hang). `seafile-fs` is
commented out of `plakar-sources`; seafile's S3 coverage comes from
the incus leg instead, which reads the quiesced replica on the idle
`backup` pool. If a live-pool retry is ever wanted:
`GODEBUG=asyncpreemptoff=1` is the standard mitigation.

Findings from the first dry-run (2026-08-22):

- `livetrail`'s `/root/.my.cnf` had a stale `database = spot` default
  that broke discovery — fixed 2026-08-22 (its DB is `livetrail`).
- `login` still runs a **native** postgres holding the pre-migration
  `outline` DB (live Outline uses `outline-postgres-1`). Dumped
  harmlessly for now; cleanup once the S3 leg is verified:
  `dropdb outline` + remove the native postgres package.

## 4. Filesystem sources

Derived from the old `incus-backup.db` manifest (minus `spot`; DBs
are handled by §3's auto-discovery). FS paths are read from the live
containers' rootfs on the host — crash-consistent, same fidelity as
the old rsync (the per-file
[incus integration](plakar-incus-integration.md) supersedes this
later). Verify the pool mount prefix first:
`ls /var/lib/incus/storage-pools/data/containers/mail/rootfs/`.

`outline`/`login` non-DB state identified 2026-08-22 via
`docker inspect` (Outline's local file storage volume; authentik's
data + custom templates — `/opt/authentik` also catches the compose
file). ⚠️ Still to review: whether `livetrail` holds state.

Register sources (root on ks4):

```sh
R=/var/lib/incus/storage-pools/data/containers
plakar source add bitwarden-fs  $R/bitwarden/rootfs/opt/bitwarden
plakar source add gateway-fs    $R/gateway/rootfs/var/www
plakar source add git-fs        $R/git/rootfs/home/git/projects
plakar source add login-fs      $R/login/rootfs/opt/authentik
plakar source add mail-vmail    $R/mail/rootfs/var/vmail
plakar source add mail-www      $R/mail/rootfs/var/www
plakar source add nextcloud-fs  $R/nextcloud/rootfs/nextcloud
plakar source add outline-fs    $R/outline/rootfs/var/lib/docker/volumes/outline_storage-data/_data
plakar source add seafile-fs    $R/seafile/rootfs/opt/seafile
plakar source add solar-fs      $R/solar/rootfs/var/www/html/solar
```

Then list the names (one per line) in `/root/scripts/plakar-sources` —
the script iterates that file.

## 5. Script + cron

```sh
# copy scripts/plakar-backup.sh from this repo to /root/scripts/, then:
chmod +x /root/scripts/plakar-backup.sh
/root/scripts/plakar-backup.sh -k /backup/plakar-test   # dry-run the pipeline now
/root/scripts/plakar-backup.sh -k @s3                   # first real run (once creds exist)
plakar at @s3 ls                                        # inspect snapshots
```

Cron (04:30, after the 04:00 stopgap so they don't hammer the same
DBs simultaneously; move to 04:00 once the rsync stopgap is dropped).
Inline `backup -check` was removed 2026-08-22: against S3 it re-reads
every written chunk (~2.6 MB/s latency-bound; gateway-fs: 7 min
backup, 23 min check — the giants would add 10–20 h nightly).
Verification is a weekly deep check of the latest snapshots instead:

```cron
30 4 * * * /root/scripts/plakar-backup.sh -k @s3 >> /var/log/plakar-backup.log 2>&1
0 12 * * 0 PLAKAR_PASSPHRASE=$(cat /root/.plakar-passphrase) plakar -quiet at @s3 check -latest >> /var/log/plakar-check.log 2>&1
```

Once validated against @s3, delete the throwaway kloset:
`rm -rf /backup/plakar-test`.

## 6. S3 hardening

Scoped credentials (no delete permission if the provider allows it),
bucket versioning or object lock, and prefer an OVH region — ks4 →
OVH Object Storage stays on the OVH backbone, so backup and restore
speed is LAN-like.

## 7. Restore test (release gate)

Two restores from `@s3`, exercising both data types:

```sh
export PLAKAR_PASSPHRASE=$(cat /root/.plakar-passphrase)
plakar at @s3 ls          # expect 11 snapshots: dumps + 10 fs sources

# (a) file tree — restore solar-fs and diff against the live tree
plakar at @s3 restore -to /tmp/restore-test <solar-fs-snap-id>
diff -r /tmp/restore-test/var/lib/incus/storage-pools/data/containers/solar/rootfs/var/www/html/solar \
        /var/lib/incus/storage-pools/data/containers/solar/rootfs/var/www/html/solar
rm -rf /tmp/restore-test

# (b) database — restore a dump and load it into a scratch container
plakar at @s3 restore -to /tmp/restore-db <dumps-snap-id>
incus launch images:ubuntu/24.04 dbtest
incus exec dbtest -- apt-get install -y mariadb-server
incus file push /tmp/restore-db/backup/plakar-dumps/mariadb/freshrss/freshrss.sql dbtest/root/
incus exec dbtest -- sh -c 'mariadb < /root/freshrss.sql'   # dump embeds CREATE DATABASE
incus exec dbtest -- mariadb -e 'SELECT COUNT(*) FROM information_schema.tables WHERE table_schema="freshrss"'
incus delete -f dbtest && rm -rf /tmp/restore-db
```

Pass = diff empty (modulo files changed since the snapshot) and the
scratch DB loads with a sane table count.

## Retirement

Once this leg is verified (and once more from @s3): remove the 04:00
`incus-backup.sh` cron to ks2 — precondition for
[decommissioning ks2](../ks2/plan.md).