doc/ks4/plakar-incus-integration.md

plakar incus integration — design notes

Status: importer scaffolded and compiling 2026-08-23 in perso/plakar/integration-incus (walker modeled on the official sftp integration; enumerate project → per-instance sftp walk → per-file records + synthetic .incus.yaml; allow/deny lists; VM sftp failures reported as record errors, not fatal). See its README for build/install/TODO. Untested against a live incus yet.

Goal: plakar source connector for Incus so instance backups land in a plakar kloset on S3 (leg 4 of the ks4 3-2-1 plan — see plan.md).

Decision 1 — per-file importer, not tarball streaming

The proxmox integration streams one full vzdump archive per VM (FLAG_STREAM). For ks4 that model reads ~1.7 TB nightly (seafile 933 G, nextcloud 725 G): the incus backup/export API produces full tarballs only — shared/api/instance_backup.go has just instance_only and optimized_storage, no incremental option.

Instead the importer emits one record per file with real stat info and a lazy reader. kloset's engine then does the incremental work for us: snapshot/backup.go (checkVFSCache) reuses the cached object when a path's stat matches the previous run and never opens the file. Nightly cost after the initial seed = stat-walk + changed files only; S3 receives only new chunks. Bonus: per-file browse and restore of any snapshot.

File access is native incus REST: GET /1.0/instances/{name}/sftp (a real SFTP session, also available per storage volume). Works over the local unix socket or remote :8443, on stopped containers (the daemon mounts the volume on demand), no ZFS assumptions.

Decision 2 — do not reuse incus copy --refresh / migration API

Considered and rejected. copy --refresh is POST /1.0/instances with source.refresh=true (shared/api/instance.go:373) and is not zfs-only — transports are negotiated per storage driver (MigrationTypes(): zfs/btrfs/ceph native streams, universal RSYNC fallback, driver_common.go:230). But:

• the receiver must be another incus daemon speaking the migration websocket protocol (internal/migration/migrate.proto);
• the optimized payload is an opaque zfs send stream — storing those in plakar means chain-of-increments restore onto a real zpool, no per-file browse, no pruning, one corrupt link breaks the chain;
• the RSYNC transport needs a materialized previous copy on the receiver — a full-size local mirror, i.e. leg 1 rebuilt inside a plugin;
• gotcha: driver_zfs.go:801 — refresh without snapshot copying forces RSYNC even zfs↔zfs (keep snapshots.schedule on sources).

copy --refresh stays what legs 1–2 use (host↔host replication); plakar is content-addressed archiving to dumb storage. Complementary, not competing.

Decision 3 — read the backup project replicas, not live instances

Reading a live rootfs is fuzzy. Leg 1 (incus-copy.sh -p backup -s backup, 01:00) already produces quiescent, crash-consistent stopped replicas — the importer targets those (?project=backup), scheduled after the copy completes. DB consistency remains the mariadb-dump job's responsibility (plakar mysql importer / dump step).

Decision 4 — enumerate everything by default

incus-backup.db manifest drift (stale spot, missing livetrail, outline, login…) is the failure mode to kill: the importer backs up all instances in the project by default, opt-out via config, never opt-in.

Other design points

• Location scheme incus://<remote>/<project>/<instance>/... — stable pathnames keep the kloset VFS cache effective.
• Synthetic records per instance: config + profiles (GET /1.0/instances/{name}, expanded) so restore can recreate the instance before pushing files back.
• Optional later mode=image: full tarball via the backups API for occasional exact-image restores (incus import); plakar dedups unchanged chunks between runs. Not the nightly path.
• Restore path (exporter): create instance from stored config → push file tree back via sftp. Weaker than incus import but granular.

Upstreaming (decided 2026-08-23: Route B — community integration)

Repo: github.com/jlutran/plakar-integration-incus (remote set, initial commit done). Path to publication, in order:

1. unit tests + first real run against ks4's backup project
2. open a PlakarKorp/plakar issue proposing the integration (disclose the Apache-2.0 lxc/incus client dependency)
3. push the repo, then PR a recipe against PlakarKorp/hub → installable as plakar pkg add incus
4. (optional, later) adoption into the integrations monorepo: module rename to github.com/PlakarKorp/integrations/incus, orphan integration/incus branch, incus/incus layout

Packaging & deployment

Plugins are standalone executables (gRPC over stdio) — no rebuild of plakar, fully compatible with the APT-installed binary on ks4:

cd plakar/integration-incus && make          # build importer/exporter
plakar pkg create manifest.yaml v0.1.0       # → incus_v0.1.0_linux_amd64.ptar
# on ks4, as root (plugins are per-user, cron runs as root):
plakar pkg add ./incus_v0.1.0_linux_amd64.ptar
plakar source add ks4-incus incus://...      # then list it in /root/scripts/plakar-sources

Installing a local .ptar file needs no plakar login (the plugin registry is only for fetching by name). Model the manifest.yaml on integration-proxmox (tier: third-party, api_version: v1.1.0). Compatibility contract = the go-kloset-sdk wire protocol: pin the SDK to the installed plakar line (v1.1.x), rebuild the ptar on a plakar major upgrade; a mismatch surfaces at backup time, so run a manual plakar backup @ks4-incus after plakar upgrades.

The giant-instance problem (2026-08-24, seed attempt #1)

The per-file S3 seed of the whole backup project stalled at ~1 MiB/s once it reached the big replicas: sdb at 96 % util doing ~275 × 4 KiB reads/s — cold random tiny-file reads on an HDD are seek-bound, regardless of importer (the fs importer hit the same wall on the live pool; sftp changes nothing). At ~275 IOPS, nextcloud (725 G, ~500 k files) + seafile (933 G, millions of block files) need weeks to seed. The 16 small instances are unaffected (solar smoke test: 5 GiB in 2m38s).

Options considered for nextcloud/seafile:

1. backups-API tarball (mode=image) — sequential-ish but incus materializes the tarball server-side first (no room on /; would need storage.backups_volume on the backup pool) and the tar walk itself seeks like any tree walk. Double IO nightly. Weak.
2. zfs send of the replica dataset — the only truly sequential read (disk block order, ~100+ MiB/s, no temp space): seed in hours. Full send nightly is ~4 h read for both giants; CDC chunking dedups unchanged stream regions. Restore = zfs receive + incus admin recover (document!). Backend-specific — acceptable as an opt-in mode for exactly these two.
3. Per-file + warm metadata — after a seed, nightly walks are stat-only; keeping dnodes resident (zfs_arc_min ≈ 8 G) could make them fast. Doesn't solve the seed.
4. Exclude giants from this leg — leaves seafile without any fresh off-site copy (data leg already excludes it). Not acceptable long-term.

Direction (pending decision): seed the 16 small instances per-file (exclude_instances=nextcloud,seafile), then implement option 2 as a zfs+send:// mode or side-channel for the two giants, and revisit 3 for nightly stat-walk speed.

Open questions

• uid/gid view through the instance sftp endpoint (idmap handling for unprivileged containers) — verify what stat returns and what restore must remap.
• xattrs / ACLs / device nodes over sftp — coverage and fidelity.
• VMs: file access needs a running incus-agent — out of scope for ks4 (containers only); document the limitation.
• Scheduling/locking decided 2026-08-23: the S3 instance leg is only as fresh as the last completed local incus-copy, so both run chained in one cron entry — incus-copy.sh -p backup -s backup ; <plakar incus leg> — never at a fixed offset (a long refresh, e.g. a post-gap full re-send, would race it). ; not &&: a partially failed copy leaves stale but internally consistent replicas, still worth uploading. Same rule for manual test runs: check /var/log/incus-copy.log completed first.
• S3 hardening: scoped credentials (no delete), versioning/object lock, plakar client-side encryption passphrase storage.