doc/ks4/plakar-incus-integration.md

# plakar incus integration — design notes (RETIRED)

> **Historical.** The importer works (smoke-tested against ks4's
> `backup` project) but production moved to restic, which needs no
> plugin: `incus file mount` exposes any instance — including stopped
> replicas — over the same per-instance sftp API
> ([restic-backup.md](restic-backup.md) §6). Code lives on at
> <https://github.com/jlutran/plakar-integration-incus>; the analysis
> below (per-file vs image, replica-vs-live consistency, the
> seek-bound wall) is what shaped the restic design.


Status: **importer scaffolded and compiling 2026-08-23** in
`perso/plakar/integration-incus` (walker modeled on the official sftp
integration; enumerate project → per-instance sftp walk → per-file
records + synthetic `.incus.yaml`; allow/deny lists; VM sftp failures
reported as record errors, not fatal). See its README for
build/install/TODO. Untested against a live incus yet.

Goal: plakar source connector for Incus so instance backups land in a
plakar kloset on S3 (leg 4 of the ks4 3-2-1 plan — see
[plan.md](../ks2/plan.md)).

## Decision 1 — per-file importer, not tarball streaming

The proxmox integration streams one full vzdump archive per VM
(`FLAG_STREAM`). For ks4 that model reads ~1.7 TB nightly (seafile
933 G, nextcloud 725 G): the incus backup/export API produces **full
tarballs only** — `shared/api/instance_backup.go` has just
`instance_only` and `optimized_storage`, no incremental option.

Instead the importer emits **one record per file** with real stat info
and a lazy reader. kloset's engine then does the incremental work for
us: `snapshot/backup.go` (`checkVFSCache`) reuses the cached object
when a path's stat matches the previous run and **never opens the
file**. Nightly cost after the initial seed = stat-walk + changed
files only; S3 receives only new chunks. Bonus: per-file browse and
restore of any snapshot.

File access is native incus REST: `GET /1.0/instances/{name}/sftp`
(a real SFTP session, also available per storage volume). Works over
the local unix socket or remote :8443, on stopped containers (the
daemon mounts the volume on demand), no ZFS assumptions.

## Decision 2 — do not reuse `incus copy --refresh` / migration API

Considered and rejected. `copy --refresh` is `POST /1.0/instances`
with `source.refresh=true` (`shared/api/instance.go:373`) and is
**not** zfs-only — transports are negotiated per storage driver
(`MigrationTypes()`: zfs/btrfs/ceph native streams, universal RSYNC
fallback, `driver_common.go:230`). But:

- the receiver must be another incus daemon speaking the migration
  websocket protocol (`internal/migration/migrate.proto`);
- the optimized payload is an opaque `zfs send` stream — storing
  those in plakar means chain-of-increments restore onto a real
  zpool, no per-file browse, no pruning, one corrupt link breaks the
  chain;
- the RSYNC transport needs a materialized previous copy on the
  receiver — a full-size local mirror, i.e. leg 1 rebuilt inside a
  plugin;
- gotcha: `driver_zfs.go:801` — refresh **without** snapshot copying
  forces RSYNC even zfs↔zfs (keep `snapshots.schedule` on sources).

`copy --refresh` stays what legs 1–2 use (host↔host replication);
plakar is content-addressed archiving to dumb storage. Complementary,
not competing.

## Decision 3 — read the `backup` project replicas, not live instances

Reading a live rootfs is fuzzy. Leg 1 (`incus-copy.sh -p backup -s
backup`, 01:00) already produces quiescent, crash-consistent stopped
replicas — the importer targets those (`?project=backup`), scheduled
after the copy completes. DB consistency remains the mariadb-dump
job's responsibility (plakar `mysql` importer / dump step).

## Decision 4 — enumerate everything by default

`incus-backup.db` manifest drift (stale `spot`, missing `livetrail`,
`outline`, `login`…) is the failure mode to kill: the importer backs
up **all instances in the project by default**, opt-out via config,
never opt-in.

## Other design points

- Location scheme `incus://<remote>/<project>/<instance>/...` —
  stable pathnames keep the kloset VFS cache effective.
- Synthetic records per instance: config + profiles
  (`GET /1.0/instances/{name}`, expanded) so restore can recreate the
  instance before pushing files back.
- Optional later `mode=image`: full tarball via the backups API for
  occasional exact-image restores (`incus import`); plakar dedups
  unchanged chunks between runs. Not the nightly path.
- Restore path (exporter): create instance from stored config → push
  file tree back via sftp. Weaker than `incus import` but granular.

## Upstreaming (decided 2026-08-23: Route B — community integration)

Repo: `github.com/jlutran/plakar-integration-incus` (remote set,
initial commit done). Path to publication, in order:

1. unit tests + first real run against ks4's `backup` project
2. open a PlakarKorp/plakar issue proposing the integration
   (disclose the Apache-2.0 `lxc/incus` client dependency)
3. push the repo, then PR a recipe against `PlakarKorp/hub` →
   installable as `plakar pkg add incus`
4. (optional, later) adoption into the `integrations` monorepo:
   module rename to `github.com/PlakarKorp/integrations/incus`,
   orphan `integration/incus` branch, `incus/incus` layout

## Packaging & deployment

Plugins are standalone executables (gRPC over stdio) — no rebuild of
plakar, fully compatible with the APT-installed binary on ks4:

```sh
cd plakar/integration-incus && make          # build importer/exporter
plakar pkg create manifest.yaml v0.1.0       # → incus_v0.1.0_linux_amd64.ptar
# on ks4, as root (plugins are per-user, cron runs as root):
plakar pkg add ./incus_v0.1.0_linux_amd64.ptar
plakar source add ks4-incus incus://...      # then list it in /root/scripts/plakar-sources
```

Installing a *local* `.ptar` file needs no `plakar login` (the plugin
registry is only for fetching by name). Model the `manifest.yaml` on
integration-proxmox (`tier: third-party`, `api_version: v1.1.0`).
Compatibility contract = the go-kloset-sdk wire protocol: pin the SDK
to the installed plakar line (v1.1.x), rebuild the ptar on a plakar
major upgrade; a mismatch surfaces at backup time, so run a manual
`plakar backup @ks4-incus` after plakar upgrades.

## The giant-instance problem (2026-08-24, seed attempt #1)

The per-file S3 seed of the whole `backup` project stalled at
~1 MiB/s once it reached the big replicas: sdb at 96 % util doing
~275 × 4 KiB reads/s — **cold random tiny-file reads on an HDD are
seek-bound**, regardless of importer (the fs importer hit the same
wall on the live pool; sftp changes nothing). At ~275 IOPS,
nextcloud (725 G, ~500 k files) + seafile (933 G, millions of block
files) need *weeks* to seed. The 16 small instances are unaffected
(solar smoke test: 5 GiB in 2m38s).

Options considered for nextcloud/seafile:

1. **backups-API tarball (`mode=image`)** — sequential-ish but incus
   materializes the tarball server-side first (no room on `/`; would
   need `storage.backups_volume` on the backup pool) and the tar walk
   itself seeks like any tree walk. Double IO nightly. Weak.
2. **`zfs send` of the replica dataset** — the only truly sequential
   read (disk block order, ~100+ MiB/s, no temp space): seed in
   hours. Full send nightly is ~4 h read for both giants; CDC chunking
   dedups unchanged stream regions. Restore = `zfs receive` +
   `incus admin recover` (document!). Backend-specific — acceptable as
   an opt-in mode for exactly these two.
3. **Per-file + warm metadata** — after a seed, nightly walks are
   stat-only; keeping dnodes resident (`zfs_arc_min` ≈ 8 G) could make
   them fast. Doesn't solve the *seed*.
4. **Exclude giants from this leg** — leaves seafile without any
   fresh off-site copy (data leg already excludes it). Not acceptable
   long-term.

Direction (pending decision): seed the 16 small instances per-file
(`exclude_instances=nextcloud,seafile`), then implement option 2 as a
`zfs+send://` mode or side-channel for the two giants, and revisit 3
for nightly stat-walk speed.

## Open questions

- uid/gid view through the instance sftp endpoint (idmap handling for
  unprivileged containers) — verify what stat returns and what restore
  must remap.
- xattrs / ACLs / device nodes over sftp — coverage and fidelity.
- VMs: file access needs a running incus-agent — out of scope for ks4
  (containers only); document the limitation.
- ~~Scheduling/locking~~ decided 2026-08-23: the S3 instance leg is
  only as fresh as the last **completed** local incus-copy, so both
  run chained in one cron entry —
  `incus-copy.sh -p backup -s backup ; <plakar incus leg>` — never at
  a fixed offset (a long refresh, e.g. a post-gap full re-send, would
  race it). `;` not `&&`: a partially failed copy leaves stale but
  internally consistent replicas, still worth uploading. Same rule
  for manual test runs: check `/var/log/incus-copy.log` completed
  first.
- S3 hardening: scoped credentials (no delete), versioning/object
  lock, plakar client-side encryption passphrase storage.