134 lines
6.2 KiB
Markdown
134 lines
6.2 KiB
Markdown
# plakar incus integration — design notes
|
||
|
||
Status: **importer scaffolded and compiling 2026-08-23** in
|
||
`perso/plakar/integration-incus` (walker modeled on the official sftp
|
||
integration; enumerate project → per-instance sftp walk → per-file
|
||
records + synthetic `.incus.yaml`; allow/deny lists; VM sftp failures
|
||
reported as record errors, not fatal). See its README for
|
||
build/install/TODO. Untested against a live incus yet.
|
||
|
||
Goal: plakar source connector for Incus so instance backups land in a
|
||
plakar kloset on S3 (leg 4 of the ks4 3-2-1 plan — see
|
||
[plan.md](../ks2/plan.md)).
|
||
|
||
## Decision 1 — per-file importer, not tarball streaming
|
||
|
||
The proxmox integration streams one full vzdump archive per VM
|
||
(`FLAG_STREAM`). For ks4 that model reads ~1.7 TB nightly (seafile
|
||
933 G, nextcloud 725 G): the incus backup/export API produces **full
|
||
tarballs only** — `shared/api/instance_backup.go` has just
|
||
`instance_only` and `optimized_storage`, no incremental option.
|
||
|
||
Instead the importer emits **one record per file** with real stat info
|
||
and a lazy reader. kloset's engine then does the incremental work for
|
||
us: `snapshot/backup.go` (`checkVFSCache`) reuses the cached object
|
||
when a path's stat matches the previous run and **never opens the
|
||
file**. Nightly cost after the initial seed = stat-walk + changed
|
||
files only; S3 receives only new chunks. Bonus: per-file browse and
|
||
restore of any snapshot.
|
||
|
||
File access is native incus REST: `GET /1.0/instances/{name}/sftp`
|
||
(a real SFTP session, also available per storage volume). Works over
|
||
the local unix socket or remote :8443, on stopped containers (the
|
||
daemon mounts the volume on demand), no ZFS assumptions.
|
||
|
||
## Decision 2 — do not reuse `incus copy --refresh` / migration API
|
||
|
||
Considered and rejected. `copy --refresh` is `POST /1.0/instances`
|
||
with `source.refresh=true` (`shared/api/instance.go:373`) and is
|
||
**not** zfs-only — transports are negotiated per storage driver
|
||
(`MigrationTypes()`: zfs/btrfs/ceph native streams, universal RSYNC
|
||
fallback, `driver_common.go:230`). But:
|
||
|
||
- the receiver must be another incus daemon speaking the migration
|
||
websocket protocol (`internal/migration/migrate.proto`);
|
||
- the optimized payload is an opaque `zfs send` stream — storing
|
||
those in plakar means chain-of-increments restore onto a real
|
||
zpool, no per-file browse, no pruning, one corrupt link breaks the
|
||
chain;
|
||
- the RSYNC transport needs a materialized previous copy on the
|
||
receiver — a full-size local mirror, i.e. leg 1 rebuilt inside a
|
||
plugin;
|
||
- gotcha: `driver_zfs.go:801` — refresh **without** snapshot copying
|
||
forces RSYNC even zfs↔zfs (keep `snapshots.schedule` on sources).
|
||
|
||
`copy --refresh` stays what legs 1–2 use (host↔host replication);
|
||
plakar is content-addressed archiving to dumb storage. Complementary,
|
||
not competing.
|
||
|
||
## Decision 3 — read the `backup` project replicas, not live instances
|
||
|
||
Reading a live rootfs is fuzzy. Leg 1 (`incus-copy.sh -p backup -s
|
||
backup`, 01:00) already produces quiescent, crash-consistent stopped
|
||
replicas — the importer targets those (`?project=backup`), scheduled
|
||
after the copy completes. DB consistency remains the mariadb-dump
|
||
job's responsibility (plakar `mysql` importer / dump step).
|
||
|
||
## Decision 4 — enumerate everything by default
|
||
|
||
`incus-backup.db` manifest drift (stale `spot`, missing `livetrail`,
|
||
`outline`, `login`…) is the failure mode to kill: the importer backs
|
||
up **all instances in the project by default**, opt-out via config,
|
||
never opt-in.
|
||
|
||
## Other design points
|
||
|
||
- Location scheme `incus://<remote>/<project>/<instance>/...` —
|
||
stable pathnames keep the kloset VFS cache effective.
|
||
- Synthetic records per instance: config + profiles
|
||
(`GET /1.0/instances/{name}`, expanded) so restore can recreate the
|
||
instance before pushing files back.
|
||
- Optional later `mode=image`: full tarball via the backups API for
|
||
occasional exact-image restores (`incus import`); plakar dedups
|
||
unchanged chunks between runs. Not the nightly path.
|
||
- Restore path (exporter): create instance from stored config → push
|
||
file tree back via sftp. Weaker than `incus import` but granular.
|
||
|
||
## Upstreaming (decided 2026-08-23: Route B — community integration)
|
||
|
||
Repo: `github.com/jlutran/plakar-integration-incus` (remote set,
|
||
initial commit done). Path to publication, in order:
|
||
|
||
1. unit tests + first real run against ks4's `backup` project
|
||
2. open a PlakarKorp/plakar issue proposing the integration
|
||
(disclose the Apache-2.0 `lxc/incus` client dependency)
|
||
3. push the repo, then PR a recipe against `PlakarKorp/hub` →
|
||
installable as `plakar pkg add incus`
|
||
4. (optional, later) adoption into the `integrations` monorepo:
|
||
module rename to `github.com/PlakarKorp/integrations/incus`,
|
||
orphan `integration/incus` branch, `incus/incus` layout
|
||
|
||
## Packaging & deployment
|
||
|
||
Plugins are standalone executables (gRPC over stdio) — no rebuild of
|
||
plakar, fully compatible with the APT-installed binary on ks4:
|
||
|
||
```sh
|
||
cd plakar/integration-incus && make # build importer/exporter
|
||
plakar pkg create manifest.yaml v0.1.0 # → incus_v0.1.0_linux_amd64.ptar
|
||
# on ks4, as root (plugins are per-user, cron runs as root):
|
||
plakar pkg add ./incus_v0.1.0_linux_amd64.ptar
|
||
plakar source add ks4-incus incus://... # then list it in /root/scripts/plakar-sources
|
||
```
|
||
|
||
Installing a *local* `.ptar` file needs no `plakar login` (the plugin
|
||
registry is only for fetching by name). Model the `manifest.yaml` on
|
||
integration-proxmox (`tier: third-party`, `api_version: v1.1.0`).
|
||
Compatibility contract = the go-kloset-sdk wire protocol: pin the SDK
|
||
to the installed plakar line (v1.1.x), rebuild the ptar on a plakar
|
||
major upgrade; a mismatch surfaces at backup time, so run a manual
|
||
`plakar backup @ks4-incus` after plakar upgrades.
|
||
|
||
## Open questions
|
||
|
||
- uid/gid view through the instance sftp endpoint (idmap handling for
|
||
unprivileged containers) — verify what stat returns and what restore
|
||
must remap.
|
||
- xattrs / ACLs / device nodes over sftp — coverage and fidelity.
|
||
- VMs: file access needs a running incus-agent — out of scope for ks4
|
||
(containers only); document the limitation.
|
||
- Scheduling/locking: chain after the 01:00 incus-copy (flock on the
|
||
same lock, or a wrapper script).
|
||
- S3 hardening: scoped credentials (no delete), versioning/object
|
||
lock, plakar client-side encryption passphrase storage.
|