Julien LutranandClaude Fable 5
a0e3a7dd86
- nuc/README.md and ks4/README.md carry the host sections (+ network flows) that lived in the top-level README; links rebased - top README: repo links (doc/scripts on git.lutran.fr), index points at the new per-host pages - cross-repo references now use https://git.lutran.fr/julien/scripts instead of relative ../scripts paths that resolve nowhere - plakar-s3-data.md and plakar-incus-integration.md marked SUPERSEDED / RETIRED with pointers to restic-backup.md; their measurements and rationale kept - install.md, local-backup-cron.md, incus-copy.md: crontab sections updated to the live schedule (01:00 replicas, 05:00 restic, Sun maintenance); retired legs labelled as such - restic-backup.md: status live, cutover recorded, post-GC memory estimate, seed plan dated - seafile-gc.md: online GC noted, stale 'crons commented out' removed - ks2/: what-ks2-does-today rewritten (nothing writes to it any more), legs table and gates reflect restic, decommission steps updated - db-exclude replaces the plakar-era config name (script keeps a fallback) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
186 lines
8.9 KiB
Markdown
186 lines
8.9 KiB
Markdown
# plakar incus integration — design notes (RETIRED)
|
||
|
||
> **Historical.** The importer works (smoke-tested against ks4's
|
||
> `backup` project) but production moved to restic, which needs no
|
||
> plugin: `incus file mount` exposes any instance — including stopped
|
||
> replicas — over the same per-instance sftp API
|
||
> ([restic-backup.md](restic-backup.md) §6). Code lives on at
|
||
> <https://github.com/jlutran/plakar-integration-incus>; the analysis
|
||
> below (per-file vs image, replica-vs-live consistency, the
|
||
> seek-bound wall) is what shaped the restic design.
|
||
|
||
|
||
Status: **importer scaffolded and compiling 2026-08-23** in
|
||
`perso/plakar/integration-incus` (walker modeled on the official sftp
|
||
integration; enumerate project → per-instance sftp walk → per-file
|
||
records + synthetic `.incus.yaml`; allow/deny lists; VM sftp failures
|
||
reported as record errors, not fatal). See its README for
|
||
build/install/TODO. Untested against a live incus yet.
|
||
|
||
Goal: plakar source connector for Incus so instance backups land in a
|
||
plakar kloset on S3 (leg 4 of the ks4 3-2-1 plan — see
|
||
[plan.md](../ks2/plan.md)).
|
||
|
||
## Decision 1 — per-file importer, not tarball streaming
|
||
|
||
The proxmox integration streams one full vzdump archive per VM
|
||
(`FLAG_STREAM`). For ks4 that model reads ~1.7 TB nightly (seafile
|
||
933 G, nextcloud 725 G): the incus backup/export API produces **full
|
||
tarballs only** — `shared/api/instance_backup.go` has just
|
||
`instance_only` and `optimized_storage`, no incremental option.
|
||
|
||
Instead the importer emits **one record per file** with real stat info
|
||
and a lazy reader. kloset's engine then does the incremental work for
|
||
us: `snapshot/backup.go` (`checkVFSCache`) reuses the cached object
|
||
when a path's stat matches the previous run and **never opens the
|
||
file**. Nightly cost after the initial seed = stat-walk + changed
|
||
files only; S3 receives only new chunks. Bonus: per-file browse and
|
||
restore of any snapshot.
|
||
|
||
File access is native incus REST: `GET /1.0/instances/{name}/sftp`
|
||
(a real SFTP session, also available per storage volume). Works over
|
||
the local unix socket or remote :8443, on stopped containers (the
|
||
daemon mounts the volume on demand), no ZFS assumptions.
|
||
|
||
## Decision 2 — do not reuse `incus copy --refresh` / migration API
|
||
|
||
Considered and rejected. `copy --refresh` is `POST /1.0/instances`
|
||
with `source.refresh=true` (`shared/api/instance.go:373`) and is
|
||
**not** zfs-only — transports are negotiated per storage driver
|
||
(`MigrationTypes()`: zfs/btrfs/ceph native streams, universal RSYNC
|
||
fallback, `driver_common.go:230`). But:
|
||
|
||
- the receiver must be another incus daemon speaking the migration
|
||
websocket protocol (`internal/migration/migrate.proto`);
|
||
- the optimized payload is an opaque `zfs send` stream — storing
|
||
those in plakar means chain-of-increments restore onto a real
|
||
zpool, no per-file browse, no pruning, one corrupt link breaks the
|
||
chain;
|
||
- the RSYNC transport needs a materialized previous copy on the
|
||
receiver — a full-size local mirror, i.e. leg 1 rebuilt inside a
|
||
plugin;
|
||
- gotcha: `driver_zfs.go:801` — refresh **without** snapshot copying
|
||
forces RSYNC even zfs↔zfs (keep `snapshots.schedule` on sources).
|
||
|
||
`copy --refresh` stays what legs 1–2 use (host↔host replication);
|
||
plakar is content-addressed archiving to dumb storage. Complementary,
|
||
not competing.
|
||
|
||
## Decision 3 — read the `backup` project replicas, not live instances
|
||
|
||
Reading a live rootfs is fuzzy. Leg 1 (`incus-copy.sh -p backup -s
|
||
backup`, 01:00) already produces quiescent, crash-consistent stopped
|
||
replicas — the importer targets those (`?project=backup`), scheduled
|
||
after the copy completes. DB consistency remains the mariadb-dump
|
||
job's responsibility (plakar `mysql` importer / dump step).
|
||
|
||
## Decision 4 — enumerate everything by default
|
||
|
||
`incus-backup.db` manifest drift (stale `spot`, missing `livetrail`,
|
||
`outline`, `login`…) is the failure mode to kill: the importer backs
|
||
up **all instances in the project by default**, opt-out via config,
|
||
never opt-in.
|
||
|
||
## Other design points
|
||
|
||
- Location scheme `incus://<remote>/<project>/<instance>/...` —
|
||
stable pathnames keep the kloset VFS cache effective.
|
||
- Synthetic records per instance: config + profiles
|
||
(`GET /1.0/instances/{name}`, expanded) so restore can recreate the
|
||
instance before pushing files back.
|
||
- Optional later `mode=image`: full tarball via the backups API for
|
||
occasional exact-image restores (`incus import`); plakar dedups
|
||
unchanged chunks between runs. Not the nightly path.
|
||
- Restore path (exporter): create instance from stored config → push
|
||
file tree back via sftp. Weaker than `incus import` but granular.
|
||
|
||
## Upstreaming (decided 2026-08-23: Route B — community integration)
|
||
|
||
Repo: `github.com/jlutran/plakar-integration-incus` (remote set,
|
||
initial commit done). Path to publication, in order:
|
||
|
||
1. unit tests + first real run against ks4's `backup` project
|
||
2. open a PlakarKorp/plakar issue proposing the integration
|
||
(disclose the Apache-2.0 `lxc/incus` client dependency)
|
||
3. push the repo, then PR a recipe against `PlakarKorp/hub` →
|
||
installable as `plakar pkg add incus`
|
||
4. (optional, later) adoption into the `integrations` monorepo:
|
||
module rename to `github.com/PlakarKorp/integrations/incus`,
|
||
orphan `integration/incus` branch, `incus/incus` layout
|
||
|
||
## Packaging & deployment
|
||
|
||
Plugins are standalone executables (gRPC over stdio) — no rebuild of
|
||
plakar, fully compatible with the APT-installed binary on ks4:
|
||
|
||
```sh
|
||
cd plakar/integration-incus && make # build importer/exporter
|
||
plakar pkg create manifest.yaml v0.1.0 # → incus_v0.1.0_linux_amd64.ptar
|
||
# on ks4, as root (plugins are per-user, cron runs as root):
|
||
plakar pkg add ./incus_v0.1.0_linux_amd64.ptar
|
||
plakar source add ks4-incus incus://... # then list it in /root/scripts/plakar-sources
|
||
```
|
||
|
||
Installing a *local* `.ptar` file needs no `plakar login` (the plugin
|
||
registry is only for fetching by name). Model the `manifest.yaml` on
|
||
integration-proxmox (`tier: third-party`, `api_version: v1.1.0`).
|
||
Compatibility contract = the go-kloset-sdk wire protocol: pin the SDK
|
||
to the installed plakar line (v1.1.x), rebuild the ptar on a plakar
|
||
major upgrade; a mismatch surfaces at backup time, so run a manual
|
||
`plakar backup @ks4-incus` after plakar upgrades.
|
||
|
||
## The giant-instance problem (2026-08-24, seed attempt #1)
|
||
|
||
The per-file S3 seed of the whole `backup` project stalled at
|
||
~1 MiB/s once it reached the big replicas: sdb at 96 % util doing
|
||
~275 × 4 KiB reads/s — **cold random tiny-file reads on an HDD are
|
||
seek-bound**, regardless of importer (the fs importer hit the same
|
||
wall on the live pool; sftp changes nothing). At ~275 IOPS,
|
||
nextcloud (725 G, ~500 k files) + seafile (933 G, millions of block
|
||
files) need *weeks* to seed. The 16 small instances are unaffected
|
||
(solar smoke test: 5 GiB in 2m38s).
|
||
|
||
Options considered for nextcloud/seafile:
|
||
|
||
1. **backups-API tarball (`mode=image`)** — sequential-ish but incus
|
||
materializes the tarball server-side first (no room on `/`; would
|
||
need `storage.backups_volume` on the backup pool) and the tar walk
|
||
itself seeks like any tree walk. Double IO nightly. Weak.
|
||
2. **`zfs send` of the replica dataset** — the only truly sequential
|
||
read (disk block order, ~100+ MiB/s, no temp space): seed in
|
||
hours. Full send nightly is ~4 h read for both giants; CDC chunking
|
||
dedups unchanged stream regions. Restore = `zfs receive` +
|
||
`incus admin recover` (document!). Backend-specific — acceptable as
|
||
an opt-in mode for exactly these two.
|
||
3. **Per-file + warm metadata** — after a seed, nightly walks are
|
||
stat-only; keeping dnodes resident (`zfs_arc_min` ≈ 8 G) could make
|
||
them fast. Doesn't solve the *seed*.
|
||
4. **Exclude giants from this leg** — leaves seafile without any
|
||
fresh off-site copy (data leg already excludes it). Not acceptable
|
||
long-term.
|
||
|
||
Direction (pending decision): seed the 16 small instances per-file
|
||
(`exclude_instances=nextcloud,seafile`), then implement option 2 as a
|
||
`zfs+send://` mode or side-channel for the two giants, and revisit 3
|
||
for nightly stat-walk speed.
|
||
|
||
## Open questions
|
||
|
||
- uid/gid view through the instance sftp endpoint (idmap handling for
|
||
unprivileged containers) — verify what stat returns and what restore
|
||
must remap.
|
||
- xattrs / ACLs / device nodes over sftp — coverage and fidelity.
|
||
- VMs: file access needs a running incus-agent — out of scope for ks4
|
||
(containers only); document the limitation.
|
||
- ~~Scheduling/locking~~ decided 2026-08-23: the S3 instance leg is
|
||
only as fresh as the last **completed** local incus-copy, so both
|
||
run chained in one cron entry —
|
||
`incus-copy.sh -p backup -s backup ; <plakar incus leg>` — never at
|
||
a fixed offset (a long refresh, e.g. a post-gap full re-send, would
|
||
race it). `;` not `&&`: a partially failed copy leaves stale but
|
||
internally consistent replicas, still worth uploading. Same rule
|
||
for manual test runs: check `/var/log/incus-copy.log` completed
|
||
first.
|
||
- S3 hardening: scoped credentials (no delete), versioning/object
|
||
lock, plakar client-side encryption passphrase storage.
|