Document the ks2 decommission plan and the new ks4 backup legs
- ks2/: decommission plan with release gates (rental ends Sep 30), nuc-seed and final-decommission runbooks - ks4/local-backup-cron.md: leg-1 cron re-enabled, ks2 rsync stopgap, full-resend caveat when refreshes lapse past snapshots.expiry - ks4/plakar-s3-data.md: plakar direct-to-S3 leg (auto-discovered DB dumps incl. grants/globals, fs sources, exclude list, restore test) - ks4/plakar-incus-integration.md: design + scaffold status of the plakar incus importer - README.md: index ks2/, update the ks4 durability bullet Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Julien Lutran
co-authored by
Claude Fable 5
Claude Fable 5
parent
4b30b43c7f
commit
79ad1b7179
@@ -4,6 +4,11 @@ Personal infrastructure notes.
|
|||||||
|
|
||||||
- [`nuc/`](nuc/) — home lab on `nuc` (see below)
|
- [`nuc/`](nuc/) — home lab on `nuc` (see below)
|
||||||
- [`ks4/`](ks4/) — prod server `ks4` at OVH (see below)
|
- [`ks4/`](ks4/) — prod server `ks4` at OVH (see below)
|
||||||
|
- [`ks2/`](ks2/) — legacy backup server being decommissioned
|
||||||
|
(rental ends Sep 30, 2026): [plan](ks2/plan.md)
|
||||||
|
- [`archer-c7/`](archer-c7/) — home router (TP-Link Archer C7 v5,
|
||||||
|
OpenWrt, `192.168.0.2`, WAN = phone USB tether):
|
||||||
|
[upgrade to 25.12](archer-c7/upgrade-openwrt-25.12.md)
|
||||||
|
|
||||||
## Homelab: nuc
|
## Homelab: nuc
|
||||||
|
|
||||||
@@ -56,10 +61,15 @@ Incus host at OVH — public-facing self-hosted services.
|
|||||||
outline, login (SSO), wireguard, + more — full table in
|
outline, login (SSO), wireguard, + more — full table in
|
||||||
[ks4/install.md](ks4/install.md).
|
[ks4/install.md](ks4/install.md).
|
||||||
- ⚠️ The ZFS `data` pool is single-disk (not mirrored); durability
|
- ⚠️ The ZFS `data` pool is single-disk (not mirrored); durability
|
||||||
rests on two nightly root cron jobs — `incus copy --refresh` to remote
|
rests on nightly root cron jobs — 01:00 `incus copy --refresh` of
|
||||||
host `ks2` (moving to nuc pool `ks4backup`, see
|
all instances to the local `backup` pool on sdb5
|
||||||
[ks4/incus-copy.md](ks4/incus-copy.md)), and an rsync backup
|
([ks4/local-backup-cron.md](ks4/local-backup-cron.md),
|
||||||
(`scripts/incus-backup.sh`) to `164.132.173.57:/backup/ks4`.
|
[ks4/incus-copy.md](ks4/incus-copy.md)), 04:30 plakar backups
|
||||||
|
(DB dumps + data trees) to S3
|
||||||
|
([ks4/plakar-s3-data.md](ks4/plakar-s3-data.md)), plus during the
|
||||||
|
transition a 04:00 rsync stopgap to ks2
|
||||||
|
([ks2 decommission plan](ks2/plan.md)); a pull leg to nuc pool
|
||||||
|
`ks4backup` awaits FTTH.
|
||||||
|
|
||||||
## Network flows (nuc <-> ks4)
|
## Network flows (nuc <-> ks4)
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,60 @@
|
|||||||
|
# ks2 decommission — final checklist
|
||||||
|
|
||||||
|
Status: **prepared — execute only when every gate in
|
||||||
|
[plan.md](plan.md) is ticked.** Rental ends **Sep 30, 2026**; leave a
|
||||||
|
few days of margin.
|
||||||
|
|
||||||
|
## Preconditions (the plan's release gates)
|
||||||
|
|
||||||
|
- [ ] plakar S3 leg running nightly for ≥ a week, `done (rc=0)`,
|
||||||
|
restore test passed ([plakar-s3-data.md](../ks4/plakar-s3-data.md) §7)
|
||||||
|
- [ ] nuc pull leg seeded and one instance test-restored
|
||||||
|
([nuc-seed.md](nuc-seed.md))
|
||||||
|
- [ ] local leg (sdb5) cron green in `/var/log/incus-copy.log`
|
||||||
|
|
||||||
|
## 1. Cut the last flows to ks2 (root on ks4)
|
||||||
|
|
||||||
|
```sh
|
||||||
|
crontab -e # delete the 04:00 incus-backup.sh line (164.132.173.57)
|
||||||
|
incus remote list # if a "ks2" remote is still defined:
|
||||||
|
incus remote remove ks2
|
||||||
|
# drop the rsyncbackup SSH key if it exists only for ks2:
|
||||||
|
ls /root/.ssh/id_rsyncbackup* && rm -i /root/.ssh/id_rsyncbackup*
|
||||||
|
```
|
||||||
|
|
||||||
|
## 2. Last look at ks2 before wiping (root on ks2)
|
||||||
|
|
||||||
|
```sh
|
||||||
|
# anything in the rsync tree newer than the plakar cutover?
|
||||||
|
find /backup/ns3061243 -newermt '2026-08-23' | head
|
||||||
|
# pre-2023 dirs (catc, mythoughts, qcm, scripts): instances still
|
||||||
|
# exist stopped on ks4 — spot-check nothing unique remains:
|
||||||
|
ls /backup/ns3061243/{catc,mythoughts,qcm,scripts}
|
||||||
|
```
|
||||||
|
|
||||||
|
Archive anything unique to the plakar dump dir on ks4 (it rides the
|
||||||
|
nightly `dumps` snapshot to S3 for free once it stops changing).
|
||||||
|
|
||||||
|
## 3. Wipe and terminate
|
||||||
|
|
||||||
|
```sh
|
||||||
|
# on ks2 — destroy the data (both pools hold copies of personal data):
|
||||||
|
zpool destroy backup && zpool destroy data
|
||||||
|
# optional: shred the OS disks via OVH rescue mode / reinstall
|
||||||
|
```
|
||||||
|
|
||||||
|
Then in the OVH manager: cancel the ks2 (Kimsufi) server before the
|
||||||
|
renewal date.
|
||||||
|
|
||||||
|
## 4. Post-mortem cleanup
|
||||||
|
|
||||||
|
- [ ] Update [doc/README.md](../README.md): remove ks2 from the ks4
|
||||||
|
durability bullet and the index line for `ks2/`s "being
|
||||||
|
decommissioned" wording (keep the folder as history)
|
||||||
|
- [ ] Retire `scripts/incus-backup.sh` + `incus-backup.db` in the repo
|
||||||
|
(superseded by `scripts/plakar-backup.sh`) — keep for reference,
|
||||||
|
note the retirement in their headers
|
||||||
|
- [ ] `login` container cleanup parked in
|
||||||
|
[plakar-s3-data.md](../ks4/plakar-s3-data.md): `dropdb outline`,
|
||||||
|
remove native postgres
|
||||||
|
- [ ] Mark the ks2 line in `plan.md` done, date it
|
||||||
@@ -0,0 +1,49 @@
|
|||||||
|
# nuc pull leg — seed after FTTH
|
||||||
|
|
||||||
|
Status: **prepared, waiting on the FTTH link.** Everything is already
|
||||||
|
configured on nuc (see the main [README](../README.md)): incus remote
|
||||||
|
`ks4` over the WireGuard tunnel (`wg-ks4`, 10.8.0.20 → 10.8.0.1),
|
||||||
|
target pool `ks4backup` on the USB ZFS pool (`usb4t/backup/ks4`) —
|
||||||
|
only the seed itself waited on bandwidth.
|
||||||
|
|
||||||
|
## Seed (root on nuc)
|
||||||
|
|
||||||
|
```sh
|
||||||
|
# sanity: remote reachable through the tunnel
|
||||||
|
incus list ks4: | head
|
||||||
|
|
||||||
|
# full pull of every ks4 instance into pool ks4backup (screen/tmux —
|
||||||
|
# first pass moves ~1.7 T through the WG tunnel)
|
||||||
|
/root/scripts/incus-copy.sh -r ks4 -s ks4backup 2>&1 | tee -a /var/log/incus-copy-ks4.log
|
||||||
|
```
|
||||||
|
|
||||||
|
Notes:
|
||||||
|
|
||||||
|
- First pass is a full send per instance; later refreshes are
|
||||||
|
ZFS-incremental **as long as they run at least every
|
||||||
|
`snapshots.expiry` (7 d on ks4)** — same caveat as
|
||||||
|
[ks4's local leg](../ks4/local-backup-cron.md).
|
||||||
|
- Replicas arrive stopped with `boot.autostart=false` (script does
|
||||||
|
this) — they must never come up on the LAN with ks4's proxy devices.
|
||||||
|
|
||||||
|
## Cron (after the seed)
|
||||||
|
|
||||||
|
Add to nuc's root crontab, offset from the 03:30 local nucbackup copy
|
||||||
|
and ks4's own 01:00/04:30 jobs:
|
||||||
|
|
||||||
|
```cron
|
||||||
|
0 5 * * * /root/scripts/incus-copy.sh -r ks4 -s ks4backup >> /var/log/incus-copy-ks4.log 2>&1
|
||||||
|
```
|
||||||
|
|
||||||
|
## Verification (release gate for ks2)
|
||||||
|
|
||||||
|
```sh
|
||||||
|
incus list --project backup 2>/dev/null || incus list | grep -c . # all ks4 instances present
|
||||||
|
# test-restore one instance: copy a replica to the default pool,
|
||||||
|
# start it isolated, check the service answers, then delete it
|
||||||
|
incus copy solar solar-restoretest -s default
|
||||||
|
incus start solar-restoretest && incus exec solar-restoretest -- systemctl is-system-running
|
||||||
|
incus delete -f solar-restoretest
|
||||||
|
```
|
||||||
|
|
||||||
|
Once verified, tick the nuc gate in the [ks2 plan](plan.md).
|
||||||
+78
@@ -0,0 +1,78 @@
|
|||||||
|
# ks2 decommission — summary plan
|
||||||
|
|
||||||
|
Goal: release `ks2.lutran.fr` (`164.132.173.57`, SSH :2233, host
|
||||||
|
`ns3247221`) once ks4 has a real 3-2-1 backup without it.
|
||||||
|
**Deadline: rental ends Sep 30, 2026.**
|
||||||
|
|
||||||
|
## What ks2 does today (updated 2026-08-22)
|
||||||
|
|
||||||
|
One remaining role, fed nightly by a root cron **on ks4**:
|
||||||
|
|
||||||
|
- `04:00` `incus-backup.sh … -d 164.132.173.57` — DB dumps + selected
|
||||||
|
paths rsync'd to ks2 `/backup/ns3061243` on pool `backup` (1.59 T,
|
||||||
|
88 % full), snapshotted daily on ks2 (`zfs-auto-snapshot.sh`,
|
||||||
|
2-month expiry). **Kept as a stopgap** until the plakar S3 leg
|
||||||
|
replaces it.
|
||||||
|
|
||||||
|
Retired: the `01:00` replica push to ks2 pool `data` — replaced by
|
||||||
|
the local leg on ks4's sdb5 pool
|
||||||
|
([local-backup-cron.md](../ks4/local-backup-cron.md)); `biwiki` and `spot`
|
||||||
|
deleted from ks2.
|
||||||
|
|
||||||
|
## Inventory findings (2026-08-22)
|
||||||
|
|
||||||
|
- `biwiki` exists only on ks2, `spot` only on ks2 + ks4's `backup`
|
||||||
|
project — **decision: both abandoned**, safe to delete from backups.
|
||||||
|
- `incus-backup.db` is stale (lists `spot`, misses `livetrail`,
|
||||||
|
`outline`, `login`, …) — forgotten-manifest drift is exactly what
|
||||||
|
the plakar-based replacement must eliminate (back up *all*
|
||||||
|
instances by default, opt-out instead of opt-in).
|
||||||
|
- `livetrail` (running on ks4) is **absent from ks4's local `backup`
|
||||||
|
project** — the local leg was run manually once and never again;
|
||||||
|
automating its cron fixes this.
|
||||||
|
- Both ks4 backup crons were **commented out — no ks4 backup ran at
|
||||||
|
all** (hence the stale ks2 rsync data: dirs `Jul 23`, incus dumps
|
||||||
|
`Aug 9`). Fixed 2026-08-22 — see
|
||||||
|
[local-backup-cron.md](../ks4/local-backup-cron.md).
|
||||||
|
- `/backup/ns3061243` also holds pre-2023 dirs (`catc`, `mythoughts`,
|
||||||
|
`qcm`) — those instances still exist stopped on ks4, so nothing
|
||||||
|
unique expected there; spot-check before wiping.
|
||||||
|
- Losing ks2 also loses its 2 months of rsync-snapshot history —
|
||||||
|
acceptable once plakar has built equivalent retention.
|
||||||
|
|
||||||
|
## Target architecture (3-2-1 for ks4)
|
||||||
|
|
||||||
|
| Leg | Mechanism | Status |
|
||||||
|
|---|---|---|
|
||||||
|
| local, 2nd disk | `incus-copy.sh -p backup -s backup` → sdb5 pool | works, **cron missing** |
|
||||||
|
| off-site, nuc | nuc pulls `ks4:*` → pool `ks4backup` over WG | waiting FTTH seed |
|
||||||
|
| off-site, S3 (data+DB) | plakar on ks4 → S3 (replaces `incus-backup.sh`) | to do |
|
||||||
|
| off-site, S3 (instances) | plakar **incus integration** (to develop) → S3 | to do |
|
||||||
|
|
||||||
|
## Actions (backup work documented in [`../ks4/`](../ks4/), ks2-only tasks here)
|
||||||
|
|
||||||
|
1. ~~salvage biwiki/spot~~ — decided 2026-08-22: both abandoned,
|
||||||
|
**deleted** from ks2 and ks4's `backup` project
|
||||||
|
2. ~~[local-backup-cron.md](../ks4/local-backup-cron.md)~~ — **done
|
||||||
|
2026-08-22**: leg 1 cron re-enabled, ks2 rsync kept as stopgap;
|
||||||
|
`livetrail` verified present in the `backup` project
|
||||||
|
3. [plakar-s3-data.md](../ks4/plakar-s3-data.md) — **seeding S3
|
||||||
|
2026-08-22** (direct-to-S3 kloset `lutran-ks4-plakar-data`, driver
|
||||||
|
script `scripts/plakar-backup.sh`, pipeline validated locally);
|
||||||
|
next: restore test + cron
|
||||||
|
4. [plakar-incus-integration.md](../ks4/plakar-incus-integration.md) —
|
||||||
|
design seeded 2026-08-22: per-file importer over the incus sftp
|
||||||
|
API, instances → S3
|
||||||
|
5. [nuc-seed.md](nuc-seed.md) — **prepared**; after FTTH: seed nuc
|
||||||
|
pull leg, verify all instances, test-restore one
|
||||||
|
6. [decommission.md](decommission.md) — **prepared**; cut flows,
|
||||||
|
final diff of `/backup/ns3061243`, wipe pools, terminate at OVH
|
||||||
|
|
||||||
|
## Release gates — ks2 can be dropped only when
|
||||||
|
|
||||||
|
- [x] biwiki + spot consciously abandoned (2026-08-22)
|
||||||
|
- [ ] local leg cron running ≥ a few days, all instances present
|
||||||
|
- [ ] nuc leg fully seeded **and** one instance test-restored
|
||||||
|
- [ ] plakar S3 data/DB backups running daily **and** one DB + one
|
||||||
|
file tree test-restored
|
||||||
|
- [ ] ks4 crons pointing at ks2 disabled
|
||||||
@@ -0,0 +1,56 @@
|
|||||||
|
# local backup cron on ks4 (leg 1) — re-enabled
|
||||||
|
|
||||||
|
Done 2026-08-22. Context: **both** ks4 backup crons had been commented
|
||||||
|
out — no backup of any kind was running.
|
||||||
|
|
||||||
|
## What runs now (root crontab on ks4)
|
||||||
|
|
||||||
|
```cron
|
||||||
|
0 1 * * * /root/scripts/incus-copy.sh -p backup -s backup >> /var/log/incus-copy.log 2>&1
|
||||||
|
0 4 * * * /root/scripts/incus-backup.sh -d 164.132.173.57 -u rsyncbackup -i /root/.ssh/id_rsyncbackup -p 2233 -f /root/scripts/incus-backup.db -s data >> /var/log/incus-backup.log 2>&1
|
||||||
|
```
|
||||||
|
|
||||||
|
- `01:00` — **local leg**: `incus copy --refresh` of all instances
|
||||||
|
into project `backup` on the sdb5 `backup` zpool (replaces the old
|
||||||
|
push to ks2; see [incus-copy.md](incus-copy.md)).
|
||||||
|
- `04:00` — **stopgap**: `incus-backup.sh` rsync to ks2 kept alive
|
||||||
|
(manifest pruned of `spot`) until the plakar S3 leg replaces it —
|
||||||
|
then this line goes away as part of
|
||||||
|
[decommissioning ks2](../ks2/plan.md). ⚠️ keep
|
||||||
|
`scripts/incus-backup.db` in this repo in sync with
|
||||||
|
`/root/scripts/incus-backup.db` on ks4.
|
||||||
|
|
||||||
|
Both were also run once by hand (in a `screen`) on 2026-08-22 to
|
||||||
|
refresh the backups immediately. The old push-to-ks2 copy line stays
|
||||||
|
commented out — replica refreshes to ks2 are retired.
|
||||||
|
|
||||||
|
## Cleanup done the same day
|
||||||
|
|
||||||
|
- ks2: `incus delete biwiki spot` (abandoned instances — freed space
|
||||||
|
on the 95 %-full `data` pool)
|
||||||
|
- ks4: `incus delete spot --project backup`
|
||||||
|
|
||||||
|
## Verification
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# all instances present, incl. livetrail (first copy is a full send)
|
||||||
|
sudo incus list --project backup -c ns -f csv
|
||||||
|
# cron runs land here (manual screen runs don't):
|
||||||
|
sudo tail /var/log/incus-copy.log /var/log/incus-backup.log
|
||||||
|
# rsync leg freshness, on ks2:
|
||||||
|
ls -lat /backup/ns3061243/ | head
|
||||||
|
```
|
||||||
|
|
||||||
|
Notes:
|
||||||
|
|
||||||
|
- `flock` in both scripts makes overlapping cron/manual runs abort
|
||||||
|
safely instead of racing.
|
||||||
|
- Replicas get `boot.autostart=false` set by the script — they must
|
||||||
|
never fight the live instances for their static IPs.
|
||||||
|
- Keep `snapshots.schedule` set on the sources so refreshes stay
|
||||||
|
ZFS-incremental (a refresh without common snapshots falls back to
|
||||||
|
rsync). Corollary: the cron must run at least every
|
||||||
|
`snapshots.expiry` (7 d) — a longer gap expires every common
|
||||||
|
snapshot and the next refresh silently degrades to a **full
|
||||||
|
re-send** (observed 2026-08-22: seafile re-sent 933 G after the
|
||||||
|
weeks-long gap while the crons were commented out).
|
||||||
@@ -0,0 +1,119 @@
|
|||||||
|
# plakar incus integration — design notes
|
||||||
|
|
||||||
|
Status: **importer scaffolded and compiling 2026-08-23** in
|
||||||
|
`perso/plakar/integration-incus` (walker modeled on the official sftp
|
||||||
|
integration; enumerate project → per-instance sftp walk → per-file
|
||||||
|
records + synthetic `.incus.yaml`; allow/deny lists; VM sftp failures
|
||||||
|
reported as record errors, not fatal). See its README for
|
||||||
|
build/install/TODO. Untested against a live incus yet.
|
||||||
|
|
||||||
|
Goal: plakar source connector for Incus so instance backups land in a
|
||||||
|
plakar kloset on S3 (leg 4 of the ks4 3-2-1 plan — see
|
||||||
|
[plan.md](../ks2/plan.md)).
|
||||||
|
|
||||||
|
## Decision 1 — per-file importer, not tarball streaming
|
||||||
|
|
||||||
|
The proxmox integration streams one full vzdump archive per VM
|
||||||
|
(`FLAG_STREAM`). For ks4 that model reads ~1.7 TB nightly (seafile
|
||||||
|
933 G, nextcloud 725 G): the incus backup/export API produces **full
|
||||||
|
tarballs only** — `shared/api/instance_backup.go` has just
|
||||||
|
`instance_only` and `optimized_storage`, no incremental option.
|
||||||
|
|
||||||
|
Instead the importer emits **one record per file** with real stat info
|
||||||
|
and a lazy reader. kloset's engine then does the incremental work for
|
||||||
|
us: `snapshot/backup.go` (`checkVFSCache`) reuses the cached object
|
||||||
|
when a path's stat matches the previous run and **never opens the
|
||||||
|
file**. Nightly cost after the initial seed = stat-walk + changed
|
||||||
|
files only; S3 receives only new chunks. Bonus: per-file browse and
|
||||||
|
restore of any snapshot.
|
||||||
|
|
||||||
|
File access is native incus REST: `GET /1.0/instances/{name}/sftp`
|
||||||
|
(a real SFTP session, also available per storage volume). Works over
|
||||||
|
the local unix socket or remote :8443, on stopped containers (the
|
||||||
|
daemon mounts the volume on demand), no ZFS assumptions.
|
||||||
|
|
||||||
|
## Decision 2 — do not reuse `incus copy --refresh` / migration API
|
||||||
|
|
||||||
|
Considered and rejected. `copy --refresh` is `POST /1.0/instances`
|
||||||
|
with `source.refresh=true` (`shared/api/instance.go:373`) and is
|
||||||
|
**not** zfs-only — transports are negotiated per storage driver
|
||||||
|
(`MigrationTypes()`: zfs/btrfs/ceph native streams, universal RSYNC
|
||||||
|
fallback, `driver_common.go:230`). But:
|
||||||
|
|
||||||
|
- the receiver must be another incus daemon speaking the migration
|
||||||
|
websocket protocol (`internal/migration/migrate.proto`);
|
||||||
|
- the optimized payload is an opaque `zfs send` stream — storing
|
||||||
|
those in plakar means chain-of-increments restore onto a real
|
||||||
|
zpool, no per-file browse, no pruning, one corrupt link breaks the
|
||||||
|
chain;
|
||||||
|
- the RSYNC transport needs a materialized previous copy on the
|
||||||
|
receiver — a full-size local mirror, i.e. leg 1 rebuilt inside a
|
||||||
|
plugin;
|
||||||
|
- gotcha: `driver_zfs.go:801` — refresh **without** snapshot copying
|
||||||
|
forces RSYNC even zfs↔zfs (keep `snapshots.schedule` on sources).
|
||||||
|
|
||||||
|
`copy --refresh` stays what legs 1–2 use (host↔host replication);
|
||||||
|
plakar is content-addressed archiving to dumb storage. Complementary,
|
||||||
|
not competing.
|
||||||
|
|
||||||
|
## Decision 3 — read the `backup` project replicas, not live instances
|
||||||
|
|
||||||
|
Reading a live rootfs is fuzzy. Leg 1 (`incus-copy.sh -p backup -s
|
||||||
|
backup`, 01:00) already produces quiescent, crash-consistent stopped
|
||||||
|
replicas — the importer targets those (`?project=backup`), scheduled
|
||||||
|
after the copy completes. DB consistency remains the mariadb-dump
|
||||||
|
job's responsibility (plakar `mysql` importer / dump step).
|
||||||
|
|
||||||
|
## Decision 4 — enumerate everything by default
|
||||||
|
|
||||||
|
`incus-backup.db` manifest drift (stale `spot`, missing `livetrail`,
|
||||||
|
`outline`, `login`…) is the failure mode to kill: the importer backs
|
||||||
|
up **all instances in the project by default**, opt-out via config,
|
||||||
|
never opt-in.
|
||||||
|
|
||||||
|
## Other design points
|
||||||
|
|
||||||
|
- Location scheme `incus://<remote>/<project>/<instance>/...` —
|
||||||
|
stable pathnames keep the kloset VFS cache effective.
|
||||||
|
- Synthetic records per instance: config + profiles
|
||||||
|
(`GET /1.0/instances/{name}`, expanded) so restore can recreate the
|
||||||
|
instance before pushing files back.
|
||||||
|
- Optional later `mode=image`: full tarball via the backups API for
|
||||||
|
occasional exact-image restores (`incus import`); plakar dedups
|
||||||
|
unchanged chunks between runs. Not the nightly path.
|
||||||
|
- Restore path (exporter): create instance from stored config → push
|
||||||
|
file tree back via sftp. Weaker than `incus import` but granular.
|
||||||
|
|
||||||
|
## Packaging & deployment
|
||||||
|
|
||||||
|
Plugins are standalone executables (gRPC over stdio) — no rebuild of
|
||||||
|
plakar, fully compatible with the APT-installed binary on ks4:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
cd plakar/integration-incus && make # build importer/exporter
|
||||||
|
plakar pkg create manifest.yaml v0.1.0 # → incus_v0.1.0_linux_amd64.ptar
|
||||||
|
# on ks4, as root (plugins are per-user, cron runs as root):
|
||||||
|
plakar pkg add ./incus_v0.1.0_linux_amd64.ptar
|
||||||
|
plakar source add ks4-incus incus://... # then list it in /root/scripts/plakar-sources
|
||||||
|
```
|
||||||
|
|
||||||
|
Installing a *local* `.ptar` file needs no `plakar login` (the plugin
|
||||||
|
registry is only for fetching by name). Model the `manifest.yaml` on
|
||||||
|
integration-proxmox (`tier: third-party`, `api_version: v1.1.0`).
|
||||||
|
Compatibility contract = the go-kloset-sdk wire protocol: pin the SDK
|
||||||
|
to the installed plakar line (v1.1.x), rebuild the ptar on a plakar
|
||||||
|
major upgrade; a mismatch surfaces at backup time, so run a manual
|
||||||
|
`plakar backup @ks4-incus` after plakar upgrades.
|
||||||
|
|
||||||
|
## Open questions
|
||||||
|
|
||||||
|
- uid/gid view through the instance sftp endpoint (idmap handling for
|
||||||
|
unprivileged containers) — verify what stat returns and what restore
|
||||||
|
must remap.
|
||||||
|
- xattrs / ACLs / device nodes over sftp — coverage and fidelity.
|
||||||
|
- VMs: file access needs a running incus-agent — out of scope for ks4
|
||||||
|
(containers only); document the limitation.
|
||||||
|
- Scheduling/locking: chain after the 01:00 incus-copy (flock on the
|
||||||
|
same lock, or a wrapper script).
|
||||||
|
- S3 hardening: scoped credentials (no delete), versioning/object
|
||||||
|
lock, plakar client-side encryption passphrase storage.
|
||||||
@@ -0,0 +1,221 @@
|
|||||||
|
# plakar data + DB backups on ks4 (leg 3)
|
||||||
|
|
||||||
|
Status: **pipeline validated 2026-08-22** against a local test kloset
|
||||||
|
(`/backup/plakar-test`): full dump auto-discovery incl. grants/globals,
|
||||||
|
exclude mechanism, fs sources, prune + maintenance, rc=0; second run
|
||||||
|
confirmed incremental behavior (21 GiB dumps snapshot → 60 MiB
|
||||||
|
written). **S3 credentials ready — next: store add, create, seed.**
|
||||||
|
Replaces `incus-backup.sh` (the 04:00 rsync to ks2 stays alive as a
|
||||||
|
stopgap until this leg is verified — see
|
||||||
|
[local-backup-cron.md](local-backup-cron.md)).
|
||||||
|
|
||||||
|
Architecture: plakar runs **on ks4** (decision recorded in
|
||||||
|
[plan.md](../ks2/plan.md)) and backs up **directly into an S3 kloset**
|
||||||
|
(`plakar at @s3 backup @src`). A local staging kloset on sdb5 was
|
||||||
|
considered and rejected: it would double-store ~1.5 T that the leg-1
|
||||||
|
replicas already hold on the same disk, and local restore is already
|
||||||
|
leg 1's job. Direct-to-S3 keeps the incremental behavior — the
|
||||||
|
skip-unchanged baseline is the parent snapshot's VFS read from the
|
||||||
|
kloset (plakar caches repository metadata locally; v1.1.4's dirpack
|
||||||
|
prefetcher targets exactly this remote-backend walk). Driver script:
|
||||||
|
[`scripts/plakar-backup.sh`](../../scripts/plakar-backup.sh)
|
||||||
|
(flock, incus DB dumps, iterates a sources file, prune + maintenance;
|
||||||
|
its `-k` takes any `plakar at` target, so `-k @s3` — a throwaway
|
||||||
|
local kloset, e.g. `-k /backup/plakar-test`, works for testing before
|
||||||
|
credentials arrive and can be deleted after).
|
||||||
|
|
||||||
|
## 1. Install plakar via the official APT repo (root on ks4)
|
||||||
|
|
||||||
|
Repo verified reachable 2026-08-22 (suite `stable`, amd64/arm64);
|
||||||
|
upgrades then ride the normal `apt upgrade` flow.
|
||||||
|
|
||||||
|
```sh
|
||||||
|
curl -fsSL https://plakar.io/dist/keys/community-v1.1.0.gpg \
|
||||||
|
| gpg --dearmor -o /usr/share/keyrings/plakar.gpg
|
||||||
|
echo "deb [signed-by=/usr/share/keyrings/plakar.gpg] https://plakar.io/dist/repos/deb/ stable main" \
|
||||||
|
> /etc/apt/sources.list.d/plakar.list
|
||||||
|
apt update && apt install plakar
|
||||||
|
plakar version # v1.1.4 at time of writing
|
||||||
|
```
|
||||||
|
|
||||||
|
The mysql integration is a plugin; official plugins install from the
|
||||||
|
plugin server after `plakar login`, or build locally (needs make + Go):
|
||||||
|
|
||||||
|
```sh
|
||||||
|
plakar pkg add mysql # or: clone integrations, cd mysql && make install
|
||||||
|
```
|
||||||
|
|
||||||
|
## 2. Kloset + passphrase
|
||||||
|
|
||||||
|
```sh
|
||||||
|
head -c 32 /dev/urandom | base64 > /root/.plakar-passphrase
|
||||||
|
chmod 600 /root/.plakar-passphrase
|
||||||
|
# ⚠️ copy the passphrase to the password manager NOW — without it
|
||||||
|
# the S3 kloset is unreadable garbage.
|
||||||
|
export PLAKAR_PASSPHRASE=$(cat /root/.plakar-passphrase)
|
||||||
|
|
||||||
|
# when S3 credentials are available:
|
||||||
|
plakar store add s3 s3://<endpoint>/<bucket> access_key=<AK> secret_access_key=<SK> use_tls=true
|
||||||
|
plakar at @s3 create
|
||||||
|
# meanwhile, a throwaway kloset lets everything below be tested now:
|
||||||
|
plakar at /backup/plakar-test create
|
||||||
|
```
|
||||||
|
|
||||||
|
## 3. Database dumps — `incus exec`, no DB users needed
|
||||||
|
|
||||||
|
The plakar mysql integration was considered and rejected: it connects
|
||||||
|
over TCP, which would have required a bind-address change plus a
|
||||||
|
dedicated backup user in every DB container. Instead the driver
|
||||||
|
script keeps the old `incus-backup.sh` method — `incus exec <ct> --
|
||||||
|
mariadb-dump --single-transaction ...` (container-local root over the
|
||||||
|
unix socket) — with one upgrade: **databases are auto-discovered**,
|
||||||
|
so there is no manifest to go stale (the failure mode that bit
|
||||||
|
`incus-backup.db`). Three probes per *running* container:
|
||||||
|
|
||||||
|
- native MariaDB/MySQL (either binary naming — `mail` has a
|
||||||
|
hand-installed `mariadb-dump` + mysql-named client; every
|
||||||
|
non-system DB) → `/backup/plakar-dumps/mariadb/<ct>/<db>.sql`,
|
||||||
|
plus a replayable users/grants dump (`SHOW GRANTS` per user, incl.
|
||||||
|
password hashes) → `grants.sql` — so the S3 dump path is
|
||||||
|
self-sufficient without the replicas, mirroring pg's `globals.sql`
|
||||||
|
- native PostgreSQL (`pg_dump` per DB + `pg_dumpall --globals-only`
|
||||||
|
for roles, peer auth as `postgres`) →
|
||||||
|
`/backup/plakar-dumps/postgres/<ct>/`
|
||||||
|
- PostgreSQL **inside docker** (covers `outline`/`login`): docker
|
||||||
|
containers whose image name contains "postgres", dump user from
|
||||||
|
`$POSTGRES_USER` → `/backup/plakar-dumps/postgres/<ct>/<docker-name>/`
|
||||||
|
(a pg bundled in an image *not* named `*postgres*` would be missed —
|
||||||
|
check the nightly log's `dump` lines after adding such a service)
|
||||||
|
|
||||||
|
Stopped containers (e.g. `databap`) are skipped —
|
||||||
|
their static state is covered by the leg-1 replicas.
|
||||||
|
|
||||||
|
Opt-out: `/root/scripts/plakar-db-exclude` (optional) lists dump
|
||||||
|
paths to skip, one per line, exactly as the `dump` log lines print
|
||||||
|
them (`<ct>/<db>` or `<ct>/<docker-name>/<db>`). Every skip is logged
|
||||||
|
(`SKIP <path>`), so unlike the old manifest the list can't rot
|
||||||
|
silently. Applied 2026-08-22: `livetrail/geo` (10 GiB static map
|
||||||
|
data, 3 h dump). Its last dump deliberately stays in the dump dir:
|
||||||
|
unchanged → skipped by the VFS cache, so every nightly `dumps`
|
||||||
|
snapshot keeps carrying it to S3 at zero cost. To refresh after a
|
||||||
|
re-import: drop the exclude line for one night.
|
||||||
|
|
||||||
|
Dumps are staged **plain, not gzipped**: plakar chunks and compresses
|
||||||
|
anyway, and CDC dedup across near-identical consecutive dumps only
|
||||||
|
works on uncompressed input. The dump tree (incus's own DBs +
|
||||||
|
MariaDB) is backed up as one snapshot tagged `dumps`. The nightly log
|
||||||
|
lists every `dump <ct>/<db>` line — a database disappearing from the
|
||||||
|
log is visible. (The installed mysql plugin is now unused; harmless
|
||||||
|
to keep.)
|
||||||
|
|
||||||
|
Beyond databases, `outline`/`login` may hold non-DB state (uploaded
|
||||||
|
files, docker volumes) — decide per instance whether an fs source is
|
||||||
|
needed in §4.
|
||||||
|
|
||||||
|
Findings from the first dry-run (2026-08-22):
|
||||||
|
|
||||||
|
- `livetrail`'s `/root/.my.cnf` had a stale `database = spot` default
|
||||||
|
that broke discovery — fixed 2026-08-22 (its DB is `livetrail`).
|
||||||
|
- `login` still runs a **native** postgres holding the pre-migration
|
||||||
|
`outline` DB (live Outline uses `outline-postgres-1`). Dumped
|
||||||
|
harmlessly for now; cleanup once the S3 leg is verified:
|
||||||
|
`dropdb outline` + remove the native postgres package.
|
||||||
|
|
||||||
|
## 4. Filesystem sources
|
||||||
|
|
||||||
|
Derived from the old `incus-backup.db` manifest (minus `spot`; DBs
|
||||||
|
are handled by §3's auto-discovery). FS paths are read from the live
|
||||||
|
containers' rootfs on the host — crash-consistent, same fidelity as
|
||||||
|
the old rsync (the per-file
|
||||||
|
[incus integration](plakar-incus-integration.md) supersedes this
|
||||||
|
later). Verify the pool mount prefix first:
|
||||||
|
`ls /var/lib/incus/storage-pools/data/containers/mail/rootfs/`.
|
||||||
|
|
||||||
|
`outline`/`login` non-DB state identified 2026-08-22 via
|
||||||
|
`docker inspect` (Outline's local file storage volume; authentik's
|
||||||
|
data + custom templates — `/opt/authentik` also catches the compose
|
||||||
|
file). ⚠️ Still to review: whether `livetrail` holds state.
|
||||||
|
|
||||||
|
Register sources (root on ks4):
|
||||||
|
|
||||||
|
```sh
|
||||||
|
R=/var/lib/incus/storage-pools/data/containers
|
||||||
|
plakar source add bitwarden-fs $R/bitwarden/rootfs/opt/bitwarden
|
||||||
|
plakar source add gateway-fs $R/gateway/rootfs/var/www
|
||||||
|
plakar source add git-fs $R/git/rootfs/home/git/projects
|
||||||
|
plakar source add login-fs $R/login/rootfs/opt/authentik
|
||||||
|
plakar source add mail-vmail $R/mail/rootfs/var/vmail
|
||||||
|
plakar source add mail-www $R/mail/rootfs/var/www
|
||||||
|
plakar source add nextcloud-fs $R/nextcloud/rootfs/nextcloud
|
||||||
|
plakar source add outline-fs $R/outline/rootfs/var/lib/docker/volumes/outline_storage-data/_data
|
||||||
|
plakar source add seafile-fs $R/seafile/rootfs/opt/seafile
|
||||||
|
plakar source add solar-fs $R/solar/rootfs/var/www/html/solar
|
||||||
|
```
|
||||||
|
|
||||||
|
Then list the names (one per line) in `/root/scripts/plakar-sources` —
|
||||||
|
the script iterates that file.
|
||||||
|
|
||||||
|
## 5. Script + cron
|
||||||
|
|
||||||
|
```sh
|
||||||
|
# copy scripts/plakar-backup.sh from this repo to /root/scripts/, then:
|
||||||
|
chmod +x /root/scripts/plakar-backup.sh
|
||||||
|
/root/scripts/plakar-backup.sh -k /backup/plakar-test # dry-run the pipeline now
|
||||||
|
/root/scripts/plakar-backup.sh -k @s3 # first real run (once creds exist)
|
||||||
|
plakar at @s3 ls # inspect snapshots
|
||||||
|
```
|
||||||
|
|
||||||
|
Cron (04:30, after the 04:00 stopgap so they don't hammer the same
|
||||||
|
DBs simultaneously; move to 04:00 once the rsync stopgap is dropped).
|
||||||
|
Inline `backup -check` was removed 2026-08-22: against S3 it re-reads
|
||||||
|
every written chunk (~2.6 MB/s latency-bound; gateway-fs: 7 min
|
||||||
|
backup, 23 min check — the giants would add 10–20 h nightly).
|
||||||
|
Verification is a weekly deep check of the latest snapshots instead:
|
||||||
|
|
||||||
|
```cron
|
||||||
|
30 4 * * * /root/scripts/plakar-backup.sh -k @s3 >> /var/log/plakar-backup.log 2>&1
|
||||||
|
0 12 * * 0 PLAKAR_PASSPHRASE=$(cat /root/.plakar-passphrase) plakar -quiet at @s3 check -latest >> /var/log/plakar-check.log 2>&1
|
||||||
|
```
|
||||||
|
|
||||||
|
Once validated against @s3, delete the throwaway kloset:
|
||||||
|
`rm -rf /backup/plakar-test`.
|
||||||
|
|
||||||
|
## 6. S3 hardening
|
||||||
|
|
||||||
|
Scoped credentials (no delete permission if the provider allows it),
|
||||||
|
bucket versioning or object lock, and prefer an OVH region — ks4 →
|
||||||
|
OVH Object Storage stays on the OVH backbone, so backup and restore
|
||||||
|
speed is LAN-like.
|
||||||
|
|
||||||
|
## 7. Restore test (release gate)
|
||||||
|
|
||||||
|
Two restores from `@s3`, exercising both data types:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
export PLAKAR_PASSPHRASE=$(cat /root/.plakar-passphrase)
|
||||||
|
plakar at @s3 ls # expect 11 snapshots: dumps + 10 fs sources
|
||||||
|
|
||||||
|
# (a) file tree — restore solar-fs and diff against the live tree
|
||||||
|
plakar at @s3 restore -to /tmp/restore-test <solar-fs-snap-id>
|
||||||
|
diff -r /tmp/restore-test/var/lib/incus/storage-pools/data/containers/solar/rootfs/var/www/html/solar \
|
||||||
|
/var/lib/incus/storage-pools/data/containers/solar/rootfs/var/www/html/solar
|
||||||
|
rm -rf /tmp/restore-test
|
||||||
|
|
||||||
|
# (b) database — restore a dump and load it into a scratch container
|
||||||
|
plakar at @s3 restore -to /tmp/restore-db <dumps-snap-id>
|
||||||
|
incus launch images:ubuntu/24.04 dbtest
|
||||||
|
incus exec dbtest -- apt-get install -y mariadb-server
|
||||||
|
incus file push /tmp/restore-db/backup/plakar-dumps/mariadb/freshrss/freshrss.sql dbtest/root/
|
||||||
|
incus exec dbtest -- sh -c 'mariadb < /root/freshrss.sql' # dump embeds CREATE DATABASE
|
||||||
|
incus exec dbtest -- mariadb -e 'SELECT COUNT(*) FROM information_schema.tables WHERE table_schema="freshrss"'
|
||||||
|
incus delete -f dbtest && rm -rf /tmp/restore-db
|
||||||
|
```
|
||||||
|
|
||||||
|
Pass = diff empty (modulo files changed since the snapshot) and the
|
||||||
|
scratch DB loads with a sane table count.
|
||||||
|
|
||||||
|
## Retirement
|
||||||
|
|
||||||
|
Once this leg is verified (and once more from @s3): remove the 04:00
|
||||||
|
`incus-backup.sh` cron to ks2 — precondition for
|
||||||
|
[decommissioning ks2](../ks2/plan.md).
|
||||||
Reference in New Issue
Block a user