Document the ks2 decommission plan and the new ks4 backup legs

- ks2/: decommission plan with release gates (rental ends Sep 30),
  nuc-seed and final-decommission runbooks
- ks4/local-backup-cron.md: leg-1 cron re-enabled, ks2 rsync stopgap,
  full-resend caveat when refreshes lapse past snapshots.expiry
- ks4/plakar-s3-data.md: plakar direct-to-S3 leg (auto-discovered DB
  dumps incl. grants/globals, fs sources, exclude list, restore test)
- ks4/plakar-incus-integration.md: design + scaffold status of the
  plakar incus importer
- README.md: index ks2/, update the ks4 durability bullet

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Julien Lutran
2026-08-23 08:56:28 +02:00
co-authored by Claude Fable 5
parent 4b30b43c7f
commit 79ad1b7179
7 changed files with 597 additions and 4 deletions
+56
View File
@@ -0,0 +1,56 @@
# local backup cron on ks4 (leg 1) — re-enabled
Done 2026-08-22. Context: **both** ks4 backup crons had been commented
out — no backup of any kind was running.
## What runs now (root crontab on ks4)
```cron
0 1 * * * /root/scripts/incus-copy.sh -p backup -s backup >> /var/log/incus-copy.log 2>&1
0 4 * * * /root/scripts/incus-backup.sh -d 164.132.173.57 -u rsyncbackup -i /root/.ssh/id_rsyncbackup -p 2233 -f /root/scripts/incus-backup.db -s data >> /var/log/incus-backup.log 2>&1
```
- `01:00`**local leg**: `incus copy --refresh` of all instances
into project `backup` on the sdb5 `backup` zpool (replaces the old
push to ks2; see [incus-copy.md](incus-copy.md)).
- `04:00`**stopgap**: `incus-backup.sh` rsync to ks2 kept alive
(manifest pruned of `spot`) until the plakar S3 leg replaces it —
then this line goes away as part of
[decommissioning ks2](../ks2/plan.md). ⚠️ keep
`scripts/incus-backup.db` in this repo in sync with
`/root/scripts/incus-backup.db` on ks4.
Both were also run once by hand (in a `screen`) on 2026-08-22 to
refresh the backups immediately. The old push-to-ks2 copy line stays
commented out — replica refreshes to ks2 are retired.
## Cleanup done the same day
- ks2: `incus delete biwiki spot` (abandoned instances — freed space
on the 95 %-full `data` pool)
- ks4: `incus delete spot --project backup`
## Verification
```bash
# all instances present, incl. livetrail (first copy is a full send)
sudo incus list --project backup -c ns -f csv
# cron runs land here (manual screen runs don't):
sudo tail /var/log/incus-copy.log /var/log/incus-backup.log
# rsync leg freshness, on ks2:
ls -lat /backup/ns3061243/ | head
```
Notes:
- `flock` in both scripts makes overlapping cron/manual runs abort
safely instead of racing.
- Replicas get `boot.autostart=false` set by the script — they must
never fight the live instances for their static IPs.
- Keep `snapshots.schedule` set on the sources so refreshes stay
ZFS-incremental (a refresh without common snapshots falls back to
rsync). Corollary: the cron must run at least every
`snapshots.expiry` (7 d) — a longer gap expires every common
snapshot and the next refresh silently degrades to a **full
re-send** (observed 2026-08-22: seafile re-sent 933 G after the
weeks-long gap while the crons were commented out).