Document the ks2 decommission plan and the new ks4 backup legs
- ks2/: decommission plan with release gates (rental ends Sep 30), nuc-seed and final-decommission runbooks - ks4/local-backup-cron.md: leg-1 cron re-enabled, ks2 rsync stopgap, full-resend caveat when refreshes lapse past snapshots.expiry - ks4/plakar-s3-data.md: plakar direct-to-S3 leg (auto-discovered DB dumps incl. grants/globals, fs sources, exclude list, restore test) - ks4/plakar-incus-integration.md: design + scaffold status of the plakar incus importer - README.md: index ks2/, update the ks4 durability bullet Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Julien Lutran
co-authored by
Claude Fable 5
Claude Fable 5
parent
4b30b43c7f
commit
79ad1b7179
@@ -0,0 +1,60 @@
|
||||
# ks2 decommission — final checklist
|
||||
|
||||
Status: **prepared — execute only when every gate in
|
||||
[plan.md](plan.md) is ticked.** Rental ends **Sep 30, 2026**; leave a
|
||||
few days of margin.
|
||||
|
||||
## Preconditions (the plan's release gates)
|
||||
|
||||
- [ ] plakar S3 leg running nightly for ≥ a week, `done (rc=0)`,
|
||||
restore test passed ([plakar-s3-data.md](../ks4/plakar-s3-data.md) §7)
|
||||
- [ ] nuc pull leg seeded and one instance test-restored
|
||||
([nuc-seed.md](nuc-seed.md))
|
||||
- [ ] local leg (sdb5) cron green in `/var/log/incus-copy.log`
|
||||
|
||||
## 1. Cut the last flows to ks2 (root on ks4)
|
||||
|
||||
```sh
|
||||
crontab -e # delete the 04:00 incus-backup.sh line (164.132.173.57)
|
||||
incus remote list # if a "ks2" remote is still defined:
|
||||
incus remote remove ks2
|
||||
# drop the rsyncbackup SSH key if it exists only for ks2:
|
||||
ls /root/.ssh/id_rsyncbackup* && rm -i /root/.ssh/id_rsyncbackup*
|
||||
```
|
||||
|
||||
## 2. Last look at ks2 before wiping (root on ks2)
|
||||
|
||||
```sh
|
||||
# anything in the rsync tree newer than the plakar cutover?
|
||||
find /backup/ns3061243 -newermt '2026-08-23' | head
|
||||
# pre-2023 dirs (catc, mythoughts, qcm, scripts): instances still
|
||||
# exist stopped on ks4 — spot-check nothing unique remains:
|
||||
ls /backup/ns3061243/{catc,mythoughts,qcm,scripts}
|
||||
```
|
||||
|
||||
Archive anything unique to the plakar dump dir on ks4 (it rides the
|
||||
nightly `dumps` snapshot to S3 for free once it stops changing).
|
||||
|
||||
## 3. Wipe and terminate
|
||||
|
||||
```sh
|
||||
# on ks2 — destroy the data (both pools hold copies of personal data):
|
||||
zpool destroy backup && zpool destroy data
|
||||
# optional: shred the OS disks via OVH rescue mode / reinstall
|
||||
```
|
||||
|
||||
Then in the OVH manager: cancel the ks2 (Kimsufi) server before the
|
||||
renewal date.
|
||||
|
||||
## 4. Post-mortem cleanup
|
||||
|
||||
- [ ] Update [doc/README.md](../README.md): remove ks2 from the ks4
|
||||
durability bullet and the index line for `ks2/`s "being
|
||||
decommissioned" wording (keep the folder as history)
|
||||
- [ ] Retire `scripts/incus-backup.sh` + `incus-backup.db` in the repo
|
||||
(superseded by `scripts/plakar-backup.sh`) — keep for reference,
|
||||
note the retirement in their headers
|
||||
- [ ] `login` container cleanup parked in
|
||||
[plakar-s3-data.md](../ks4/plakar-s3-data.md): `dropdb outline`,
|
||||
remove native postgres
|
||||
- [ ] Mark the ks2 line in `plan.md` done, date it
|
||||
@@ -0,0 +1,49 @@
|
||||
# nuc pull leg — seed after FTTH
|
||||
|
||||
Status: **prepared, waiting on the FTTH link.** Everything is already
|
||||
configured on nuc (see the main [README](../README.md)): incus remote
|
||||
`ks4` over the WireGuard tunnel (`wg-ks4`, 10.8.0.20 → 10.8.0.1),
|
||||
target pool `ks4backup` on the USB ZFS pool (`usb4t/backup/ks4`) —
|
||||
only the seed itself waited on bandwidth.
|
||||
|
||||
## Seed (root on nuc)
|
||||
|
||||
```sh
|
||||
# sanity: remote reachable through the tunnel
|
||||
incus list ks4: | head
|
||||
|
||||
# full pull of every ks4 instance into pool ks4backup (screen/tmux —
|
||||
# first pass moves ~1.7 T through the WG tunnel)
|
||||
/root/scripts/incus-copy.sh -r ks4 -s ks4backup 2>&1 | tee -a /var/log/incus-copy-ks4.log
|
||||
```
|
||||
|
||||
Notes:
|
||||
|
||||
- First pass is a full send per instance; later refreshes are
|
||||
ZFS-incremental **as long as they run at least every
|
||||
`snapshots.expiry` (7 d on ks4)** — same caveat as
|
||||
[ks4's local leg](../ks4/local-backup-cron.md).
|
||||
- Replicas arrive stopped with `boot.autostart=false` (script does
|
||||
this) — they must never come up on the LAN with ks4's proxy devices.
|
||||
|
||||
## Cron (after the seed)
|
||||
|
||||
Add to nuc's root crontab, offset from the 03:30 local nucbackup copy
|
||||
and ks4's own 01:00/04:30 jobs:
|
||||
|
||||
```cron
|
||||
0 5 * * * /root/scripts/incus-copy.sh -r ks4 -s ks4backup >> /var/log/incus-copy-ks4.log 2>&1
|
||||
```
|
||||
|
||||
## Verification (release gate for ks2)
|
||||
|
||||
```sh
|
||||
incus list --project backup 2>/dev/null || incus list | grep -c . # all ks4 instances present
|
||||
# test-restore one instance: copy a replica to the default pool,
|
||||
# start it isolated, check the service answers, then delete it
|
||||
incus copy solar solar-restoretest -s default
|
||||
incus start solar-restoretest && incus exec solar-restoretest -- systemctl is-system-running
|
||||
incus delete -f solar-restoretest
|
||||
```
|
||||
|
||||
Once verified, tick the nuc gate in the [ks2 plan](plan.md).
|
||||
+78
@@ -0,0 +1,78 @@
|
||||
# ks2 decommission — summary plan
|
||||
|
||||
Goal: release `ks2.lutran.fr` (`164.132.173.57`, SSH :2233, host
|
||||
`ns3247221`) once ks4 has a real 3-2-1 backup without it.
|
||||
**Deadline: rental ends Sep 30, 2026.**
|
||||
|
||||
## What ks2 does today (updated 2026-08-22)
|
||||
|
||||
One remaining role, fed nightly by a root cron **on ks4**:
|
||||
|
||||
- `04:00` `incus-backup.sh … -d 164.132.173.57` — DB dumps + selected
|
||||
paths rsync'd to ks2 `/backup/ns3061243` on pool `backup` (1.59 T,
|
||||
88 % full), snapshotted daily on ks2 (`zfs-auto-snapshot.sh`,
|
||||
2-month expiry). **Kept as a stopgap** until the plakar S3 leg
|
||||
replaces it.
|
||||
|
||||
Retired: the `01:00` replica push to ks2 pool `data` — replaced by
|
||||
the local leg on ks4's sdb5 pool
|
||||
([local-backup-cron.md](../ks4/local-backup-cron.md)); `biwiki` and `spot`
|
||||
deleted from ks2.
|
||||
|
||||
## Inventory findings (2026-08-22)
|
||||
|
||||
- `biwiki` exists only on ks2, `spot` only on ks2 + ks4's `backup`
|
||||
project — **decision: both abandoned**, safe to delete from backups.
|
||||
- `incus-backup.db` is stale (lists `spot`, misses `livetrail`,
|
||||
`outline`, `login`, …) — forgotten-manifest drift is exactly what
|
||||
the plakar-based replacement must eliminate (back up *all*
|
||||
instances by default, opt-out instead of opt-in).
|
||||
- `livetrail` (running on ks4) is **absent from ks4's local `backup`
|
||||
project** — the local leg was run manually once and never again;
|
||||
automating its cron fixes this.
|
||||
- Both ks4 backup crons were **commented out — no ks4 backup ran at
|
||||
all** (hence the stale ks2 rsync data: dirs `Jul 23`, incus dumps
|
||||
`Aug 9`). Fixed 2026-08-22 — see
|
||||
[local-backup-cron.md](../ks4/local-backup-cron.md).
|
||||
- `/backup/ns3061243` also holds pre-2023 dirs (`catc`, `mythoughts`,
|
||||
`qcm`) — those instances still exist stopped on ks4, so nothing
|
||||
unique expected there; spot-check before wiping.
|
||||
- Losing ks2 also loses its 2 months of rsync-snapshot history —
|
||||
acceptable once plakar has built equivalent retention.
|
||||
|
||||
## Target architecture (3-2-1 for ks4)
|
||||
|
||||
| Leg | Mechanism | Status |
|
||||
|---|---|---|
|
||||
| local, 2nd disk | `incus-copy.sh -p backup -s backup` → sdb5 pool | works, **cron missing** |
|
||||
| off-site, nuc | nuc pulls `ks4:*` → pool `ks4backup` over WG | waiting FTTH seed |
|
||||
| off-site, S3 (data+DB) | plakar on ks4 → S3 (replaces `incus-backup.sh`) | to do |
|
||||
| off-site, S3 (instances) | plakar **incus integration** (to develop) → S3 | to do |
|
||||
|
||||
## Actions (backup work documented in [`../ks4/`](../ks4/), ks2-only tasks here)
|
||||
|
||||
1. ~~salvage biwiki/spot~~ — decided 2026-08-22: both abandoned,
|
||||
**deleted** from ks2 and ks4's `backup` project
|
||||
2. ~~[local-backup-cron.md](../ks4/local-backup-cron.md)~~ — **done
|
||||
2026-08-22**: leg 1 cron re-enabled, ks2 rsync kept as stopgap;
|
||||
`livetrail` verified present in the `backup` project
|
||||
3. [plakar-s3-data.md](../ks4/plakar-s3-data.md) — **seeding S3
|
||||
2026-08-22** (direct-to-S3 kloset `lutran-ks4-plakar-data`, driver
|
||||
script `scripts/plakar-backup.sh`, pipeline validated locally);
|
||||
next: restore test + cron
|
||||
4. [plakar-incus-integration.md](../ks4/plakar-incus-integration.md) —
|
||||
design seeded 2026-08-22: per-file importer over the incus sftp
|
||||
API, instances → S3
|
||||
5. [nuc-seed.md](nuc-seed.md) — **prepared**; after FTTH: seed nuc
|
||||
pull leg, verify all instances, test-restore one
|
||||
6. [decommission.md](decommission.md) — **prepared**; cut flows,
|
||||
final diff of `/backup/ns3061243`, wipe pools, terminate at OVH
|
||||
|
||||
## Release gates — ks2 can be dropped only when
|
||||
|
||||
- [x] biwiki + spot consciously abandoned (2026-08-22)
|
||||
- [ ] local leg cron running ≥ a few days, all instances present
|
||||
- [ ] nuc leg fully seeded **and** one instance test-restored
|
||||
- [ ] plakar S3 data/DB backups running daily **and** one DB + one
|
||||
file tree test-restored
|
||||
- [ ] ks4 crons pointing at ks2 disabled
|
||||
Reference in New Issue
Block a user