doc: plakar out of the current architecture, kept as reference
backup-strategy.md now leads with the three target legs (local incus push to sdb, nuc pull over WireGuard, restic to S3 for DBs and selected trees) and carries the plakar/ks2 story in a closing History section. restic-backup.md, install.md, local-backup-cron.md and the ks2 docs describe the predecessor only as 'the first S3 implementation', with one operational note that plakar is still installed for issue #2338 reproduction. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Julien Lutran
co-authored by
Claude Fable 5
Claude Fable 5
parent
b5aef914f5
commit
d7ec780747
+34
-15
@@ -4,25 +4,29 @@ Audience: anyone with root on ks4 who needs to know **what is protected,
|
||||
where, by which tool, and how to check it**. Details live in the linked
|
||||
docs; this page is the map.
|
||||
|
||||
## Two tools, no more
|
||||
## Two tools, three legs
|
||||
|
||||
| tool | job | why this one |
|
||||
| leg | mechanism | protects |
|
||||
|---|---|---|
|
||||
| **`incus copy --refresh`** | replicate whole instances (ready-to-start copies) | ZFS-incremental, native to the platform, restores are `incus start` |
|
||||
| **restic** | off-site backups to S3 (files, database dumps, instance trees) | dedup + encryption, local metadata cache → a night costs only the churn ([restic-backup.md](ks4/restic-backup.md)) |
|
||||
| **local replication** | `incus copy --refresh` → pool `backup` on ks4's second disk (sdb) | instances, against losing the `data` pool |
|
||||
| **remote replication** | nuc pulls the same replicas over WireGuard → pool `ks4backup` | instances, against losing ks4 or the site |
|
||||
| **remote backup** | **restic** → S3 bucket `restic-data`: database dumps + selected filesystem trees | the data itself, versioned and encrypted, independent of every disk above |
|
||||
|
||||
plakar was evaluated first and replaced (its nightly cost scaled with
|
||||
tree size — [PlakarKorp/plakar#2338](https://github.com/PlakarKorp/plakar/issues/2338));
|
||||
the old backup server **ks2 is being retired** (decommission by
|
||||
Sep 30, 2026 — [ks2/plan.md](ks2/plan.md)): it still holds an ageing
|
||||
copy of the instances until the nuc leg below takes over.
|
||||
Only two tools are involved: `incus copy` (ZFS-incremental, native to
|
||||
the platform — a restore is `incus start`) and restic (dedup +
|
||||
encryption, local metadata cache, so a night costs only the churn —
|
||||
[ks4/restic-backup.md](ks4/restic-backup.md)).
|
||||
|
||||
**Status 2026-08-28**: the local replica leg and the S3 data leg are
|
||||
live. The **nuc pull leg is waiting for FTTH** (expected before end of
|
||||
September); until it lands, the ks2 push is re-enabled as an interim
|
||||
off-site copy of the instances. Backing up instance *images* to S3 was
|
||||
designed and shelved — S3 holds the data, the databases and the incus
|
||||
configuration, which is what a rebuild needs.
|
||||
The old backup server **ks2 is being retired** (decommission by
|
||||
Sep 30, 2026 — [ks2/plan.md](ks2/plan.md)); until FTTH enables the nuc
|
||||
leg it still receives an interim replica push.
|
||||
|
||||
**Status 2026-08-28**: local replication and the S3 backup leg are
|
||||
live. The **nuc pull leg waits for FTTH** (expected before end of
|
||||
September); until then the ks2 push stands in. Backing up whole
|
||||
instance *images* to S3 was considered and left out — S3 holds the
|
||||
data, the databases and the incus configuration, which is what a
|
||||
rebuild needs.
|
||||
|
||||
## The map
|
||||
|
||||
@@ -120,3 +124,18 @@ New instances are picked up automatically by the replica and
|
||||
instance legs (all instances, opt-out only). Databases are
|
||||
auto-discovered. Only **data trees** need one line in
|
||||
[`restic-paths`](https://git.lutran.fr/julien/scripts/src/branch/main/restic-paths) — see [new-container.md](new-container.md).
|
||||
|
||||
## History (reference only)
|
||||
|
||||
The road to the setup above, kept for the measurements rather than for
|
||||
operations: an rsync-based leg to ks2
|
||||
([ks2/plan.md](ks2/plan.md)) and a first S3 implementation with
|
||||
**plakar** ([ks4/plakar-s3-data.md](ks4/plakar-s3-data.md), plus a
|
||||
self-written incus connector,
|
||||
[ks4/plakar-incus-integration.md](ks4/plakar-incus-integration.md)).
|
||||
plakar was dropped because its nightly cost scaled with the size of
|
||||
the tree rather than with the churn
|
||||
([PlakarKorp/plakar#2338](https://github.com/PlakarKorp/plakar/issues/2338)).
|
||||
It is still installed on ks4 and its `plakar-data` bucket still exists
|
||||
— kept only so the upstream issue can be reproduced; nothing schedules
|
||||
it any more.
|
||||
|
||||
+4
-4
@@ -55,8 +55,8 @@ renewal date.
|
||||
- [ ] Retire `scripts/incus-backup.sh` + `incus-backup.db` in the repo
|
||||
(superseded by [`restic-backup.sh`](https://git.lutran.fr/julien/scripts/src/branch/main/restic-backup.sh)) —
|
||||
keep for reference, note the retirement in their headers
|
||||
- [ ] `login` container cleanup: the native postgres still holds the
|
||||
pre-migration `outline` DB — `dropdb outline`, remove the
|
||||
package (noted in
|
||||
[plakar-s3-data.md](../ks4/plakar-s3-data.md) §3)
|
||||
- [ ] `login` container cleanup: its **native** postgres still holds
|
||||
the pre-migration `outline` database (live Outline uses the
|
||||
dockerised `outline-postgres-1`) — `dropdb outline` and remove
|
||||
the native postgres package
|
||||
- [ ] Mark the ks2 line in `plan.md` done, date it
|
||||
|
||||
+3
-3
@@ -61,9 +61,9 @@ when nuc takes over.
|
||||
2. ~~[local-backup-cron.md](../ks4/local-backup-cron.md)~~ — **done
|
||||
2026-08-22**: leg 1 cron re-enabled, ks2 rsync kept as stopgap;
|
||||
`livetrail` verified present in the `backup` project
|
||||
3. ~~plakar S3 leg~~ → **replaced by restic**
|
||||
([restic-backup.md](../ks4/restic-backup.md), live 2026-08-28;
|
||||
plakar's own doc kept as history)
|
||||
3. ~~first S3 leg~~ → **restic** is the S3 tool
|
||||
([restic-backup.md](../ks4/restic-backup.md), live 2026-08-28; the
|
||||
predecessor's doc is kept as reference)
|
||||
4. ~~instance leg to S3~~ — **shelved 2026-08-28**: instances are
|
||||
protected by replication (sdb + nuc/ks2), their data and configs by
|
||||
`restic-data`
|
||||
|
||||
+4
-3
@@ -164,9 +164,10 @@ non-mirrored `data` pool:
|
||||
into the `restic-data` bucket ([restic-backup.md](restic-backup.md)).
|
||||
- **Sun 14:00 `restic-maintenance.sh`** → prune + integrity checks.
|
||||
|
||||
Historical (retired 2026-08): the `ks2` push and `incus-backup.sh`
|
||||
rsync leg, and the plakar data leg
|
||||
([plakar-s3-data.md](plakar-s3-data.md)).
|
||||
A `ks2` interim replica push (`incus-copy.sh -d ks2 -m push`, 02:00)
|
||||
runs until the nuc pull leg is seeded. Retired 2026-08: the
|
||||
`incus-backup.sh` rsync leg and the first S3 implementation
|
||||
([plakar-s3-data.md](plakar-s3-data.md) — reference only).
|
||||
|
||||
- **01:00 — `incus-copy.sh` → `ks2`**: differential `incus copy … \
|
||||
--refresh --refresh-exclude-older --mode push` of every *running*
|
||||
|
||||
@@ -15,9 +15,9 @@ backup of any kind was running). Overview of all legs:
|
||||
push to ks2; see [incus-copy.md](incus-copy.md)). An S3 leg for the
|
||||
instance filesystems was designed and shelved
|
||||
([restic-backup.md](restic-backup.md) §6).
|
||||
- The rsync stopgap to ks2 (`incus-backup.sh`, 04:00) and the plakar
|
||||
data leg (04:30) are both **retired**; S3 backups are restic's job
|
||||
at 05:00.
|
||||
- The rsync stopgap to ks2 (`incus-backup.sh`, 04:00) and the first
|
||||
S3 implementation (04:30) are both **retired**; S3 backups are
|
||||
restic's job at 05:00 ([restic-backup.md](restic-backup.md)).
|
||||
|
||||
## Cleanup done the same day
|
||||
|
||||
|
||||
+30
-36
@@ -1,25 +1,18 @@
|
||||
# restic backups on ks4 — the S3 leg
|
||||
|
||||
Status: **live since 2026-08-28** — restic replaced plakar as the S3
|
||||
backup tool (plakar crons off, its kloset kept for
|
||||
[#2338](https://github.com/PlakarKorp/plakar/issues/2338); see §8).
|
||||
Architecture: **two tools** — `incus copy` for replication (local sdb,
|
||||
nuc after FTTH), **restic** for everything S3. Data leg (`restic-data`)
|
||||
runs nightly at 05:00. The instance leg (`restic-incus`, §6) is
|
||||
**shelved** (2026-08-28): instances are covered by replication, so
|
||||
duplicating their filesystems into S3 buys little. The script and the
|
||||
(empty) repo stay in place should that change.
|
||||
Status: **live since 2026-08-28.** This is the S3 backup leg of the
|
||||
[strategy](../backup-strategy.md): database dumps + selected
|
||||
filesystem trees into the `restic-data` bucket, nightly at 05:00.
|
||||
Instances themselves are handled by replication (sdb, nuc after FTTH),
|
||||
not by this leg — an instance-filesystem leg (`restic-incus`, §6) was
|
||||
written and **shelved**; its script and empty repo stay in place
|
||||
should that change.
|
||||
|
||||
Why: plakar's incremental cost scales with tree size, not churn
|
||||
(~800 MiB parent metadata re-read from S3 + hours of CPU per night on
|
||||
a 500k-file tree — measured, reported as
|
||||
[PlakarKorp/plakar#2338](https://github.com/PlakarKorp/plakar/issues/2338)).
|
||||
restic keeps its comparison state in a local cache: a night costs a
|
||||
local stat-walk plus the churn — the rsync cost model. Neither restic
|
||||
nor rustic has plakar's source-plugin architecture, but none is
|
||||
needed: instances are reached via `incus file mount` (FUSE over the
|
||||
same per-instance sftp API our plakar integration used) — zero
|
||||
patches to maintain (see §6).
|
||||
Why restic: its comparison state lives in a **local cache**, so a
|
||||
night costs a local stat-walk plus the churn — the rsync cost model,
|
||||
with dedup, compression and encryption on top. (The predecessor and
|
||||
the measurements that motivated the switch are in
|
||||
[plakar-s3-data.md](plakar-s3-data.md) — reference only.)
|
||||
|
||||
## 1. Repositories — two buckets, two drivers (decided 2026-08-25)
|
||||
|
||||
@@ -28,7 +21,7 @@ patches to maintain (see §6).
|
||||
| `restic-data` | SQL dumps + **all** fs sources (8 small + nextcloud + seafile) | `restic-backup.sh` |
|
||||
| `restic-incus` | *(shelved — empty repo, see §6)* the `backup`-project replicas, per-file via mount | `restic-incus-backup.sh` |
|
||||
|
||||
Mirrors the plakar-era split (data leg / incus leg). Accepted
|
||||
Two repos so each can be pruned and checked independently. Accepted
|
||||
trade-off: `restic-data` carries the giants' blobs (~3–5 M once
|
||||
seafile is in), so **every operation on it loads a ~1.5–2 GiB
|
||||
index** — mitigated by running the whole nightly as a *single*
|
||||
@@ -168,9 +161,9 @@ done
|
||||
|
||||
## 5. Nightly driver — `scripts/restic-backup.sh`
|
||||
|
||||
Same shape as the plakar drivers (flock, env, loud logging). Phases:
|
||||
Shape: flock, env file, loud logging. Phases:
|
||||
|
||||
1. **dumps** — lifted verbatim from `plakar-backup.sh`: incus DB
|
||||
1. **dumps** — incus DB
|
||||
dumps + auto-discovered MariaDB/PostgreSQL (native and docker)
|
||||
dumps into the dump dir (during the parallel week restic dumps
|
||||
into its own `/backup/dumps`; carry the archived `geo.sql` over
|
||||
@@ -227,7 +220,7 @@ byte once a year, ~25 GiB read per week.
|
||||
Neither restic nor rustic accepts source plugins, and forking the
|
||||
backup tool is the wrong place to carry a patch. Instead, incus
|
||||
exposes any instance (including **stopped replicas**) as a FUSE mount
|
||||
over the same per-instance sftp API the plakar integration used:
|
||||
over the per-instance sftp API:
|
||||
|
||||
```sh
|
||||
for inst in $(incus list --project backup -c n -f csv); do
|
||||
@@ -252,8 +245,7 @@ opt-out via a variable — the no-manifest-drift rule survives the tool
|
||||
change. Chained after the 01:00 incus-copy once validated, same
|
||||
freshness reasoning as before.
|
||||
|
||||
Bonus over plakar: one shared repo dedups the Ubuntu base across all
|
||||
16 replicas. Restore granularity: per-file; instance definitions ride
|
||||
One shared repo would dedup the Ubuntu base across all replicas. Restore granularity: per-file; instance definitions ride
|
||||
the dump tree (`incus/<inst>.yaml` + `incus-global-db.sql`).
|
||||
|
||||
## 7. Restore test (gate)
|
||||
@@ -267,24 +259,26 @@ diff -r … && rm -rf /backup/restore-test
|
||||
|
||||
## 8. Cutover (done) + what is left
|
||||
|
||||
- **Cutover 2026-08-28**: plakar's 04:30 and Sunday crons are off;
|
||||
restic's 05:00 data leg is the S3 backup. The formal parallel-week
|
||||
Legacy on the box: plakar is still installed and its `plakar-data`
|
||||
bucket still exists, kept only to reproduce
|
||||
[#2338](https://github.com/PlakarKorp/plakar/issues/2338) if upstream
|
||||
asks; no cron references it. The empty `plakar-incus` bucket can be
|
||||
deleted.
|
||||
|
||||
- **Cutover 2026-08-28**: the previous S3 leg's crons are off and
|
||||
restic's 05:00 run is the S3 backup. The formal parallel-week
|
||||
comparison was cut short — the evidence had already settled it
|
||||
(plakar: ~800 MiB metadata re-read and 3–4 h per night on the
|
||||
(predecessor: ~800 MiB metadata re-read and 3–4 h per night on the
|
||||
nextcloud tree, churn-independent; restic: seed at 60–115 MiB/s,
|
||||
local-cache incrementals). plakar's measurements stay in
|
||||
local-cache incrementals). Details in
|
||||
[plakar-s3-data.md](plakar-s3-data.md).
|
||||
- **Left to do**: restore test on the complete `restic-data`; re-enable
|
||||
the interim ks2 push until FTTH; then the ks2 decommission gates
|
||||
([../ks2/plan.md](../ks2/plan.md)). The instance leg (§6) is
|
||||
shelved, not pending.
|
||||
- **plakar retirement, one exception**: keep plakar installed and
|
||||
keep the `plakar-data` kloset until
|
||||
[#2338](https://github.com/PlakarKorp/plakar/issues/2338) concludes
|
||||
— the issue offers debug runs against that repository. Delete the
|
||||
`plakar-incus` bucket immediately; `plakar-data` a few weeks later.
|
||||
The `integration-incus` repo retires from production; optionally
|
||||
still publish to PlakarKorp/hub as a community contribution.
|
||||
- **Old tool**: see the legacy note above; the `integration-incus`
|
||||
repo retires from production (optionally still published to
|
||||
PlakarKorp/hub as a community contribution).
|
||||
- **Interim ks2 push** (until the nuc leg seeds after FTTH, expected
|
||||
before end of September): re-enable the replica push so instances
|
||||
keep an off-site copy meanwhile:
|
||||
|
||||
Reference in New Issue
Block a user