diff --git a/backup-strategy.md b/backup-strategy.md index d7d0688..592174f 100644 --- a/backup-strategy.md +++ b/backup-strategy.md @@ -9,7 +9,7 @@ docs; this page is the map. | leg | mechanism | protects | |---|---|---| | **local replication** | `incus copy --refresh` → pool `backup` on ks4's second disk (sdb) | instances, against losing the `data` pool | -| **remote replication** | nuc pulls the same replicas over WireGuard → pool `ks4backup` | instances, against losing ks4 or the site | +| **remote replication** | **nas** pulls the same replicas over WireGuard → pool `ks4backup` (on `tank`, direct SATA) | instances, against losing ks4 or the site | | **remote backup** | **restic** → S3 bucket `restic-data`: database dumps + selected filesystem trees | the data itself, versioned and encrypted, independent of every disk above (a nightly run walks 1.1 M files / 1.24 TiB in under two minutes) | Only two tools are involved: `incus copy` (ZFS-incremental, native to @@ -18,15 +18,22 @@ encryption, local metadata cache, so a night costs only the churn — [ks4/restic-backup.md](ks4/restic-backup.md)). The old backup server **ks2 is being retired** (decommission by -Sep 30, 2026 — [ks2/plan.md](ks2/plan.md)); until FTTH enables the nuc -leg it still receives an interim replica push. +Sep 30, 2026 — [ks2/plan.md](ks2/plan.md)); nothing is written to it +any more. -**Status 2026-08-28**: local replication and the S3 backup leg are +**Status 2026-08-31**: local replication and the S3 backup leg are live, and the S3 leg has been **restore-tested** (a file tree came back identical to the live one; a database dump loaded into a scratch -server with all its tables). The **nuc pull leg waits for FTTH** -(expected before end of September); until then the ks2 push stands -in. Backing up whole +server with all its tables). The **nas pull leg waits for FTTH** +(expected before end of September) — it moved off nuc on 2026-08-30, +onto a host where the 4 TB disk is on direct SATA rather than a USB +bridge that suspended the pool weekly +([nas/README.md](nas/README.md), [ks2/nas-seed.md](ks2/nas-seed.md)). +In the meantime instances have no *fresh* off-site copy: the ks2 push +was deliberately not re-enabled (a 3-week-old replica set on a +94 %-full pool that is about to be wiped), so off-site protection +rests on `restic-data`, which holds the data, the databases and the +incus configuration needed to rebuild. Backing up whole instance *images* to S3 was considered and left out — S3 holds the data, the databases and the incus configuration, which is what a rebuild needs. @@ -39,10 +46,10 @@ rebuild needs. │ live instances pool `data` (sda) │ │ nextcloud, seafile, mail, git, ... │ │ │ │ - │ │ 01:00 incus copy --refresh │ nuc (home LAN, via WireGuard) + │ │ 01:00 incus copy --refresh │ nas (home LAN, via WireGuard) │ ▼ │ ┌───────────────────────────┐ │ replicas (stopped) pool `backup` (sdb) │────▶│ 05:00 incus copy (pull) │ - │ project `backup` ready to start │ │ pool `ks4backup` (USB) │ + │ project `backup` ready to start │ │ ks4backup on `tank` (SATA)│ │ │ └───────────────────────────┘ │ ──────────────────────────────────────── │ │ │ OVH Object Storage (S3, sbg) @@ -59,7 +66,7 @@ rebuild needs. Two different kinds of protection, on purpose: - **instances** (the running systems) are protected by *replication* — - a ready-to-start copy on ks4's second disk and, after FTTH, on nuc. + a ready-to-start copy on ks4's second disk and, after FTTH, on nas. Restoring one is `incus copy` + `incus start`. - **the data inside them** (files, databases, incus configuration) is protected by *backup* — encrypted, deduplicated, versioned on S3, @@ -77,10 +84,9 @@ instances two homes. | when | what | log | |---|---|---| | 01:00 daily | `incus-copy.sh -p backup -s backup` — refresh all replicas onto sdb | `/var/log/incus-copy.log` | -| 02:00 daily | `incus-copy.sh -d ks2 -m push` — interim off-site replicas, until the nuc leg replaces it | `/var/log/incus-copy.log` | | 03:00 daily | instance snapshots (incus profile, 7-day expiry) — what keeps the refreshes incremental | `incus info ` | | 05:00 daily | `restic-backup.sh` — dumps + data trees → S3 | `/var/log/restic-backup.log` | -| 05:00 daily (nuc) | nuc pulls all replicas over WireGuard | nuc: `/var/log/incus-copy-ks4.log` | +| 05:00 daily (**nas**) | nas pulls all ks4 replicas over WireGuard → `ks4backup` (after FTTH) | nas: `/var/log/incus-copy-ks4.log` | | Sun 14:00 | `restic-maintenance.sh` — prune (capped), structure check, 1/52 data verification | `/var/log/restic-maintenance.log` | | 1st of month 06:00 / quarterly 15th | seafile GC dry-run report / real GC ([ks4/seafile-gc.md](ks4/seafile-gc.md)) | `/var/log/seafile-gc.log` | @@ -101,8 +107,8 @@ never silently. - **A whole instance, fast (same box)**: `incus copy backup:` style — copy the replica from project `backup` back into `default` - ([ks4/incus-copy.md](ks4/incus-copy.md)); from nuc the same via the - remote. + ([ks4/incus-copy.md](ks4/incus-copy.md)); from nas the same via the + incus remote. - **A file or directory** (any date within retention): `restic -r s3:s3.sbg.io.cloud.ovh.net/restic-data restore --target /backup/restore-x --include ` (never restore into `/tmp` — it is RAM). Env: `. /root/.restic-env`. diff --git a/ks2/plan.md b/ks2/plan.md index 1e67811..320528d 100644 --- a/ks2/plan.md +++ b/ks2/plan.md @@ -18,10 +18,14 @@ What remains on the box is **cold history**: instance replicas on pool snapshotted daily there (`zfs-auto-snapshot.sh`, 2-month expiry). ⚠️ While the nas leg waits for FTTH, instances have no *fresh* -off-site copy — the ks2 push is to be re-enabled as soon as the -initial restic sync finishes (decided 2026-08-28), and retired again -when nas takes over. (The leg moved from nuc to the new host `nas` -on 2026-08-30 — see [nas-seed.md](nas-seed.md).) +off-site copy. The ks2 push was **deliberately not re-enabled** +(2026-08-30): ks2's replicas are three weeks old, its `data` pool is +94 % full, no common snapshot survives ks4's 7-day expiry, and the box +is wiped within the month — so a full ~1.5 T re-send buys four weeks +of freshness on hardware already scheduled for destruction. Off-site +protection meanwhile rests on `restic-data` (data, databases and the +incus configuration — enough to rebuild). (The leg moved from nuc to +the new host `nas` on 2026-08-30 — see [nas-seed.md](nas-seed.md).) ## Inventory findings (2026-08-22) @@ -53,7 +57,7 @@ on 2026-08-30 — see [nas-seed.md](nas-seed.md).) | off-site, S3 (data+DB) | **restic** → bucket `restic-data` ([restic-backup.md](../ks4/restic-backup.md)) | **live** (05:00) | | off-site, S3 (instances) | restic over `incus file mount` of the `backup`-project replicas | **shelved** 2026-08-28 (replication covers instances) | | off-site, nas | **nas** pulls `ks4:*` → pool `ks4backup` over WG ([nas-seed.md](nas-seed.md)) | waiting FTTH (< Sep 30); moved off nuc 2026-08-30 | -| off-site, ks2 (interim) | `incus-copy.sh -d ks2 -m push` at 02:00 until the nas leg seeds | to re-enable once the restic seed finishes | +| off-site, ks2 (interim) | `incus-copy.sh -d ks2 -m push` | **not re-enabled** 2026-08-30 — see above | ## Actions (backup work documented in [`../ks4/`](../ks4/), ks2-only tasks here) diff --git a/ks4/README.md b/ks4/README.md index 8226aca..02941e9 100644 --- a/ks4/README.md +++ b/ks4/README.md @@ -21,40 +21,43 @@ Incus host at OVH — public-facing self-hosted services. - ⚠️ The ZFS `data` pool is single-disk (not mirrored); durability rests on nightly cron jobs — 01:00 `incus copy --refresh` of all instances to the local `backup` pool on sdb5, then the instance leg - to S3; 05:00 restic (DB dumps + data trees) to S3; nuc pulls the - replicas over WireGuard. Full picture and restore procedures: + to S3; 05:00 restic (DB dumps + data trees) to S3; **nas** pulls the + replicas over WireGuard (after FTTH). Full picture and restore procedures: **[backup-strategy.md](../backup-strategy.md)** ([local-backup-cron.md](local-backup-cron.md), [incus-copy.md](incus-copy.md), [restic-backup.md](restic-backup.md)). -## Network flows (nuc ↔ ks4) +## Network flows (home ↔ ks4) ``` - nuc — home LAN 192.168.0.0/24 ks4 — OVH 193.70.35.17 + nas — home LAN 192.168.0.4 ks4 — OVH 193.70.35.17 +-----------------------------------+ +-------------------------------------+ | | | | -| host: wg-ks4 (10.8.0.20) | | [wireguard] 192.168.1.18 | +| host: wg-ks4 (10.8.0.22) | | [wireguard] 192.168.1.18 | | incus remote "ks4" ------+--WG-->| wg0 10.8.0.1/24, udp 51845 | | pull ks4:* -> pool ks4backup | udp | | masquerade -> eth0 | -| on usb4t [pending FTTH seed] | 51845 | | | +| on tank (SATA) [pending FTTH] | 51845 | | | | | | +-> incus API 192.168.1.1:8443 | | [transmission-bt] wg0 (10.8.0.21) | | | (ufw: only from .18) | | full tunnel 0.0.0.0/0 ------+--WG-->| | | | kill switch: no default route | udp | +-> WAN egress: torrents + | -| downloads -> /srv/media | 51845 | apt of transmission-bt | -| (usb4t/media, read by jellyfin) | | exit as 193.70.35.17 | +| downloads -> /export/media | 51845 | apt of transmission-bt | +| (NFS-exported to nuc) | | exit as 193.70.35.17 | | | | | -| 03:00 instance snapshots | | 03:00 instance snapshots | -| 03:30 incus-copy: all instances | | 01:00 incus-copy: all instances | -| -> project backup, pool | | -> project backup, zpool sdb5 | -| nucbackup (usb4t/backup/nuc) | | then restic instance leg -> S3 | +| 03:30 nuc pushes its instances | | 03:00 instance snapshots | +| -> nucbackup on tank | | 01:00 incus-copy: all instances | +| 04:00 nas replicates its own | | -> project backup, zpool sdb5 | +| -> nasbackup on tank | | | | 05:00 pull ks4:* -> ks4backup | | 05:00 restic: DB dumps + data | | [pending FTTH] | | trees -> S3 (restic-data) | -| 05:30 apt upgrade all containers | | Sun 14:00 restic maintenance | +| | | Sun 14:00 restic maintenance | +-----------------------------------+ +-------------------------------------+ phones/laptops: WG peers 10.8.0.2-3 reach 192.168.1.x through the same endpoint + nuc (on-demand media box) mounts /export/media from nas over NFSv4 ``` -Both tunnels initiate **from** nuc (home NAT, dynamic IP) toward ks4's +Both tunnels initiate **from home** (NAT, dynamic IP) toward ks4's fixed endpoint; ks4's incus API is never exposed to the internet. +The pull leg and its tunnel moved from nuc to nas on 2026-08-30 +([nas/README.md](../nas/README.md)). diff --git a/ks4/incus-copy.md b/ks4/incus-copy.md index cb76c3d..c50bbac 100644 --- a/ks4/incus-copy.md +++ b/ks4/incus-copy.md @@ -8,7 +8,7 @@ push to `ks2` (decommissioning): 1. **local** — replicas + dumps on a dedicated `backup` zpool on ks4's second disk (`sdb5`), survives `sda` death -2. **off-site** — replicas pulled by **nuc** into pool `ks4backup` +2. **off-site** — replicas pulled by **nas** into pool `ks4backup` (dataset `usb4t/backup/ks4`), survives losing ks4 entirely ## The script @@ -74,7 +74,7 @@ Cron (root on ks4) — replaces both ks2 jobs: ⚠️ Replicas in the `backup` project must stay **stopped** — they keep the live containers' static `192.168.1.x` addresses. -## Leg 2 — off-site pull from nuc +## Leg 2 — off-site pull from nas (was nuc until 2026-08-30) Storage pool on nuc (done 2026-08-09): `ks4backup`, backed by the dataset `usb4t/backup/ks4` on the USB 4 TB pool (quota on `usb4t/backup` @@ -87,7 +87,7 @@ incus storage create ks4backup zfs source=usb4t/backup/ks4 Replicas live only on the USB drive — if it fails, only backups are lost; nuc's own instances (pool `data` on the SSD) are unaffected. -**Direction: nuc pulls, through the WireGuard tunnel.** Verified +**Direction: nas pulls, through the WireGuard tunnel.** Verified 2026-08-09: ks4's API listens on wildcard `:8443` (so it answers on `192.168.1.1`, the `incusbr0` host address) but is **firewalled from the internet** — the VPN path keeps it that way, needs no inbound port diff --git a/nuc/usb4t-dropouts.md b/nuc/usb4t-dropouts.md index 5519d6b..d8211bd 100644 --- a/nuc/usb4t-dropouts.md +++ b/nuc/usb4t-dropouts.md @@ -1,4 +1,13 @@ -# usb4t: USB dropouts suspend the pool (2026-08) +# usb4t: USB dropouts suspend the pool (2026-08) — RESOLVED + +> **Outcome (2026-08-30): the disk moved off USB entirely.** It now +> runs on **direct SATA** in the new host `nas` +> ([nas/README.md](../nas/README.md)), as pool `tank`, and the ks4 +> pull leg plus its WireGuard tunnel moved with it +> ([ks2/nas-seed.md](../ks2/nas-seed.md)). Everything below is the +> investigation that led there — worth keeping for the diagnosis +> method and for the alerting gap it exposed, which applies to any +> host. Symptom seen first in the nightly backup log (`/var/log/incus-copy.log`, job in `/etc/cron.d/incus-copy`): every