diff --git a/README.md b/README.md index 912adab..30668c2 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,10 @@ # Personal infrastructure notes +Two repositories back this setup: +[**doc**](https://git.lutran.fr/julien/doc) (this one) and +[**scripts**](https://git.lutran.fr/julien/scripts) (the cron drivers +and their config files, deployed to `/root/scripts` on each host). + **Start here**: - [backup-strategy.md](backup-strategy.md) — what is backed up, by which tool, on what schedule, and how to restore. @@ -7,97 +12,12 @@ backed up, by which tool, on what schedule, and how to restore. service to ks4 (and getting it backed up automatically). **Tech notes** -- [`nuc/`](nuc/) — home lab on `nuc` (see below) -- [`ks4/`](ks4/) — prod server `ks4` at OVH (see below) +- [`nuc/`](nuc/README.md) — home lab on `nuc`: host, storage, instances +- [`ks4/`](ks4/README.md) — prod server `ks4` at OVH: host, services, network flows - [`ks2/`](ks2/) — legacy backup server being [decommissioned](ks2/plan.md) - [`archer-c7/`](archer-c7/) — home router (TP-Link Archer C7 v5 running on OpenWrt) -## Homelab: nuc - -Incus host on the LAN. - -- Debian 13, Intel Alder Lake-N (iGPU `i915`, shared by both Jellyfin containers) — - bare-metal reinstall: [nuc/nuc-install.md](nuc/nuc-install.md) -- Instances are bridged onto the LAN (192.168.0.0/24) -- USB 4 TB WD Red: ZFS pool `usb4t` — `usb4t/backup` → `/backup` - (incus exports, `nuc/` + `ks4/` subdatasets, 1 TB quota) and - `usb4t/media` → `/srv/media` (media library, shared into containers - via `shift=true` disk devices; works because ZFS ≥ 2.2 supports - idmapped mounts) -- Backups: all local instances replicated to the USB pool - (`/root/scripts/incus-copy.sh -p backup -s nucbackup`; replicas - stopped, autostart off) — see - [nuc/nuc-install.md](nuc/nuc-install.md); ks4 replicas pulled into - pool `ks4backup` — see [ks4/incus-copy.md](ks4/incus-copy.md) - -### Instances - -| Name | IP | Doc | Features | -|---|---|---|---| -| [jellyfin-server](nuc/jellyfin-server.md) | 192.168.0.5 | ✅ | unprivileged, autostart; iGPU render node (`gpu` device, render gid) for QSV/VAAPI transcoding; `/srv/media` disk device (`shift=true`); proxy device → host :8096 | -| [jellyfin-client](nuc/jellyfin-client.md) | 192.168.0.6 | ✅ | **privileged**, autostart; full iGPU (`gpu` device, gid 44) → HDMI kiosk (cage + Jellyfin Media Player); custom `raw.lxc` (bind `/dev/snd`, `/dev/input`, host `/run/udev`); Pioneer USB audio as ALSA default; FR keymap; go-librespot Spotify Connect ("Pioneer A-70") | -| [transmission-bt](nuc/transmission-bt.md) | 192.168.0.7 | ✅ | unprivileged, autostart; always-on WireGuard full tunnel → ks4 (egress = 193.70.35.17, kill switch: no default route); `/srv/media` disk device (`shift=true`), downloads to `/media/downloads`; web UI :9091 (LAN only) | -| blocky | 192.168.0.254 | — | unprivileged, autostart; DNS ad-blocker | -| privoxy | 192.168.0.11 | — | unprivileged, autostart; filtering HTTP proxy | -| homeassistant | (stopped) | — | **virtual machine**, 50 GiB root disk on pool `data` | - -## Prod: ks4 - -Incus host at OVH — public-facing self-hosted services. - -- Debian 13, Xeon D-1521 / 32 GiB, 2× 6 TB (OS on mdraid RAID1, - ZFS pool `data` on `sda5`) — setup & rebuild: - [ks4/install.md](ks4/install.md) -- SSH: `ssh -i id_rsa_claude -p 2233 julien@193.70.35.17` - (`julien` has passwordless `sudo incus …`) -- Instances are on a NAT bridge `incusbr0` (192.168.1.0/24); public - ingress via Incus proxy devices. `gateway` (nginx) is the sole HTTP/S - entry point and fans out to the app containers. -- Services: gateway, mail, nextcloud, seafile, git, freshrss, bitwarden, - outline, login (SSO), wireguard, + more — full table in - [ks4/install.md](ks4/install.md). -- ⚠️ The ZFS `data` pool is single-disk (not mirrored); durability - rests on nightly cron jobs — 01:00 `incus copy --refresh` of all - instances to the local `backup` pool on sdb5, then the instance leg - to S3; 05:00 restic (DB dumps + data trees) to S3; nuc pulls the - replicas over WireGuard. Full picture and restore procedures: - **[backup-strategy.md](backup-strategy.md)** - ([ks4/local-backup-cron.md](ks4/local-backup-cron.md), - [ks4/incus-copy.md](ks4/incus-copy.md), - [ks4/restic-backup.md](ks4/restic-backup.md)). - -## Network flows - -``` - nuc — home LAN 192.168.0.0/24 ks4 — OVH 193.70.35.17 -+-----------------------------------+ +-------------------------------------+ -| | | | -| host: wg-ks4 (10.8.0.20) | | [wireguard] 192.168.1.18 | -| incus remote "ks4" ------+--WG-->| wg0 10.8.0.1/24, udp 51845 | -| pull ks4:* -> pool ks4backup | udp | | masquerade -> eth0 | -| on usb4t [pending FTTH seed] | 51845 | | | -| | | +-> incus API 192.168.1.1:8443 | -| [transmission-bt] wg0 (10.8.0.21) | | | (ufw: only from .18) | -| full tunnel 0.0.0.0/0 ------+--WG-->| | | -| kill switch: no default route | udp | +-> WAN egress: torrents + | -| downloads -> /srv/media | 51845 | apt of transmission-bt | -| (usb4t/media, read by jellyfin) | | exit as 193.70.35.17 | -| | | | -| 03:00 instance snapshots | | 03:00 instance snapshots | -| 03:30 incus-copy: all instances | | 01:00 incus-copy: all instances | -| -> project backup, pool | | -> project backup, zpool sdb5 | -| nucbackup (usb4t/backup/nuc) | | then restic instance leg -> S3 | -| 05:00 pull ks4:* -> ks4backup | | 05:00 restic: DB dumps + data | -| [pending FTTH] | | trees -> S3 (restic-data) | -| 05:30 apt upgrade all containers | | Sun 14:00 restic maintenance | -+-----------------------------------+ +-------------------------------------+ - phones/laptops: WG peers 10.8.0.2-3 reach 192.168.1.x through the same endpoint -``` - -Both tunnels initiate **from** nuc (home NAT, dynamic IP) toward ks4's -fixed endpoint; ks4's incus API is never exposed to the internet. - -### Conventions +## Conventions - One markdown file per topic in the correct subdir, containing the full implementation notes, configuration, and troubleshooting notes. @@ -107,4 +27,5 @@ fixed endpoint; ks4's incus API is never exposed to the internet. - Adding a service to ks4: follow [new-container.md](new-container.md) (no Docker in containers — incus runs OCI images natively and `incus-compose` handles stacks; - data paths go into `scripts/restic-paths`). + data paths go into + [`scripts/restic-paths`](https://git.lutran.fr/julien/scripts/src/branch/main/restic-paths)). diff --git a/backup-strategy.md b/backup-strategy.md index 3d9ff79..802d465 100644 --- a/backup-strategy.md +++ b/backup-strategy.md @@ -13,7 +13,15 @@ docs; this page is the map. plakar was evaluated first and replaced (its nightly cost scaled with tree size — [PlakarKorp/plakar#2338](https://github.com/PlakarKorp/plakar/issues/2338)); -the old backup server **ks2 is retired** ([ks2/plan.md](ks2/plan.md)). +the old backup server **ks2 is being retired** (decommission by +Sep 30, 2026 — [ks2/plan.md](ks2/plan.md)): it still holds an ageing +copy of the instances until the nuc leg below takes over. + +**Status 2026-08-28**: the local replica leg and the S3 data leg are +live. The **nuc pull leg is waiting for FTTH** (expected before end of +September) and the **S3 instance leg is written but not yet seeded** — +until both land, instances have only the ks4-local replica off the +live pool. ## The map @@ -95,4 +103,4 @@ without the passphrase the S3 buckets are unreadable. New instances are picked up automatically by the replica and instance legs (all instances, opt-out only). Databases are auto-discovered. Only **data trees** need one line in -`/root/scripts/restic-paths` — see [new-container.md](new-container.md). +[`restic-paths`](https://git.lutran.fr/julien/scripts/src/branch/main/restic-paths) — see [new-container.md](new-container.md). diff --git a/ks2/decommission.md b/ks2/decommission.md index 31625c3..5a27ac9 100644 --- a/ks2/decommission.md +++ b/ks2/decommission.md @@ -6,8 +6,8 @@ few days of margin. ## Preconditions (the plan's release gates) -- [ ] plakar S3 leg running nightly for ≥ a week, `done (rc=0)`, - restore test passed ([plakar-s3-data.md](../ks4/plakar-s3-data.md) §7) +- [ ] restic S3 leg running nightly for ≥ a week, `done (rc=0)`, + restore test passed ([restic-backup.md](../ks4/restic-backup.md) §7) - [ ] nuc pull leg seeded and one instance test-restored ([nuc-seed.md](nuc-seed.md)) - [ ] local leg (sdb5) cron green in `/var/log/incus-copy.log` @@ -15,7 +15,8 @@ few days of margin. ## 1. Cut the last flows to ks2 (root on ks4) ```sh -crontab -e # delete the 04:00 incus-backup.sh line (164.132.173.57) +crontab -e # remove any remaining ks2 line (the 02:00 push, if it was + # re-enabled; the 04:00 incus-backup.sh line is long gone) incus remote list # if a "ks2" remote is still defined: incus remote remove ks2 # drop the rsyncbackup SSH key if it exists only for ks2: @@ -25,15 +26,15 @@ ls /root/.ssh/id_rsyncbackup* && rm -i /root/.ssh/id_rsyncbackup* ## 2. Last look at ks2 before wiping (root on ks2) ```sh -# anything in the rsync tree newer than the plakar cutover? +# anything in the rsync tree newer than the S3 cutover? find /backup/ns3061243 -newermt '2026-08-23' | head # pre-2023 dirs (catc, mythoughts, qcm, scripts): instances still # exist stopped on ks4 — spot-check nothing unique remains: ls /backup/ns3061243/{catc,mythoughts,qcm,scripts} ``` -Archive anything unique to the plakar dump dir on ks4 (it rides the -nightly `dumps` snapshot to S3 for free once it stops changing). +Archive anything unique into the restic dump dir on ks4 +(`/backup/dumps`) — it then rides the nightly S3 snapshot for free. ## 3. Wipe and terminate @@ -52,9 +53,10 @@ renewal date. durability bullet and the index line for `ks2/`s "being decommissioned" wording (keep the folder as history) - [ ] Retire `scripts/incus-backup.sh` + `incus-backup.db` in the repo - (superseded by `scripts/plakar-backup.sh`) — keep for reference, - note the retirement in their headers -- [ ] `login` container cleanup parked in - [plakar-s3-data.md](../ks4/plakar-s3-data.md): `dropdb outline`, - remove native postgres + (superseded by [`restic-backup.sh`](https://git.lutran.fr/julien/scripts/src/branch/main/restic-backup.sh)) — + keep for reference, note the retirement in their headers +- [ ] `login` container cleanup: the native postgres still holds the + pre-migration `outline` DB — `dropdb outline`, remove the + package (noted in + [plakar-s3-data.md](../ks4/plakar-s3-data.md) §3) - [ ] Mark the ks2 line in `plan.md` done, date it diff --git a/ks2/nuc-seed.md b/ks2/nuc-seed.md index f084978..b819a5d 100644 --- a/ks2/nuc-seed.md +++ b/ks2/nuc-seed.md @@ -29,7 +29,7 @@ Notes: ## Cron (after the seed) Add to nuc's root crontab, offset from the 03:30 local nucbackup copy -and ks4's own 01:00/04:30 jobs: +and ks4's own 01:00/05:00 jobs: ```cron 0 5 * * * /root/scripts/incus-copy.sh -r ks4 -s ks4backup >> /var/log/incus-copy-ks4.log 2>&1 diff --git a/ks2/plan.md b/ks2/plan.md index c166f66..c7657a7 100644 --- a/ks2/plan.md +++ b/ks2/plan.md @@ -4,20 +4,23 @@ Goal: release `ks2.lutran.fr` (`164.132.173.57`, SSH :2233, host `ns3247221`) once ks4 has a real 3-2-1 backup without it. **Deadline: rental ends Sep 30, 2026.** -## What ks2 does today (updated 2026-08-22) +## What ks2 does today (updated 2026-08-28) -One remaining role, fed nightly by a root cron **on ks4**: +**Nothing is written to ks2 any more.** Both feeds are retired: the +`01:00` replica push (replaced by the local leg on ks4's sdb5 pool, +[local-backup-cron.md](../ks4/local-backup-cron.md)) and the `04:00` +`incus-backup.sh` rsync (replaced by restic → S3, +[restic-backup.md](../ks4/restic-backup.md)). -- `04:00` `incus-backup.sh … -d 164.132.173.57` — DB dumps + selected - paths rsync'd to ks2 `/backup/ns3061243` on pool `backup` (1.59 T, - 88 % full), snapshotted daily on ks2 (`zfs-auto-snapshot.sh`, - 2-month expiry). **Kept as a stopgap** until the plakar S3 leg - replaces it. +What remains on the box is **cold history**: instance replicas on pool +`data` (last refreshed 2026-08-09) and the rsync tree +`/backup/ns3061243` on pool `backup` (last refreshed 2026-08-28), +snapshotted daily there (`zfs-auto-snapshot.sh`, 2-month expiry). -Retired: the `01:00` replica push to ks2 pool `data` — replaced by -the local leg on ks4's sdb5 pool -([local-backup-cron.md](../ks4/local-backup-cron.md)); `biwiki` and `spot` -deleted from ks2. +⚠️ Consequence while the nuc leg waits for FTTH: **instances have no +fresh off-site copy** — only the ks4-local sdb replicas plus ks2's +ageing ones. Either re-enable the interim push (below) or accept the +gap knowingly until FTTH. ## Inventory findings (2026-08-22) @@ -25,8 +28,9 @@ deleted from ks2. project — **decision: both abandoned**, safe to delete from backups. - `incus-backup.db` is stale (lists `spot`, misses `livetrail`, `outline`, `login`, …) — forgotten-manifest drift is exactly what - the plakar-based replacement must eliminate (back up *all* - instances by default, opt-out instead of opt-in). + the replacement had to eliminate — restic's drivers back up *all* + instances and auto-discover *all* databases, opt-out instead of + opt-in. - `livetrail` (running on ks4) is **absent from ks4's local `backup` project** — the local leg was run manually once and never again; automating its cron fixes this. @@ -38,17 +42,17 @@ deleted from ks2. `qcm`) — those instances still exist stopped on ks4, so nothing unique expected there; spot-check before wiping. - Losing ks2 also loses its 2 months of rsync-snapshot history — - acceptable once plakar has built equivalent retention. + acceptable: restic keeps 14 daily / 8 weekly / 6 monthly. ## Target architecture (3-2-1 for ks4) | Leg | Mechanism | Status | |---|---|---| -| local, 2nd disk | `incus-copy.sh -p backup -s backup` → sdb5 pool | works, **cron missing** | -| off-site, nuc | nuc pulls `ks4:*` → pool `ks4backup` over WG | waiting FTTH seed | -| off-site, S3 (data+DB) | **restic** on ks4 → S3 (plakar ran it first, being replaced — see [restic-backup.md](../ks4/restic-backup.md)) | migrating | -| off-site, S3 (instances) | restic over `incus file mount` of the `backup`-project replicas | designed | -| off-site, ks2 (interim) | 02:00 replica push until the nuc leg seeds (FTTH < Sep 30) | to re-enable | +| local, 2nd disk | `incus-copy.sh -p backup -s backup` → sdb5 pool | **live** (01:00) | +| off-site, S3 (data+DB) | **restic** → bucket `restic-data` ([restic-backup.md](../ks4/restic-backup.md)) | **live** (05:00) | +| off-site, S3 (instances) | restic over `incus file mount` of the `backup`-project replicas | written, **not seeded** | +| off-site, nuc | nuc pulls `ks4:*` → pool `ks4backup` over WG ([nuc-seed.md](nuc-seed.md)) | waiting FTTH (< Sep 30) | +| off-site, ks2 (optional interim) | `incus-copy.sh -d ks2 -m push` at 02:00 until the nuc leg seeds | **not enabled** — decide | ## Actions (backup work documented in [`../ks4/`](../ks4/), ks2-only tasks here) @@ -57,13 +61,11 @@ deleted from ks2. 2. ~~[local-backup-cron.md](../ks4/local-backup-cron.md)~~ — **done 2026-08-22**: leg 1 cron re-enabled, ks2 rsync kept as stopgap; `livetrail` verified present in the `backup` project -3. [plakar-s3-data.md](../ks4/plakar-s3-data.md) — **seeding S3 - 2026-08-22** (direct-to-S3 kloset `lutran-ks4-plakar-data`, driver - script `scripts/plakar-backup.sh`, pipeline validated locally); - next: restore test + cron -4. [plakar-incus-integration.md](../ks4/plakar-incus-integration.md) — - design seeded 2026-08-22: per-file importer over the incus sftp - API, instances → S3 +3. ~~plakar S3 leg~~ → **replaced by restic** + ([restic-backup.md](../ks4/restic-backup.md), live 2026-08-28; + plakar's own doc kept as history) +4. instance leg — restic over `incus file mount` + ([restic-backup.md](../ks4/restic-backup.md) §6): **seed pending** 5. [nuc-seed.md](nuc-seed.md) — **prepared**; after FTTH: seed nuc pull leg, verify all instances, test-restore one 6. [decommission.md](decommission.md) — **prepared**; cut flows, @@ -74,6 +76,6 @@ deleted from ks2. - [x] biwiki + spot consciously abandoned (2026-08-22) - [ ] local leg cron running ≥ a few days, all instances present - [ ] nuc leg fully seeded **and** one instance test-restored -- [ ] plakar S3 data/DB backups running daily **and** one DB + one +- [ ] restic S3 data/DB backups running daily **and** one DB + one file tree test-restored - [ ] ks4 crons pointing at ks2 disabled diff --git a/ks4/README.md b/ks4/README.md new file mode 100644 index 0000000..8a57ad0 --- /dev/null +++ b/ks4/README.md @@ -0,0 +1,55 @@ +# Prod: ks4 + +Incus host at OVH — public-facing self-hosted services. + +- Debian 13, Xeon D-1521 / 32 GiB, 2× 6 TB (OS on mdraid RAID1, + ZFS pool `data` on `sda5`) — setup & rebuild: + [install.md](install.md) +- SSH: `ssh -i id_rsa_claude -p 2233 julien@193.70.35.17` + (`julien` has passwordless `sudo incus …`) +- Instances are on a NAT bridge `incusbr0` (192.168.1.0/24); public + ingress via Incus proxy devices. `gateway` (nginx) is the sole HTTP/S + entry point and fans out to the app containers. +- Services: gateway, mail, nextcloud, seafile, git, freshrss, bitwarden, + outline, login (SSO), wireguard, + more — full table in + [install.md](install.md). +- ⚠️ The ZFS `data` pool is single-disk (not mirrored); durability + rests on nightly cron jobs — 01:00 `incus copy --refresh` of all + instances to the local `backup` pool on sdb5, then the instance leg + to S3; 05:00 restic (DB dumps + data trees) to S3; nuc pulls the + replicas over WireGuard. Full picture and restore procedures: + **[backup-strategy.md](../backup-strategy.md)** + ([local-backup-cron.md](local-backup-cron.md), + [incus-copy.md](incus-copy.md), + [restic-backup.md](restic-backup.md)). + +## Network flows (nuc ↔ ks4) + +``` + nuc — home LAN 192.168.0.0/24 ks4 — OVH 193.70.35.17 ++-----------------------------------+ +-------------------------------------+ +| | | | +| host: wg-ks4 (10.8.0.20) | | [wireguard] 192.168.1.18 | +| incus remote "ks4" ------+--WG-->| wg0 10.8.0.1/24, udp 51845 | +| pull ks4:* -> pool ks4backup | udp | | masquerade -> eth0 | +| on usb4t [pending FTTH seed] | 51845 | | | +| | | +-> incus API 192.168.1.1:8443 | +| [transmission-bt] wg0 (10.8.0.21) | | | (ufw: only from .18) | +| full tunnel 0.0.0.0/0 ------+--WG-->| | | +| kill switch: no default route | udp | +-> WAN egress: torrents + | +| downloads -> /srv/media | 51845 | apt of transmission-bt | +| (usb4t/media, read by jellyfin) | | exit as 193.70.35.17 | +| | | | +| 03:00 instance snapshots | | 03:00 instance snapshots | +| 03:30 incus-copy: all instances | | 01:00 incus-copy: all instances | +| -> project backup, pool | | -> project backup, zpool sdb5 | +| nucbackup (usb4t/backup/nuc) | | then restic instance leg -> S3 | +| 05:00 pull ks4:* -> ks4backup | | 05:00 restic: DB dumps + data | +| [pending FTTH] | | trees -> S3 (restic-data) | +| 05:30 apt upgrade all containers | | Sun 14:00 restic maintenance | ++-----------------------------------+ +-------------------------------------+ + phones/laptops: WG peers 10.8.0.2-3 reach 192.168.1.x through the same endpoint +``` + +Both tunnels initiate **from** nuc (home NAT, dynamic IP) toward ks4's +fixed endpoint; ks4's incus API is never exposed to the internet. diff --git a/ks4/incus-copy.md b/ks4/incus-copy.md index 8edeb07..cb76c3d 100644 --- a/ks4/incus-copy.md +++ b/ks4/incus-copy.md @@ -1,7 +1,7 @@ # incus-copy — ks4 instance replication Incremental replication of all ks4 Incus instances via -[`scripts/incus-copy.sh`](../scripts/incus-copy.sh) +[`scripts/incus-copy.sh`](https://git.lutran.fr/julien/scripts/src/branch/main/incus-copy.sh) (`incus copy --refresh`: ZFS-incremental, produces ready-to-start replicas instead of tarballs). Two legs replace the historical nightly push to `ks2` (decommissioning): @@ -35,7 +35,7 @@ incus-copy.sh -p -s # local `--refresh-exclude-older` keeps expired snapshots from being re-sent. - Consistency: copies of running instances are crash-consistent; databases may need recovery on restore. Snapshot history mitigates, - and [`incus-backup.sh`](../scripts/incus-backup.sh) (proper + and [`incus-backup.sh`](https://git.lutran.fr/julien/scripts/src/branch/main/incus-backup.sh) (proper `mariadb-dump`s + selected paths, manifest-driven) covers the DBs. - **VMs**: `--refresh` re-sends a VM's **full block volume** every run unless source and replica share a common snapshot — containers diff @@ -63,7 +63,9 @@ Cron (root on ks4) — replaces both ks2 jobs: ```cron 0 1 * * * /root/scripts/incus-copy.sh -p backup -s backup >> /var/log/incus-copy.log 2>&1 -0 4 * * * /root/scripts/incus-backup.sh -f /root/scripts/incus-backup.db -s data >> /var/log/incus-backup.log 2>&1 +# retired 2026-08-28 — dumps + data trees are restic's job (05:00), +# see ../backup-strategy.md and restic-backup.md +#0 4 * * * /root/scripts/incus-backup.sh -f /root/scripts/incus-backup.db -s data >> /var/log/incus-backup.log 2>&1 ``` (`incus-backup.sh` without `-d` now writes locally to diff --git a/ks4/install.md b/ks4/install.md index 433ec87..b225ec5 100644 --- a/ks4/install.md +++ b/ks4/install.md @@ -142,22 +142,32 @@ All in the `default` project, on the `default` profile ## Backups -> **Migration in progress** (2026-08-09): the `ks2` destination below is -> being replaced by a local `backup` zpool on `sdb5` + off-site pull -> from nuc — plan and setup in [incus-copy.md](incus-copy.md). The -> cron blocks here describe what runs **today**; update on cutover. - -Two **root** cron jobs (this repo's [`scripts/`](../scripts/) are the -copies that run here) — the primary safety net for the non-mirrored -`data` pool: +Full picture, schedule and restore procedures: +**[../backup-strategy.md](../backup-strategy.md)**. Drivers live in the +[scripts repo](https://git.lutran.fr/julien/scripts) and are deployed +to `/root/scripts`. Current **root** cron jobs — the safety net for the +non-mirrored `data` pool: ```cron -0 1 * * * /root/scripts/incus-copy.sh -d ks2 -m push -0 4 * * * /root/scripts/incus-backup.sh -d 164.132.173.57 -u rsyncbackup \ - -i /root/.ssh/id_rsyncbackup -p 2233 \ - -f /root/scripts/incus-backup.db -s data +0 1 * * * /root/scripts/incus-copy.sh -p backup -s backup >> /var/log/incus-copy.log 2>&1 +0 5 * * * /root/scripts/restic-backup.sh >> /var/log/restic-backup.log 2>&1 +0 14 * * 0 /root/scripts/restic-maintenance.sh >> /var/log/restic-maintenance.log 2>&1 ``` +- **01:00 `incus-copy.sh`** → replicas of every instance in project + `backup` on the sdb5 pool ([incus-copy.md](incus-copy.md), + [local-backup-cron.md](local-backup-cron.md)); the S3 instance leg + chains here once seeded. +- **05:00 `restic-backup.sh`** → auto-discovered DB dumps + the data + trees listed in + [`scripts/restic-paths`](https://git.lutran.fr/julien/scripts/src/branch/main/restic-paths), + into the `restic-data` bucket ([restic-backup.md](restic-backup.md)). +- **Sun 14:00 `restic-maintenance.sh`** → prune + integrity checks. + +Historical (retired 2026-08): the `ks2` push and `incus-backup.sh` +rsync leg, and the plakar data leg +([plakar-s3-data.md](plakar-s3-data.md)). + - **01:00 — `incus-copy.sh` → `ks2`**: differential `incus copy … \ --refresh --refresh-exclude-older --mode push` of every *running* instance to remote incus host `ks2` (a near-live replica). `ks2` must diff --git a/ks4/local-backup-cron.md b/ks4/local-backup-cron.md index 972a3ad..b730ca7 100644 --- a/ks4/local-backup-cron.md +++ b/ks4/local-backup-cron.md @@ -1,28 +1,23 @@ -# local backup cron on ks4 (leg 1) — re-enabled +# local backup cron on ks4 (leg 1) -Done 2026-08-22. Context: **both** ks4 backup crons had been commented -out — no backup of any kind was running. +Enabled 2026-08-22 (both ks4 backup crons had been commented out — no +backup of any kind was running). Overview of all legs: +[../backup-strategy.md](../backup-strategy.md). ## What runs now (root crontab on ks4) ```cron 0 1 * * * /root/scripts/incus-copy.sh -p backup -s backup >> /var/log/incus-copy.log 2>&1 -0 4 * * * /root/scripts/incus-backup.sh -d 164.132.173.57 -u rsyncbackup -i /root/.ssh/id_rsyncbackup -p 2233 -f /root/scripts/incus-backup.db -s data >> /var/log/incus-backup.log 2>&1 ``` - `01:00` — **local leg**: `incus copy --refresh` of all instances into project `backup` on the sdb5 `backup` zpool (replaces the old - push to ks2; see [incus-copy.md](incus-copy.md)). -- `04:00` — **stopgap**: `incus-backup.sh` rsync to ks2 kept alive - (manifest pruned of `spot`) until the plakar S3 leg replaces it — - then this line goes away as part of - [decommissioning ks2](../ks2/plan.md). ⚠️ keep - `scripts/incus-backup.db` in this repo in sync with - `/root/scripts/incus-backup.db` on ks4. - -Both were also run once by hand (in a `screen`) on 2026-08-22 to -refresh the backups immediately. The old push-to-ks2 copy line stays -commented out — replica refreshes to ks2 are retired. + push to ks2; see [incus-copy.md](incus-copy.md)). The S3 instance + leg ([restic-backup.md](restic-backup.md) §6) chains onto this same + line once seeded — it reads the replicas this job just refreshed. +- The rsync stopgap to ks2 (`incus-backup.sh`, 04:00) and the plakar + data leg (04:30) are both **retired**; S3 backups are restic's job + at 05:00. ## Cleanup done the same day diff --git a/ks4/plakar-incus-integration.md b/ks4/plakar-incus-integration.md index 7ce34a1..0bbc2ac 100644 --- a/ks4/plakar-incus-integration.md +++ b/ks4/plakar-incus-integration.md @@ -1,4 +1,14 @@ -# plakar incus integration — design notes +# plakar incus integration — design notes (RETIRED) + +> **Historical.** The importer works (smoke-tested against ks4's +> `backup` project) but production moved to restic, which needs no +> plugin: `incus file mount` exposes any instance — including stopped +> replicas — over the same per-instance sftp API +> ([restic-backup.md](restic-backup.md) §6). Code lives on at +> ; the analysis +> below (per-file vs image, replica-vs-live consistency, the +> seek-bound wall) is what shaped the restic design. + Status: **importer scaffolded and compiling 2026-08-23** in `perso/plakar/integration-incus` (walker modeled on the official sftp diff --git a/ks4/plakar-s3-data.md b/ks4/plakar-s3-data.md index bd438b9..6bc0461 100644 --- a/ks4/plakar-s3-data.md +++ b/ks4/plakar-s3-data.md @@ -1,4 +1,15 @@ -# plakar data + DB backups on ks4 (leg 3) +# plakar data + DB backups on ks4 (leg 3) — SUPERSEDED + +> **Historical.** This leg ran 2026-08-22 → 08-28 and was replaced by +> restic: see **[restic-backup.md](restic-backup.md)**. plakar's +> incremental cost scaled with tree size rather than churn +> ([PlakarKorp/plakar#2338](https://github.com/PlakarKorp/plakar/issues/2338)). +> The `plakar-data` kloset and the plakar install are deliberately kept +> until that issue concludes (the report offers debug runs against that +> repository); the `plakar-incus` bucket can be deleted. Kept here for +> the measurements and the design rationale, which the restic doc +> builds on. + Status: **live since 2026-08-24** — S3 kloset (`lutran-ks4-plakar-data`) seeded (dumps + 9 fs sources), restore @@ -19,7 +30,7 @@ leg 1's job. Direct-to-S3 keeps the incremental behavior — the skip-unchanged baseline is the parent snapshot's VFS read from the kloset (plakar caches repository metadata locally; v1.1.4's dirpack prefetcher targets exactly this remote-backend walk). Driver script: -[`scripts/plakar-backup.sh`](../../scripts/plakar-backup.sh) +[`scripts/plakar-backup.sh`](https://git.lutran.fr/julien/scripts/src/branch/main/plakar-backup.sh) (flock, incus DB dumps, iterates a sources file, prune + maintenance; its `-k` takes any `plakar at` target, so `-k @s3` — a throwaway local kloset, e.g. `-k /backup/plakar-test`, works for testing before diff --git a/ks4/restic-backup.md b/ks4/restic-backup.md index 4b56f91..384a96c 100644 --- a/ks4/restic-backup.md +++ b/ks4/restic-backup.md @@ -1,10 +1,12 @@ # restic backups on ks4 — the S3 leg -Status: **decided 2026-08-25 — restic replaces plakar as the S3 -backup tool** (evaluation first: parallel run + comparison, see §8). -Final architecture: **two tools** — `incus copy` for replication -(local sdb, nuc after FTTH, ks2 push as interim), **restic** for -everything S3. +Status: **live since 2026-08-28** — restic replaced plakar as the S3 +backup tool (plakar crons off, its kloset kept for +[#2338](https://github.com/PlakarKorp/plakar/issues/2338); see §8). +Architecture: **two tools** — `incus copy` for replication (local sdb, +nuc after FTTH), **restic** for everything S3. Data leg (`restic-data`) +runs nightly at 05:00; the instance leg (`restic-incus`, §6) is written +and awaits its first seed. Why: plakar's incremental cost scales with tree size, not churn (~800 MiB parent metadata re-read from S3 + hours of CPU per night on @@ -58,9 +60,10 @@ property). ### Memory (the real constraint on this 23/31 GiB-used box) - Index RAM ≈ 200–300 B/blob → combined `restic-data` index - ~1–1.5 GiB per operation (nextcloud ~1 M blobs + seafile ~2–4 M); - prune peaks 2–3×. Accepted with the merged layout (§1) — measure - peak RSS during the evaluation week (`/usr/bin/time -v`). + ~1 GiB per operation; prune peaks 2–3×. The seafile side of that + estimate collapsed after its garbage collection (24.9 M → 0.5 M + objects, [seafile-gc.md](seafile-gc.md)). Measure peak RSS with + `/usr/bin/time -v` if the box ever feels tight. - `GOGC=20` trades CPU for a smaller Go heap — enable if the seafile prune ever pressures the box. @@ -140,7 +143,7 @@ for r in restic-data restic-incus; do done ``` -## 4. Seed plan +## 4. Seed plan (executed 2026-08-25 → 08-28) 1. **ARC floor first** (helps every walk on the box): ```sh @@ -252,18 +255,18 @@ restic -r s3:…/restic-data restore latest --target /backup/restore-test --incl diff -r … && rm -rf /backup/restore-test ``` -## 8. Evaluation protocol + plakar retirement +## 8. Cutover (done) + what is left -- **Parallel week**: plakar 04:30 cron keeps running untouched; - restic runs at 05:00 on the same data. Same nights, same churn → - direct comparison (wall, bytes read from store, bytes written, peak - RSS via `/usr/bin/time -v`) against plakar's measured table in - [plakar-s3-data.md](plakar-s3-data.md). Restore-speed compared on - the same 10 GiB dumps tree (plakar measured: 39 min). -- **Cutover** (after the comparison confirms + restore test passes): - drop the plakar 04:30 + Sunday-check crons; 01:00 becomes - `incus-copy ; restic instances phase` (or stays in the 05:00 run). - Update plan.md legs table. +- **Cutover 2026-08-28**: plakar's 04:30 and Sunday crons are off; + restic's 05:00 data leg is the S3 backup. The formal parallel-week + comparison was cut short — the evidence had already settled it + (plakar: ~800 MiB metadata re-read and 3–4 h per night on the + nextcloud tree, churn-independent; restic: seed at 60–115 MiB/s, + local-cache incrementals). plakar's measurements stay in + [plakar-s3-data.md](plakar-s3-data.md). +- **Left to do**: seed `restic-incus` (§6) and chain it onto the 01:00 + line; restore test on the complete `restic-data`; then the ks2 + decommission gates ([../ks2/plan.md](../ks2/plan.md)). - **plakar retirement, one exception**: keep plakar installed and keep the `plakar-data` kloset until [#2338](https://github.com/PlakarKorp/plakar/issues/2338) concludes diff --git a/ks4/seafile-gc.md b/ks4/seafile-gc.md index 207050b..f2b11f9 100644 --- a/ks4/seafile-gc.md +++ b/ks4/seafile-gc.md @@ -21,8 +21,8 @@ current state → new library → delete old → GC `-r`, which works for takes the dataset to ~600 k inodes, at which point per-file backup of seafile becomes ordinary. -All crontab backup jobs on ks4 remain commented out until restart + -validation completes. +(The ks4 backup crons were paused during the GC window and are back +on since 2026-08-27.) ## Root cause of the 24.9 M-inode / 933 G dataset @@ -73,9 +73,15 @@ setting (the existing per-library 30 d row stays). Users can still override per library in the web UI if `ENABLE_REPO_HISTORY_SETTING = True` in `seahub_settings.py`. -### 2. GC — offline, maintenance window +### 2. GC — maintenance window -CE's GC requires seafile stopped. On this pathological store expect +⚠️ Superseded by experience: with the **MySQL backend seafile runs +*online* GC** (the runs print "use online GC"), so stopping the +services is not required — the recurring cron below does not. The +2026-08-26 run stopped them anyway, out of caution. What follows is +that run's procedure. + +CE's GC traditionally requires seafile stopped. On this pathological store expect **hours** (it traverses fs/commits on a cold HDD); the dry-run sizes the window. Inside the container: @@ -115,7 +121,7 @@ du -sh .../seafile/rootfs/opt/seafile/seafile-data/storage/blocks # was 867 G 0 6 15 1,4,7,10 * incus exec seafile -- sh -c "cd /opt/seafile/seafile-server-latest && ./seaf-gc.sh && ./seaf-gc.sh -r && ./seaf-gc.sh -R" >> /var/log/seafile-gc.log 2>&1 ``` At the post-cleanup scale (~120 k live files) the full sequence is - minutes, not hours. Keep it clear of the 01:00/04:30/05:00 backup + minutes, not hours. Keep it clear of the 01:00/05:00 backup slots. - 2026-08-27 follow-ups executed: 8 old libraries deleted by user → trash emptied → `seaf-gc -r` purged their stores (18 m, blocks now diff --git a/new-container.md b/new-container.md index 7d425d2..747a7ca 100644 --- a/new-container.md +++ b/new-container.md @@ -75,7 +75,7 @@ them forces a full re-send of its whole disk every night. **Manual, one line:** data that lives in the filesystem (uploads, repositories, mail spools…) must be listed in -`/root/scripts/restic-paths`, one absolute host path per line: +[`restic-paths`](https://git.lutran.fr/julien/scripts/src/branch/main/restic-paths), one absolute host path per line: ``` /var/lib/incus/storage-pools/data/containers/myservice/rootfs/var/lib/myservice @@ -84,7 +84,7 @@ repositories, mail spools…) must be listed in (The container's `/x` is `…/containers//rootfs/x` on the host.) No comments in that file — every line is read as a path. Regenerable caches can be skipped by adding a pattern to -`/root/scripts/restic-exclude`. Both files live in the `scripts` git +[`restic-exclude`](https://git.lutran.fr/julien/scripts/src/branch/main/restic-exclude). Both files live in the `scripts` git repo: edit there, commit, `git pull` on ks4. For the database auto-discovery to work, keep the defaults: @@ -98,8 +98,8 @@ For the database auto-discovery to work, keep the defaults: "postgres" (official images do). A database you do *not* want dumped (huge, static, re-importable) -goes into `/root/scripts/plakar-db-exclude` as `/` — -every skip is logged, so the list cannot rot silently. +goes into `/root/scripts/db-exclude` as `/` — every +skip is logged, so the list cannot rot silently. ## 5. Verify, once diff --git a/nuc/README.md b/nuc/README.md new file mode 100644 index 0000000..06ebbd8 --- /dev/null +++ b/nuc/README.md @@ -0,0 +1,28 @@ +# Homelab: nuc + +Incus host on the LAN. + +- Debian 13, Intel Alder Lake-N (iGPU `i915`, shared by both Jellyfin containers) — + bare-metal reinstall: [nuc-install.md](nuc-install.md) +- Instances are bridged onto the LAN (192.168.0.0/24) +- USB 4 TB WD Red: ZFS pool `usb4t` — `usb4t/backup` → `/backup` + (incus exports, `nuc/` + `ks4/` subdatasets, 1 TB quota) and + `usb4t/media` → `/srv/media` (media library, shared into containers + via `shift=true` disk devices; works because ZFS ≥ 2.2 supports + idmapped mounts) +- Backups: all local instances replicated to the USB pool + (`/root/scripts/incus-copy.sh -p backup -s nucbackup`; replicas + stopped, autostart off) — see + [nuc-install.md](nuc-install.md); ks4 replicas pulled into + pool `ks4backup` — see [ks4/incus-copy.md](../ks4/incus-copy.md) + +## Instances + +| Name | IP | Doc | Features | +|---|---|---|---| +| [jellyfin-server](jellyfin-server.md) | 192.168.0.5 | ✅ | unprivileged, autostart; iGPU render node (`gpu` device, render gid) for QSV/VAAPI transcoding; `/srv/media` disk device (`shift=true`); proxy device → host :8096 | +| [jellyfin-client](jellyfin-client.md) | 192.168.0.6 | ✅ | **privileged**, autostart; full iGPU (`gpu` device, gid 44) → HDMI kiosk (cage + Jellyfin Media Player); custom `raw.lxc` (bind `/dev/snd`, `/dev/input`, host `/run/udev`); Pioneer USB audio as ALSA default; FR keymap; go-librespot Spotify Connect ("Pioneer A-70") | +| [transmission-bt](transmission-bt.md) | 192.168.0.7 | ✅ | unprivileged, autostart; always-on WireGuard full tunnel → ks4 (egress = 193.70.35.17, kill switch: no default route); `/srv/media` disk device (`shift=true`), downloads to `/media/downloads`; web UI :9091 (LAN only) | +| blocky | 192.168.0.254 | — | unprivileged, autostart; DNS ad-blocker | +| privoxy | 192.168.0.11 | — | unprivileged, autostart; filtering HTTP proxy | +| homeassistant | (stopped) | — | **virtual machine**, 50 GiB root disk on pool `data` | diff --git a/nuc/nuc-install.md b/nuc/nuc-install.md index 8a5d57a..b311f88 100644 --- a/nuc/nuc-install.md +++ b/nuc/nuc-install.md @@ -19,7 +19,7 @@ How to rebuild the Incus host from scratch if `/dev/sda` (512 GB SSD, The ZFS pool `data` lives on `sda5` → **all instance root disks are lost** with the OS. The USB pool `usb4t` (backups + media) survives — all instances are replicated onto it with -[`scripts/incus-copy.sh`](../scripts/incus-copy.sh) (deployed at +[`scripts/incus-copy.sh`](https://git.lutran.fr/julien/scripts/src/branch/main/incus-copy.sh) (deployed at `/root/scripts/` on nuc; set up 2026-08-09): ```sh