julien/doc
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

doc: split per-host READMEs, gitea cross-repo links, consistency pass

Browse Source 
- nuc/README.md and ks4/README.md carry the host sections (+ network
  flows) that lived in the top-level README; links rebased
- top README: repo links (doc/scripts on git.lutran.fr), index points
  at the new per-host pages
- cross-repo references now use https://git.lutran.fr/julien/scripts
  instead of relative ../scripts paths that resolve nowhere
- plakar-s3-data.md and plakar-incus-integration.md marked SUPERSEDED
  / RETIRED with pointers to restic-backup.md; their measurements and
  rationale kept
- install.md, local-backup-cron.md, incus-copy.md: crontab sections
  updated to the live schedule (01:00 replicas, 05:00 restic, Sun
  maintenance); retired legs labelled as such
- restic-backup.md: status live, cutover recorded, post-GC memory
  estimate, seed plan dated
- seafile-gc.md: online GC noted, stale 'crons commented out' removed
- ks2/: what-ks2-does-today rewritten (nothing writes to it any more),
  legs table and gates reflect restic, decommission steps updated
- db-exclude replaces the plakar-era config name (script keeps a
  fallback)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Julien Lutran
2026-08-28 16:04:07 +02:00
co-authored by Claude Fable 5
parent 80f0219716
commit a0e3a7dd86
16 changed files with 246 additions and 193 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+10 -89
View File
@@ -1,5 +1,10 @@
# Personal infrastructure notes
Two repositories back this setup:
[**doc**](https://git.lutran.fr/julien/doc) (this one) and
[**scripts**](https://git.lutran.fr/julien/scripts) (the cron drivers
and their config files, deployed to `/root/scripts` on each host).
**Start here**:
- [backup-strategy.md](backup-strategy.md) — what is
backed up, by which tool, on what schedule, and how to restore.
@@ -7,97 +12,12 @@ backed up, by which tool, on what schedule, and how to restore.
service to ks4 (and getting it backed up automatically).
**Tech notes**
- [`nuc/`](nuc/) — home lab on `nuc` (see below)
- [`ks4/`](ks4/) — prod server `ks4` at OVH (see below)
- [`nuc/`](nuc/README.md) — home lab on `nuc`: host, storage, instances
- [`ks4/`](ks4/README.md) — prod server `ks4` at OVH: host, services, network flows
- [`ks2/`](ks2/) — legacy backup server being [decommissioned](ks2/plan.md)
- [`archer-c7/`](archer-c7/) — home router (TP-Link Archer C7 v5 running on OpenWrt)
## Homelab: nuc
Incus host on the LAN.
- Debian 13, Intel Alder Lake-N (iGPU `i915`, shared by both Jellyfin containers) —
bare-metal reinstall: [nuc/nuc-install.md](nuc/nuc-install.md)
- Instances are bridged onto the LAN (192.168.0.0/24)
- USB 4 TB WD Red: ZFS pool `usb4t` — `usb4t/backup` → `/backup`
(incus exports, `nuc/` + `ks4/` subdatasets, 1 TB quota) and
`usb4t/media` → `/srv/media` (media library, shared into containers
via `shift=true` disk devices; works because ZFS ≥ 2.2 supports
idmapped mounts)
- Backups: all local instances replicated to the USB pool
(`/root/scripts/incus-copy.sh -p backup -s nucbackup`; replicas
stopped, autostart off) — see
[nuc/nuc-install.md](nuc/nuc-install.md); ks4 replicas pulled into
pool `ks4backup` — see [ks4/incus-copy.md](ks4/incus-copy.md)
### Instances
| Name | IP | Doc | Features |
|---|---|---|---|
| [jellyfin-server](nuc/jellyfin-server.md) | 192.168.0.5 | ✅ | unprivileged, autostart; iGPU render node (`gpu` device, render gid) for QSV/VAAPI transcoding; `/srv/media` disk device (`shift=true`); proxy device → host :8096 |
| [jellyfin-client](nuc/jellyfin-client.md) | 192.168.0.6 | ✅ | **privileged**, autostart; full iGPU (`gpu` device, gid 44) → HDMI kiosk (cage + Jellyfin Media Player); custom `raw.lxc` (bind `/dev/snd`, `/dev/input`, host `/run/udev`); Pioneer USB audio as ALSA default; FR keymap; go-librespot Spotify Connect ("Pioneer A-70") |
| [transmission-bt](nuc/transmission-bt.md) | 192.168.0.7 | ✅ | unprivileged, autostart; always-on WireGuard full tunnel → ks4 (egress = 193.70.35.17, kill switch: no default route); `/srv/media` disk device (`shift=true`), downloads to `/media/downloads`; web UI :9091 (LAN only) |
| blocky | 192.168.0.254 | — | unprivileged, autostart; DNS ad-blocker |
| privoxy | 192.168.0.11 | — | unprivileged, autostart; filtering HTTP proxy |
| homeassistant | (stopped) | — | **virtual machine**, 50 GiB root disk on pool `data` |
## Prod: ks4
Incus host at OVH — public-facing self-hosted services.
- Debian 13, Xeon D-1521 / 32 GiB, 2× 6 TB (OS on mdraid RAID1,
ZFS pool `data` on `sda5`) — setup & rebuild:
[ks4/install.md](ks4/install.md)
- SSH: `ssh -i id_rsa_claude -p 2233 julien@193.70.35.17`
(`julien` has passwordless `sudo incus …`)
- Instances are on a NAT bridge `incusbr0` (192.168.1.0/24); public
ingress via Incus proxy devices. `gateway` (nginx) is the sole HTTP/S
entry point and fans out to the app containers.
- Services: gateway, mail, nextcloud, seafile, git, freshrss, bitwarden,
outline, login (SSO), wireguard, + more — full table in
[ks4/install.md](ks4/install.md).
- ⚠️ The ZFS `data` pool is single-disk (not mirrored); durability
rests on nightly cron jobs — 01:00 `incus copy --refresh` of all
instances to the local `backup` pool on sdb5, then the instance leg
to S3; 05:00 restic (DB dumps + data trees) to S3; nuc pulls the
replicas over WireGuard. Full picture and restore procedures:
**[backup-strategy.md](backup-strategy.md)**
([ks4/local-backup-cron.md](ks4/local-backup-cron.md),
[ks4/incus-copy.md](ks4/incus-copy.md),
[ks4/restic-backup.md](ks4/restic-backup.md)).
## Network flows
```
nuc — home LAN 192.168.0.0/24 ks4 — OVH 193.70.35.17
+-----------------------------------+ +-------------------------------------+
| | | |
| host: wg-ks4 (10.8.0.20) | | [wireguard] 192.168.1.18 |
| incus remote "ks4" ------+--WG-->| wg0 10.8.0.1/24, udp 51845 |
| pull ks4:* -> pool ks4backup | udp | | masquerade -> eth0 |
| on usb4t [pending FTTH seed] | 51845 | | |
| | | +-> incus API 192.168.1.1:8443 |
| [transmission-bt] wg0 (10.8.0.21) | | | (ufw: only from .18) |
| full tunnel 0.0.0.0/0 ------+--WG-->| | |
| kill switch: no default route | udp | +-> WAN egress: torrents + |
| downloads -> /srv/media | 51845 | apt of transmission-bt |
| (usb4t/media, read by jellyfin) | | exit as 193.70.35.17 |
| | | |
| 03:00 instance snapshots | | 03:00 instance snapshots |
| 03:30 incus-copy: all instances | | 01:00 incus-copy: all instances |
| -> project backup, pool | | -> project backup, zpool sdb5 |
| nucbackup (usb4t/backup/nuc) | | then restic instance leg -> S3 |
| 05:00 pull ks4:* -> ks4backup | | 05:00 restic: DB dumps + data |
| [pending FTTH] | | trees -> S3 (restic-data) |
| 05:30 apt upgrade all containers | | Sun 14:00 restic maintenance |
+-----------------------------------+ +-------------------------------------+
phones/laptops: WG peers 10.8.0.2-3 reach 192.168.1.x through the same endpoint
```
Both tunnels initiate **from** nuc (home NAT, dynamic IP) toward ks4's
fixed endpoint; ks4's incus API is never exposed to the internet.
### Conventions
## Conventions
- One markdown file per topic in the correct subdir, containing the full
implementation notes, configuration, and troubleshooting notes.
@@ -107,4 +27,5 @@ fixed endpoint; ks4's incus API is never exposed to the internet.
- Adding a service to ks4: follow
[new-container.md](new-container.md) (no Docker in containers —
incus runs OCI images natively and `incus-compose` handles stacks;
data paths go into `scripts/restic-paths`).
data paths go into
[`scripts/restic-paths`](https://git.lutran.fr/julien/scripts/src/branch/main/restic-paths)).
backup-strategy.md
+10 -2
View File
@@ -13,7 +13,15 @@ docs; this page is the map.
plakar was evaluated first and replaced (its nightly cost scaled with
tree size — [PlakarKorp/plakar#2338](https://github.com/PlakarKorp/plakar/issues/2338));
the old backup server **ks2 is retired** ([ks2/plan.md](ks2/plan.md)).
the old backup server **ks2 is being retired** (decommission by
Sep 30, 2026 — [ks2/plan.md](ks2/plan.md)): it still holds an ageing
copy of the instances until the nuc leg below takes over.
**Status 2026-08-28**: the local replica leg and the S3 data leg are
live. The **nuc pull leg is waiting for FTTH** (expected before end of
September) and the **S3 instance leg is written but not yet seeded** —
until both land, instances have only the ks4-local replica off the
live pool.
## The map
@@ -95,4 +103,4 @@ without the passphrase the S3 buckets are unreadable.
New instances are picked up automatically by the replica and
instance legs (all instances, opt-out only). Databases are
auto-discovered. Only **data trees** need one line in
`/root/scripts/restic-paths` — see [new-container.md](new-container.md).
[`restic-paths`](https://git.lutran.fr/julien/scripts/src/branch/main/restic-paths) — see [new-container.md](new-container.md).
ks2/decommission.md
+13 -11
View File
@@ -6,8 +6,8 @@ few days of margin.
## Preconditions (the plan's release gates)
- [ ] plakar S3 leg running nightly for ≥ a week, `done (rc=0)`,
restore test passed ([plakar-s3-data.md](../ks4/plakar-s3-data.md) §7)
- [ ] restic S3 leg running nightly for ≥ a week, `done (rc=0)`,
restore test passed ([restic-backup.md](../ks4/restic-backup.md) §7)
- [ ] nuc pull leg seeded and one instance test-restored
([nuc-seed.md](nuc-seed.md))
- [ ] local leg (sdb5) cron green in `/var/log/incus-copy.log`
@@ -15,7 +15,8 @@ few days of margin.
## 1. Cut the last flows to ks2 (root on ks4)
```sh
crontab -e # delete the 04:00 incus-backup.sh line (164.132.173.57)
crontab -e # remove any remaining ks2 line (the 02:00 push, if it was
# re-enabled; the 04:00 incus-backup.sh line is long gone)
incus remote list # if a "ks2" remote is still defined:
incus remote remove ks2
# drop the rsyncbackup SSH key if it exists only for ks2:
@@ -25,15 +26,15 @@ ls /root/.ssh/id_rsyncbackup* && rm -i /root/.ssh/id_rsyncbackup*
## 2. Last look at ks2 before wiping (root on ks2)
```sh
# anything in the rsync tree newer than the plakar cutover?
# anything in the rsync tree newer than the S3 cutover?
find /backup/ns3061243 -newermt '2026-08-23' | head
# pre-2023 dirs (catc, mythoughts, qcm, scripts): instances still
# exist stopped on ks4 — spot-check nothing unique remains:
ls /backup/ns3061243/{catc,mythoughts,qcm,scripts}
```
Archive anything unique to the plakar dump dir on ks4 (it rides the
nightly `dumps` snapshot to S3 for free once it stops changing).
Archive anything unique into the restic dump dir on ks4
(`/backup/dumps`) — it then rides the nightly S3 snapshot for free.
## 3. Wipe and terminate
@@ -52,9 +53,10 @@ renewal date.
durability bullet and the index line for `ks2/`s "being
decommissioned" wording (keep the folder as history)
- [ ] Retire `scripts/incus-backup.sh` + `incus-backup.db` in the repo
(superseded by `scripts/plakar-backup.sh`) — keep for reference,
note the retirement in their headers
- [ ] `login` container cleanup parked in
[plakar-s3-data.md](../ks4/plakar-s3-data.md): `dropdb outline`,
remove native postgres
(superseded by [`restic-backup.sh`](https://git.lutran.fr/julien/scripts/src/branch/main/restic-backup.sh)) —
keep for reference, note the retirement in their headers
- [ ] `login` container cleanup: the native postgres still holds the
pre-migration `outline` DB — `dropdb outline`, remove the
package (noted in
[plakar-s3-data.md](../ks4/plakar-s3-data.md) §3)
- [ ] Mark the ks2 line in `plan.md` done, date it
ks2/nuc-seed.md
+1 -1
View File
@@ -29,7 +29,7 @@ Notes:
## Cron (after the seed)
Add to nuc's root crontab, offset from the 03:30 local nucbackup copy
and ks4's own 01:00/04:30 jobs:
and ks4's own 01:00/05:00 jobs:
```cron
0 5 * * * /root/scripts/incus-copy.sh -r ks4 -s ks4backup >> /var/log/incus-copy-ks4.log 2>&1
ks2/plan.md
+29 -27
View File
@@ -4,20 +4,23 @@ Goal: release `ks2.lutran.fr` (`164.132.173.57`, SSH :2233, host
`ns3247221`) once ks4 has a real 3-2-1 backup without it.
**Deadline: rental ends Sep 30, 2026.**
## What ks2 does today (updated 2026-08-22)
## What ks2 does today (updated 2026-08-28)
One remaining role, fed nightly by a root cron **on ks4**:
**Nothing is written to ks2 any more.** Both feeds are retired: the
`01:00` replica push (replaced by the local leg on ks4's sdb5 pool,
[local-backup-cron.md](../ks4/local-backup-cron.md)) and the `04:00`
`incus-backup.sh` rsync (replaced by restic → S3,
[restic-backup.md](../ks4/restic-backup.md)).
- `04:00` `incus-backup.sh … -d 164.132.173.57` — DB dumps + selected
paths rsync'd to ks2 `/backup/ns3061243` on pool `backup` (1.59 T,
88 % full), snapshotted daily on ks2 (`zfs-auto-snapshot.sh`,
2-month expiry). **Kept as a stopgap** until the plakar S3 leg
replaces it.
What remains on the box is **cold history**: instance replicas on pool
`data` (last refreshed 2026-08-09) and the rsync tree
`/backup/ns3061243` on pool `backup` (last refreshed 2026-08-28),
snapshotted daily there (`zfs-auto-snapshot.sh`, 2-month expiry).
Retired: the `01:00` replica push to ks2 pool `data` — replaced by
the local leg on ks4's sdb5 pool
([local-backup-cron.md](../ks4/local-backup-cron.md)); `biwiki` and `spot`
deleted from ks2.
⚠️ Consequence while the nuc leg waits for FTTH: **instances have no
fresh off-site copy** — only the ks4-local sdb replicas plus ks2's
ageing ones. Either re-enable the interim push (below) or accept the
gap knowingly until FTTH.
## Inventory findings (2026-08-22)
@@ -25,8 +28,9 @@ deleted from ks2.
project — **decision: both abandoned**, safe to delete from backups.
- `incus-backup.db` is stale (lists `spot`, misses `livetrail`,
`outline`, `login`, …) — forgotten-manifest drift is exactly what
the plakar-based replacement must eliminate (back up *all*
instances by default, opt-out instead of opt-in).
the replacement had to eliminate — restic's drivers back up *all*
instances and auto-discover *all* databases, opt-out instead of
opt-in.
- `livetrail` (running on ks4) is **absent from ks4's local `backup`
project** — the local leg was run manually once and never again;
automating its cron fixes this.
@@ -38,17 +42,17 @@ deleted from ks2.
`qcm`) — those instances still exist stopped on ks4, so nothing
unique expected there; spot-check before wiping.
- Losing ks2 also loses its 2 months of rsync-snapshot history —
acceptable once plakar has built equivalent retention.
acceptable: restic keeps 14 daily / 8 weekly / 6 monthly.
## Target architecture (3-2-1 for ks4)
| Leg | Mechanism | Status |
|---|---|---|
| local, 2nd disk | `incus-copy.sh -p backup -s backup` → sdb5 pool | works, **cron missing** |
| off-site, nuc | nuc pulls `ks4:*` → pool `ks4backup` over WG | waiting FTTH seed |
| off-site, S3 (data+DB) | **restic** on ks4 → S3 (plakar ran it first, being replaced — see [restic-backup.md](../ks4/restic-backup.md)) | migrating |
| off-site, S3 (instances) | restic over `incus file mount` of the `backup`-project replicas | designed |
| off-site, ks2 (interim) | 02:00 replica push until the nuc leg seeds (FTTH < Sep 30) | to re-enable |
| local, 2nd disk | `incus-copy.sh -p backup -s backup` → sdb5 pool | **live** (01:00) |
| off-site, S3 (data+DB) | **restic** → bucket `restic-data` ([restic-backup.md](../ks4/restic-backup.md)) | **live** (05:00) |
| off-site, S3 (instances) | restic over `incus file mount` of the `backup`-project replicas | written, **not seeded** |
| off-site, nuc | nuc pulls `ks4:*` → pool `ks4backup` over WG ([nuc-seed.md](nuc-seed.md)) | waiting FTTH (< Sep 30) |
| off-site, ks2 (optional interim) | `incus-copy.sh -d ks2 -m push` at 02:00 until the nuc leg seeds | **not enabled** — decide |
## Actions (backup work documented in [`../ks4/`](../ks4/), ks2-only tasks here)
@@ -57,13 +61,11 @@ deleted from ks2.
2. ~~[local-backup-cron.md](../ks4/local-backup-cron.md)~~ — **done
2026-08-22**: leg 1 cron re-enabled, ks2 rsync kept as stopgap;
`livetrail` verified present in the `backup` project
3. [plakar-s3-data.md](../ks4/plakar-s3-data.md) — **seeding S3
2026-08-22** (direct-to-S3 kloset `lutran-ks4-plakar-data`, driver
script `scripts/plakar-backup.sh`, pipeline validated locally);
next: restore test + cron
4. [plakar-incus-integration.md](../ks4/plakar-incus-integration.md) —
design seeded 2026-08-22: per-file importer over the incus sftp
API, instances → S3
3. ~~plakar S3 leg~~ → **replaced by restic**
([restic-backup.md](../ks4/restic-backup.md), live 2026-08-28;
plakar's own doc kept as history)
4. instance leg — restic over `incus file mount`
([restic-backup.md](../ks4/restic-backup.md) §6): **seed pending**
5. [nuc-seed.md](nuc-seed.md) — **prepared**; after FTTH: seed nuc
pull leg, verify all instances, test-restore one
6. [decommission.md](decommission.md) — **prepared**; cut flows,
@@ -74,6 +76,6 @@ deleted from ks2.
- [x] biwiki + spot consciously abandoned (2026-08-22)
- [ ] local leg cron running ≥ a few days, all instances present
- [ ] nuc leg fully seeded **and** one instance test-restored
- [ ] plakar S3 data/DB backups running daily **and** one DB + one
- [ ] restic S3 data/DB backups running daily **and** one DB + one
file tree test-restored
- [ ] ks4 crons pointing at ks2 disabled
ks4/README.md
+55
View File
@@ -0,0 +1,55 @@
# Prod: ks4
Incus host at OVH — public-facing self-hosted services.
- Debian 13, Xeon D-1521 / 32 GiB, 2× 6 TB (OS on mdraid RAID1,
ZFS pool `data` on `sda5`) — setup & rebuild:
[install.md](install.md)
- SSH: `ssh -i id_rsa_claude -p 2233 julien@193.70.35.17`
(`julien` has passwordless `sudo incus …`)
- Instances are on a NAT bridge `incusbr0` (192.168.1.0/24); public
ingress via Incus proxy devices. `gateway` (nginx) is the sole HTTP/S
entry point and fans out to the app containers.
- Services: gateway, mail, nextcloud, seafile, git, freshrss, bitwarden,
outline, login (SSO), wireguard, + more — full table in
[install.md](install.md).
- ⚠️ The ZFS `data` pool is single-disk (not mirrored); durability
rests on nightly cron jobs — 01:00 `incus copy --refresh` of all
instances to the local `backup` pool on sdb5, then the instance leg
to S3; 05:00 restic (DB dumps + data trees) to S3; nuc pulls the
replicas over WireGuard. Full picture and restore procedures:
**[backup-strategy.md](../backup-strategy.md)**
([local-backup-cron.md](local-backup-cron.md),
[incus-copy.md](incus-copy.md),
[restic-backup.md](restic-backup.md)).
## Network flows (nuc ↔ ks4)
```
nuc — home LAN 192.168.0.0/24 ks4 — OVH 193.70.35.17
+-----------------------------------+ +-------------------------------------+
| | | |
| host: wg-ks4 (10.8.0.20) | | [wireguard] 192.168.1.18 |
| incus remote "ks4" ------+--WG-->| wg0 10.8.0.1/24, udp 51845 |
| pull ks4:* -> pool ks4backup | udp | | masquerade -> eth0 |
| on usb4t [pending FTTH seed] | 51845 | | |
| | | +-> incus API 192.168.1.1:8443 |
| [transmission-bt] wg0 (10.8.0.21) | | | (ufw: only from .18) |
| full tunnel 0.0.0.0/0 ------+--WG-->| | |
| kill switch: no default route | udp | +-> WAN egress: torrents + |
| downloads -> /srv/media | 51845 | apt of transmission-bt |
| (usb4t/media, read by jellyfin) | | exit as 193.70.35.17 |
| | | |
| 03:00 instance snapshots | | 03:00 instance snapshots |
| 03:30 incus-copy: all instances | | 01:00 incus-copy: all instances |
| -> project backup, pool | | -> project backup, zpool sdb5 |
| nucbackup (usb4t/backup/nuc) | | then restic instance leg -> S3 |
| 05:00 pull ks4:* -> ks4backup | | 05:00 restic: DB dumps + data |
| [pending FTTH] | | trees -> S3 (restic-data) |
| 05:30 apt upgrade all containers | | Sun 14:00 restic maintenance |
+-----------------------------------+ +-------------------------------------+
phones/laptops: WG peers 10.8.0.2-3 reach 192.168.1.x through the same endpoint
```
Both tunnels initiate **from** nuc (home NAT, dynamic IP) toward ks4's
fixed endpoint; ks4's incus API is never exposed to the internet.
ks4/incus-copy.md
+5 -3
View File
@@ -1,7 +1,7 @@
# incus-copy — ks4 instance replication
Incremental replication of all ks4 Incus instances via
[`scripts/incus-copy.sh`](../scripts/incus-copy.sh)
[`scripts/incus-copy.sh`](https://git.lutran.fr/julien/scripts/src/branch/main/incus-copy.sh)
(`incus copy --refresh`: ZFS-incremental, produces ready-to-start
replicas instead of tarballs). Two legs replace the historical nightly
push to `ks2` (decommissioning):
@@ -35,7 +35,7 @@ incus-copy.sh -p <dest-project> -s <dest-pool> # local
`--refresh-exclude-older` keeps expired snapshots from being re-sent.
- Consistency: copies of running instances are crash-consistent;
databases may need recovery on restore. Snapshot history mitigates,
and [`incus-backup.sh`](../scripts/incus-backup.sh) (proper
and [`incus-backup.sh`](https://git.lutran.fr/julien/scripts/src/branch/main/incus-backup.sh) (proper
`mariadb-dump`s + selected paths, manifest-driven) covers the DBs.
- **VMs**: `--refresh` re-sends a VM's **full block volume** every run
unless source and replica share a common snapshot — containers diff
@@ -63,7 +63,9 @@ Cron (root on ks4) — replaces both ks2 jobs:
```cron
0 1 * * * /root/scripts/incus-copy.sh -p backup -s backup >> /var/log/incus-copy.log 2>&1
0 4 * * * /root/scripts/incus-backup.sh -f /root/scripts/incus-backup.db -s data >> /var/log/incus-backup.log 2>&1
# retired 2026-08-28 — dumps + data trees are restic's job (05:00),
# see ../backup-strategy.md and restic-backup.md
#0 4 * * * /root/scripts/incus-backup.sh -f /root/scripts/incus-backup.db -s data >> /var/log/incus-backup.log 2>&1
```
(`incus-backup.sh` without `-d` now writes locally to
ks4/install.md
+22 -12
View File
@@ -142,22 +142,32 @@ All in the `default` project, on the `default` profile
## Backups
> **Migration in progress** (2026-08-09): the `ks2` destination below is
> being replaced by a local `backup` zpool on `sdb5` + off-site pull
> from nuc — plan and setup in [incus-copy.md](incus-copy.md). The
> cron blocks here describe what runs **today**; update on cutover.
Two **root** cron jobs (this repo's [`scripts/`](../scripts/) are the
copies that run here) — the primary safety net for the non-mirrored
`data` pool:
Full picture, schedule and restore procedures:
**[../backup-strategy.md](../backup-strategy.md)**. Drivers live in the
[scripts repo](https://git.lutran.fr/julien/scripts) and are deployed
to `/root/scripts`. Current **root** cron jobs — the safety net for the
non-mirrored `data` pool:
```cron
0 1 * * * /root/scripts/incus-copy.sh -d ks2 -m push
0 4 * * * /root/scripts/incus-backup.sh -d 164.132.173.57 -u rsyncbackup \
-i /root/.ssh/id_rsyncbackup -p 2233 \
-f /root/scripts/incus-backup.db -s data
0 1 * * * /root/scripts/incus-copy.sh -p backup -s backup >> /var/log/incus-copy.log 2>&1
0 5 * * * /root/scripts/restic-backup.sh >> /var/log/restic-backup.log 2>&1
0 14 * * 0 /root/scripts/restic-maintenance.sh >> /var/log/restic-maintenance.log 2>&1
```
- **01:00 `incus-copy.sh`** → replicas of every instance in project
`backup` on the sdb5 pool ([incus-copy.md](incus-copy.md),
[local-backup-cron.md](local-backup-cron.md)); the S3 instance leg
chains here once seeded.
- **05:00 `restic-backup.sh`** → auto-discovered DB dumps + the data
trees listed in
[`scripts/restic-paths`](https://git.lutran.fr/julien/scripts/src/branch/main/restic-paths),
into the `restic-data` bucket ([restic-backup.md](restic-backup.md)).
- **Sun 14:00 `restic-maintenance.sh`** → prune + integrity checks.
Historical (retired 2026-08): the `ks2` push and `incus-backup.sh`
rsync leg, and the plakar data leg
([plakar-s3-data.md](plakar-s3-data.md)).
- **01:00 — `incus-copy.sh` → `ks2`**: differential `incus copy … \
--refresh --refresh-exclude-older --mode push` of every *running*
instance to remote incus host `ks2` (a near-live replica). `ks2` must
ks4/local-backup-cron.md
+10 -15
View File
@@ -1,28 +1,23 @@
# local backup cron on ks4 (leg 1) — re-enabled
# local backup cron on ks4 (leg 1)
Done 2026-08-22. Context: **both** ks4 backup crons had been commented
out — no backup of any kind was running.
Enabled 2026-08-22 (both ks4 backup crons had been commented out — no
backup of any kind was running). Overview of all legs:
[../backup-strategy.md](../backup-strategy.md).
## What runs now (root crontab on ks4)
```cron
0 1 * * * /root/scripts/incus-copy.sh -p backup -s backup >> /var/log/incus-copy.log 2>&1
0 4 * * * /root/scripts/incus-backup.sh -d 164.132.173.57 -u rsyncbackup -i /root/.ssh/id_rsyncbackup -p 2233 -f /root/scripts/incus-backup.db -s data >> /var/log/incus-backup.log 2>&1
```
- `01:00` — **local leg**: `incus copy --refresh` of all instances
into project `backup` on the sdb5 `backup` zpool (replaces the old
push to ks2; see [incus-copy.md](incus-copy.md)).
- `04:00` — **stopgap**: `incus-backup.sh` rsync to ks2 kept alive
(manifest pruned of `spot`) until the plakar S3 leg replaces it —
then this line goes away as part of
[decommissioning ks2](../ks2/plan.md). ⚠️ keep
`scripts/incus-backup.db` in this repo in sync with
`/root/scripts/incus-backup.db` on ks4.
Both were also run once by hand (in a `screen`) on 2026-08-22 to
refresh the backups immediately. The old push-to-ks2 copy line stays
commented out — replica refreshes to ks2 are retired.
push to ks2; see [incus-copy.md](incus-copy.md)). The S3 instance
leg ([restic-backup.md](restic-backup.md) §6) chains onto this same
line once seeded — it reads the replicas this job just refreshed.
- The rsync stopgap to ks2 (`incus-backup.sh`, 04:00) and the plakar
data leg (04:30) are both **retired**; S3 backups are restic's job
at 05:00.
## Cleanup done the same day
ks4/plakar-incus-integration.md
+11 -1
View File
@@ -1,4 +1,14 @@
# plakar incus integration — design notes
# plakar incus integration — design notes (RETIRED)
> **Historical.** The importer works (smoke-tested against ks4's
> `backup` project) but production moved to restic, which needs no
> plugin: `incus file mount` exposes any instance — including stopped
> replicas — over the same per-instance sftp API
> ([restic-backup.md](restic-backup.md) §6). Code lives on at
> <https://github.com/jlutran/plakar-integration-incus>; the analysis
> below (per-file vs image, replica-vs-live consistency, the
> seek-bound wall) is what shaped the restic design.
Status: **importer scaffolded and compiling 2026-08-23** in
`perso/plakar/integration-incus` (walker modeled on the official sftp
ks4/plakar-s3-data.md
+13 -2
View File
@@ -1,4 +1,15 @@
# plakar data + DB backups on ks4 (leg 3)
# plakar data + DB backups on ks4 (leg 3) — SUPERSEDED
> **Historical.** This leg ran 2026-08-22 → 08-28 and was replaced by
> restic: see **[restic-backup.md](restic-backup.md)**. plakar's
> incremental cost scaled with tree size rather than churn
> ([PlakarKorp/plakar#2338](https://github.com/PlakarKorp/plakar/issues/2338)).
> The `plakar-data` kloset and the plakar install are deliberately kept
> until that issue concludes (the report offers debug runs against that
> repository); the `plakar-incus` bucket can be deleted. Kept here for
> the measurements and the design rationale, which the restic doc
> builds on.
Status: **live since 2026-08-24** — S3 kloset
(`lutran-ks4-plakar-data`) seeded (dumps + 9 fs sources), restore
@@ -19,7 +30,7 @@ leg 1's job. Direct-to-S3 keeps the incremental behavior — the
skip-unchanged baseline is the parent snapshot's VFS read from the
kloset (plakar caches repository metadata locally; v1.1.4's dirpack
prefetcher targets exactly this remote-backend walk). Driver script:
[`scripts/plakar-backup.sh`](../../scripts/plakar-backup.sh)
[`scripts/plakar-backup.sh`](https://git.lutran.fr/julien/scripts/src/branch/main/plakar-backup.sh)
(flock, incus DB dumps, iterates a sources file, prune + maintenance;
its `-k` takes any `plakar at` target, so `-k @s3` — a throwaway
local kloset, e.g. `-k /backup/plakar-test`, works for testing before
ks4/restic-backup.md
+23 -20
View File
@@ -1,10 +1,12 @@
# restic backups on ks4 — the S3 leg
Status: **decided 2026-08-25 — restic replaces plakar as the S3
backup tool** (evaluation first: parallel run + comparison, see §8).
Final architecture: **two tools** — `incus copy` for replication
(local sdb, nuc after FTTH, ks2 push as interim), **restic** for
everything S3.
Status: **live since 2026-08-28** — restic replaced plakar as the S3
backup tool (plakar crons off, its kloset kept for
[#2338](https://github.com/PlakarKorp/plakar/issues/2338); see §8).
Architecture: **two tools** — `incus copy` for replication (local sdb,
nuc after FTTH), **restic** for everything S3. Data leg (`restic-data`)
runs nightly at 05:00; the instance leg (`restic-incus`, §6) is written
and awaits its first seed.
Why: plakar's incremental cost scales with tree size, not churn
(~800 MiB parent metadata re-read from S3 + hours of CPU per night on
@@ -58,9 +60,10 @@ property).
### Memory (the real constraint on this 23/31 GiB-used box)
- Index RAM ≈ 200–300 B/blob → combined `restic-data` index
~1–1.5 GiB per operation (nextcloud ~1 M blobs + seafile ~2–4 M);
prune peaks 2–3×. Accepted with the merged layout (§1) — measure
peak RSS during the evaluation week (`/usr/bin/time -v`).
~1 GiB per operation; prune peaks 2–3×. The seafile side of that
estimate collapsed after its garbage collection (24.9 M → 0.5 M
objects, [seafile-gc.md](seafile-gc.md)). Measure peak RSS with
`/usr/bin/time -v` if the box ever feels tight.
- `GOGC=20` trades CPU for a smaller Go heap — enable if the seafile
prune ever pressures the box.
@@ -140,7 +143,7 @@ for r in restic-data restic-incus; do
done
```
## 4. Seed plan
## 4. Seed plan (executed 2026-08-25 → 08-28)
1. **ARC floor first** (helps every walk on the box):
```sh
@@ -252,18 +255,18 @@ restic -r s3:…/restic-data restore latest --target /backup/restore-test --incl
diff -r … && rm -rf /backup/restore-test
```
## 8. Evaluation protocol + plakar retirement
## 8. Cutover (done) + what is left
- **Parallel week**: plakar 04:30 cron keeps running untouched;
restic runs at 05:00 on the same data. Same nights, same churn →
direct comparison (wall, bytes read from store, bytes written, peak
RSS via `/usr/bin/time -v`) against plakar's measured table in
[plakar-s3-data.md](plakar-s3-data.md). Restore-speed compared on
the same 10 GiB dumps tree (plakar measured: 39 min).
- **Cutover** (after the comparison confirms + restore test passes):
drop the plakar 04:30 + Sunday-check crons; 01:00 becomes
`incus-copy ; restic instances phase` (or stays in the 05:00 run).
Update plan.md legs table.
- **Cutover 2026-08-28**: plakar's 04:30 and Sunday crons are off;
restic's 05:00 data leg is the S3 backup. The formal parallel-week
comparison was cut short — the evidence had already settled it
(plakar: ~800 MiB metadata re-read and 3–4 h per night on the
nextcloud tree, churn-independent; restic: seed at 60–115 MiB/s,
local-cache incrementals). plakar's measurements stay in
[plakar-s3-data.md](plakar-s3-data.md).
- **Left to do**: seed `restic-incus` (§6) and chain it onto the 01:00
line; restore test on the complete `restic-data`; then the ks2
decommission gates ([../ks2/plan.md](../ks2/plan.md)).
- **plakar retirement, one exception**: keep plakar installed and
keep the `plakar-data` kloset until
[#2338](https://github.com/PlakarKorp/plakar/issues/2338) concludes
ks4/seafile-gc.md
+11 -5
View File
@@ -21,8 +21,8 @@ current state → new library → delete old → GC `-r`, which works for
takes the dataset to ~600 k inodes, at which point per-file backup of
seafile becomes ordinary.
All crontab backup jobs on ks4 remain commented out until restart +
validation completes.
(The ks4 backup crons were paused during the GC window and are back
on since 2026-08-27.)
## Root cause of the 24.9 M-inode / 933 G dataset
@@ -73,9 +73,15 @@ setting (the existing per-library 30 d row stays). Users can still
override per library in the web UI if
`ENABLE_REPO_HISTORY_SETTING = True` in `seahub_settings.py`.
### 2. GC — offline, maintenance window
### 2. GC — maintenance window
CE's GC requires seafile stopped. On this pathological store expect
⚠️ Superseded by experience: with the **MySQL backend seafile runs
*online* GC** (the runs print "use online GC"), so stopping the
services is not required — the recurring cron below does not. The
2026-08-26 run stopped them anyway, out of caution. What follows is
that run's procedure.
CE's GC traditionally requires seafile stopped. On this pathological store expect
**hours** (it traverses fs/commits on a cold HDD); the dry-run sizes
the window. Inside the container:
@@ -115,7 +121,7 @@ du -sh .../seafile/rootfs/opt/seafile/seafile-data/storage/blocks # was 867 G
0 6 15 1,4,7,10 * incus exec seafile -- sh -c "cd /opt/seafile/seafile-server-latest && ./seaf-gc.sh && ./seaf-gc.sh -r && ./seaf-gc.sh -R" >> /var/log/seafile-gc.log 2>&1
```
At the post-cleanup scale (~120 k live files) the full sequence is
minutes, not hours. Keep it clear of the 01:00/04:30/05:00 backup
minutes, not hours. Keep it clear of the 01:00/05:00 backup
slots.
- 2026-08-27 follow-ups executed: 8 old libraries deleted by user →
trash emptied → `seaf-gc -r` purged their stores (18 m, blocks now
new-container.md
+4 -4
View File
@@ -75,7 +75,7 @@ them forces a full re-send of its whole disk every night.
**Manual, one line:** data that lives in the filesystem (uploads,
repositories, mail spools…) must be listed in
`/root/scripts/restic-paths`, one absolute host path per line:
[`restic-paths`](https://git.lutran.fr/julien/scripts/src/branch/main/restic-paths), one absolute host path per line:
```
/var/lib/incus/storage-pools/data/containers/myservice/rootfs/var/lib/myservice
@@ -84,7 +84,7 @@ repositories, mail spools…) must be listed in
(The container's `/x` is `…/containers/<name>/rootfs/x` on the host.)
No comments in that file — every line is read as a path. Regenerable
caches can be skipped by adding a pattern to
`/root/scripts/restic-exclude`. Both files live in the `scripts` git
[`restic-exclude`](https://git.lutran.fr/julien/scripts/src/branch/main/restic-exclude). Both files live in the `scripts` git
repo: edit there, commit, `git pull` on ks4.
For the database auto-discovery to work, keep the defaults:
@@ -98,8 +98,8 @@ For the database auto-discovery to work, keep the defaults:
"postgres" (official images do).
A database you do *not* want dumped (huge, static, re-importable)
goes into `/root/scripts/plakar-db-exclude` as `<instance>/<db>` —
every skip is logged, so the list cannot rot silently.
goes into `/root/scripts/db-exclude` as `<instance>/<db>` — every
skip is logged, so the list cannot rot silently.
## 5. Verify, once
nuc/README.md
+28
View File
@@ -0,0 +1,28 @@
# Homelab: nuc
Incus host on the LAN.
- Debian 13, Intel Alder Lake-N (iGPU `i915`, shared by both Jellyfin containers) —
bare-metal reinstall: [nuc-install.md](nuc-install.md)
- Instances are bridged onto the LAN (192.168.0.0/24)
- USB 4 TB WD Red: ZFS pool `usb4t` — `usb4t/backup` → `/backup`
(incus exports, `nuc/` + `ks4/` subdatasets, 1 TB quota) and
`usb4t/media` → `/srv/media` (media library, shared into containers
via `shift=true` disk devices; works because ZFS ≥ 2.2 supports
idmapped mounts)
- Backups: all local instances replicated to the USB pool
(`/root/scripts/incus-copy.sh -p backup -s nucbackup`; replicas
stopped, autostart off) — see
[nuc-install.md](nuc-install.md); ks4 replicas pulled into
pool `ks4backup` — see [ks4/incus-copy.md](../ks4/incus-copy.md)
## Instances
| Name | IP | Doc | Features |
|---|---|---|---|
| [jellyfin-server](jellyfin-server.md) | 192.168.0.5 | ✅ | unprivileged, autostart; iGPU render node (`gpu` device, render gid) for QSV/VAAPI transcoding; `/srv/media` disk device (`shift=true`); proxy device → host :8096 |
| [jellyfin-client](jellyfin-client.md) | 192.168.0.6 | ✅ | **privileged**, autostart; full iGPU (`gpu` device, gid 44) → HDMI kiosk (cage + Jellyfin Media Player); custom `raw.lxc` (bind `/dev/snd`, `/dev/input`, host `/run/udev`); Pioneer USB audio as ALSA default; FR keymap; go-librespot Spotify Connect ("Pioneer A-70") |
| [transmission-bt](transmission-bt.md) | 192.168.0.7 | ✅ | unprivileged, autostart; always-on WireGuard full tunnel → ks4 (egress = 193.70.35.17, kill switch: no default route); `/srv/media` disk device (`shift=true`), downloads to `/media/downloads`; web UI :9091 (LAN only) |
| blocky | 192.168.0.254 | — | unprivileged, autostart; DNS ad-blocker |
| privoxy | 192.168.0.11 | — | unprivileged, autostart; filtering HTTP proxy |
| homeassistant | (stopped) | — | **virtual machine**, 50 GiB root disk on pool `data` |
nuc/nuc-install.md
+1 -1
View File
@@ -19,7 +19,7 @@ How to rebuild the Incus host from scratch if `/dev/sda` (512 GB SSD,
The ZFS pool `data` lives on `sda5` → **all instance root disks are lost**
with the OS. The USB pool `usb4t` (backups + media) survives — all
instances are replicated onto it with
[`scripts/incus-copy.sh`](../scripts/incus-copy.sh) (deployed at
[`scripts/incus-copy.sh`](https://git.lutran.fr/julien/scripts/src/branch/main/incus-copy.sh) (deployed at
`/root/scripts/` on nuc; set up 2026-08-09):
```sh