Compare commits
2
Commits
c18c8edc5b
...
a0e3a7dd86
| Author | SHA1 | Date | |
|---|---|---|---|
  Julien LutranandClaude Fable 5
|
a0e3a7dd86 | ||
|
Julien Lutran
|
80f0219716 |
@@ -1,120 +1,31 @@
|
||||
# perso
|
||||
# Personal infrastructure notes
|
||||
|
||||
Personal infrastructure notes.
|
||||
Two repositories back this setup:
|
||||
[**doc**](https://git.lutran.fr/julien/doc) (this one) and
|
||||
[**scripts**](https://git.lutran.fr/julien/scripts) (the cron drivers
|
||||
and their config files, deployed to `/root/scripts` on each host).
|
||||
|
||||
**Start here**: [backup-strategy.md](backup-strategy.md) — what is
|
||||
**Start here**:
|
||||
- [backup-strategy.md](backup-strategy.md) — what is
|
||||
backed up, by which tool, on what schedule, and how to restore.
|
||||
[new-container.md](new-container.md) — conventions for adding a
|
||||
- [new-container.md](new-container.md) — conventions for adding a
|
||||
service to ks4 (and getting it backed up automatically).
|
||||
|
||||
- [`nuc/`](nuc/) — home lab on `nuc` (see below)
|
||||
- [`ks4/`](ks4/) — prod server `ks4` at OVH (see below)
|
||||
- [`ks2/`](ks2/) — legacy backup server being decommissioned
|
||||
(rental ends Sep 30, 2026): [plan](ks2/plan.md)
|
||||
- [`archer-c7/`](archer-c7/) — home router (TP-Link Archer C7 v5,
|
||||
OpenWrt, `192.168.0.2`, WAN = phone USB tether):
|
||||
[upgrade to 25.12](archer-c7/upgrade-openwrt-25.12.md)
|
||||
**Tech notes**
|
||||
- [`nuc/`](nuc/README.md) — home lab on `nuc`: host, storage, instances
|
||||
- [`ks4/`](ks4/README.md) — prod server `ks4` at OVH: host, services, network flows
|
||||
- [`ks2/`](ks2/) — legacy backup server being [decommissioned](ks2/plan.md)
|
||||
- [`archer-c7/`](archer-c7/) — home router (TP-Link Archer C7 v5 running on OpenWrt)
|
||||
|
||||
## Homelab: nuc
|
||||
## Conventions
|
||||
|
||||
Incus host on the LAN.
|
||||
|
||||
- Host: `nuc` — `192.168.0.3`, Debian 13, Intel Alder Lake-N
|
||||
(iGPU `i915`, shared by both Jellyfin containers) —
|
||||
bare-metal reinstall: [nuc/nuc-install.md](nuc/nuc-install.md)
|
||||
- SSH: `ssh -i id_rsa_claude root@192.168.0.3` (keys at repo root, gitignored)
|
||||
- Instances are bridged onto the LAN (192.168.0.0/24)
|
||||
- USB 4 TB WD Red: ZFS pool `usb4t` — `usb4t/backup` → `/backup`
|
||||
(incus exports, `nuc/` + `ks4/` subdatasets, 1 TB quota) and
|
||||
`usb4t/media` → `/srv/media` (media library, shared into containers
|
||||
via `shift=true` disk devices; works because ZFS ≥ 2.2 supports
|
||||
idmapped mounts)
|
||||
- NAS: `//192.168.0.10/Shared` (CIFS) — original media source, normally
|
||||
offline; not mountable into containers (idmapped mounts unsupported
|
||||
on CIFS)
|
||||
- Backups: all local instances replicated to the USB pool
|
||||
(`/root/scripts/incus-copy.sh -p backup -s nucbackup`; replicas
|
||||
stopped, autostart off) — see
|
||||
[nuc/nuc-install.md](nuc/nuc-install.md); ks4 replicas pulled into
|
||||
pool `ks4backup` — see [ks4/incus-copy.md](ks4/incus-copy.md)
|
||||
|
||||
### Instances
|
||||
|
||||
| Name | IP | Doc | Features |
|
||||
|---|---|---|---|
|
||||
| [jellyfin-server](nuc/jellyfin-server.md) | 192.168.0.5 | ✅ | unprivileged, autostart; iGPU render node (`gpu` device, render gid) for QSV/VAAPI transcoding; `/srv/media` disk device (`shift=true`); proxy device → host :8096 |
|
||||
| [jellyfin-client](nuc/jellyfin-client.md) | 192.168.0.6 | ✅ | **privileged**, autostart; full iGPU (`gpu` device, gid 44) → HDMI kiosk (cage + Jellyfin Media Player); custom `raw.lxc` (bind `/dev/snd`, `/dev/input`, host `/run/udev`); Pioneer USB audio as ALSA default; FR keymap; go-librespot Spotify Connect ("Pioneer A-70") |
|
||||
| [transmission-bt](nuc/transmission-bt.md) | 192.168.0.7 | ✅ | unprivileged, autostart; always-on WireGuard full tunnel → ks4 (egress = 193.70.35.17, kill switch: no default route); `/srv/media` disk device (`shift=true`), downloads to `/media/downloads`; web UI :9091 (LAN only) |
|
||||
| blocky | 192.168.0.254 | — | unprivileged, autostart; DNS ad-blocker |
|
||||
| privoxy | 192.168.0.11 | — | unprivileged, autostart; filtering HTTP proxy |
|
||||
| homeassistant | (stopped) | — | **virtual machine**, 50 GiB root disk on pool `data` |
|
||||
|
||||
## Prod: ks4
|
||||
|
||||
Incus host at OVH — public-facing self-hosted services.
|
||||
|
||||
- Host: `ks4.lutran.fr` — `193.70.35.17`, **SSH on port 2233**,
|
||||
Debian 13, Xeon D-1521 / 32 GiB, 2× 6 TB (OS on mdraid RAID1,
|
||||
ZFS pool `data` on `sda5`) — setup & rebuild:
|
||||
[ks4/install.md](ks4/install.md)
|
||||
- SSH: `ssh -i id_rsa_claude -p 2233 julien@193.70.35.17`
|
||||
(`julien` has passwordless `sudo incus …`)
|
||||
- Instances are on a NAT bridge `incusbr0` (192.168.1.0/24); public
|
||||
ingress via Incus proxy devices. `gateway` (nginx) is the sole HTTP/S
|
||||
entry point and fans out to the app containers.
|
||||
- Services: gateway, mail, nextcloud, seafile, git, freshrss, bitwarden,
|
||||
outline, login (SSO), wireguard, + more — full table in
|
||||
[ks4/install.md](ks4/install.md).
|
||||
- ⚠️ The ZFS `data` pool is single-disk (not mirrored); durability
|
||||
rests on nightly cron jobs — 01:00 `incus copy --refresh` of all
|
||||
instances to the local `backup` pool on sdb5, then the instance leg
|
||||
to S3; 05:00 restic (DB dumps + data trees) to S3; nuc pulls the
|
||||
replicas over WireGuard. Full picture and restore procedures:
|
||||
**[backup-strategy.md](backup-strategy.md)**
|
||||
([ks4/local-backup-cron.md](ks4/local-backup-cron.md),
|
||||
[ks4/incus-copy.md](ks4/incus-copy.md),
|
||||
[ks4/restic-backup.md](ks4/restic-backup.md)).
|
||||
|
||||
## Network flows (nuc <-> ks4)
|
||||
|
||||
```
|
||||
nuc — home LAN 192.168.0.0/24 ks4 — OVH 193.70.35.17
|
||||
+-----------------------------------+ +-------------------------------------+
|
||||
| | | |
|
||||
| host: wg-ks4 (10.8.0.20) | | [wireguard] 192.168.1.18 |
|
||||
| incus remote "ks4" ------+--WG-->| wg0 10.8.0.1/24, udp 51845 |
|
||||
| pull ks4:* -> pool ks4backup | udp | | masquerade -> eth0 |
|
||||
| on usb4t [pending FTTH seed] | 51845 | | |
|
||||
| | | +-> incus API 192.168.1.1:8443 |
|
||||
| [transmission-bt] wg0 (10.8.0.21) | | | (ufw: only from .18) |
|
||||
| full tunnel 0.0.0.0/0 ------+--WG-->| | |
|
||||
| kill switch: no default route | udp | +-> WAN egress: torrents + |
|
||||
| downloads -> /srv/media | 51845 | apt of transmission-bt |
|
||||
| (usb4t/media, read by jellyfin) | | exit as 193.70.35.17 |
|
||||
| | | |
|
||||
| 03:00 instance snapshots | | 03:00 instance snapshots |
|
||||
| 03:30 incus-copy: all instances | | 01:00 incus-copy: all instances |
|
||||
| -> project backup, pool | | -> project backup, zpool sdb5 |
|
||||
| nucbackup (usb4t/backup/nuc) | | then restic instance leg -> S3 |
|
||||
| 05:00 pull ks4:* -> ks4backup | | 05:00 restic: DB dumps + data |
|
||||
| [pending FTTH] | | trees -> S3 (restic-data) |
|
||||
| 05:30 apt upgrade all containers | | Sun 14:00 restic maintenance |
|
||||
+-----------------------------------+ +-------------------------------------+
|
||||
phones/laptops: WG peers 10.8.0.2-3 reach 192.168.1.x through the same endpoint
|
||||
```
|
||||
|
||||
Both tunnels initiate **from** nuc (home NAT, dynamic IP) toward ks4's
|
||||
fixed endpoint; ks4's incus API is never exposed to the internet.
|
||||
|
||||
### Conventions
|
||||
|
||||
- One markdown file per instance in `nuc/`, containing the full
|
||||
install script (idempotent-ish, run as root on the host), first-run
|
||||
configuration, and troubleshooting notes.
|
||||
- One markdown file per topic in the correct subdir, containing the full
|
||||
implementation notes, configuration, and troubleshooting notes.
|
||||
- Containers are built from Debian/Ubuntu images and configured
|
||||
exclusively through `incus exec`/`incus config` so the doc is the
|
||||
single source of truth — rebuilding = re-running the script.
|
||||
- Adding a service to ks4: follow
|
||||
[new-container.md](new-container.md) (no Docker in containers —
|
||||
incus runs OCI images natively and `incus-compose` handles stacks;
|
||||
data paths go into `scripts/restic-paths`).
|
||||
data paths go into
|
||||
[`scripts/restic-paths`](https://git.lutran.fr/julien/scripts/src/branch/main/restic-paths)).
|
||||
|
||||
+10
-2
@@ -13,7 +13,15 @@ docs; this page is the map.
|
||||
|
||||
plakar was evaluated first and replaced (its nightly cost scaled with
|
||||
tree size — [PlakarKorp/plakar#2338](https://github.com/PlakarKorp/plakar/issues/2338));
|
||||
the old backup server **ks2 is retired** ([ks2/plan.md](ks2/plan.md)).
|
||||
the old backup server **ks2 is being retired** (decommission by
|
||||
Sep 30, 2026 — [ks2/plan.md](ks2/plan.md)): it still holds an ageing
|
||||
copy of the instances until the nuc leg below takes over.
|
||||
|
||||
**Status 2026-08-28**: the local replica leg and the S3 data leg are
|
||||
live. The **nuc pull leg is waiting for FTTH** (expected before end of
|
||||
September) and the **S3 instance leg is written but not yet seeded** —
|
||||
until both land, instances have only the ks4-local replica off the
|
||||
live pool.
|
||||
|
||||
## The map
|
||||
|
||||
@@ -95,4 +103,4 @@ without the passphrase the S3 buckets are unreadable.
|
||||
New instances are picked up automatically by the replica and
|
||||
instance legs (all instances, opt-out only). Databases are
|
||||
auto-discovered. Only **data trees** need one line in
|
||||
`/root/scripts/restic-paths` — see [new-container.md](new-container.md).
|
||||
[`restic-paths`](https://git.lutran.fr/julien/scripts/src/branch/main/restic-paths) — see [new-container.md](new-container.md).
|
||||
|
||||
+13
-11
@@ -6,8 +6,8 @@ few days of margin.
|
||||
|
||||
## Preconditions (the plan's release gates)
|
||||
|
||||
- [ ] plakar S3 leg running nightly for ≥ a week, `done (rc=0)`,
|
||||
restore test passed ([plakar-s3-data.md](../ks4/plakar-s3-data.md) §7)
|
||||
- [ ] restic S3 leg running nightly for ≥ a week, `done (rc=0)`,
|
||||
restore test passed ([restic-backup.md](../ks4/restic-backup.md) §7)
|
||||
- [ ] nuc pull leg seeded and one instance test-restored
|
||||
([nuc-seed.md](nuc-seed.md))
|
||||
- [ ] local leg (sdb5) cron green in `/var/log/incus-copy.log`
|
||||
@@ -15,7 +15,8 @@ few days of margin.
|
||||
## 1. Cut the last flows to ks2 (root on ks4)
|
||||
|
||||
```sh
|
||||
crontab -e # delete the 04:00 incus-backup.sh line (164.132.173.57)
|
||||
crontab -e # remove any remaining ks2 line (the 02:00 push, if it was
|
||||
# re-enabled; the 04:00 incus-backup.sh line is long gone)
|
||||
incus remote list # if a "ks2" remote is still defined:
|
||||
incus remote remove ks2
|
||||
# drop the rsyncbackup SSH key if it exists only for ks2:
|
||||
@@ -25,15 +26,15 @@ ls /root/.ssh/id_rsyncbackup* && rm -i /root/.ssh/id_rsyncbackup*
|
||||
## 2. Last look at ks2 before wiping (root on ks2)
|
||||
|
||||
```sh
|
||||
# anything in the rsync tree newer than the plakar cutover?
|
||||
# anything in the rsync tree newer than the S3 cutover?
|
||||
find /backup/ns3061243 -newermt '2026-08-23' | head
|
||||
# pre-2023 dirs (catc, mythoughts, qcm, scripts): instances still
|
||||
# exist stopped on ks4 — spot-check nothing unique remains:
|
||||
ls /backup/ns3061243/{catc,mythoughts,qcm,scripts}
|
||||
```
|
||||
|
||||
Archive anything unique to the plakar dump dir on ks4 (it rides the
|
||||
nightly `dumps` snapshot to S3 for free once it stops changing).
|
||||
Archive anything unique into the restic dump dir on ks4
|
||||
(`/backup/dumps`) — it then rides the nightly S3 snapshot for free.
|
||||
|
||||
## 3. Wipe and terminate
|
||||
|
||||
@@ -52,9 +53,10 @@ renewal date.
|
||||
durability bullet and the index line for `ks2/`s "being
|
||||
decommissioned" wording (keep the folder as history)
|
||||
- [ ] Retire `scripts/incus-backup.sh` + `incus-backup.db` in the repo
|
||||
(superseded by `scripts/plakar-backup.sh`) — keep for reference,
|
||||
note the retirement in their headers
|
||||
- [ ] `login` container cleanup parked in
|
||||
[plakar-s3-data.md](../ks4/plakar-s3-data.md): `dropdb outline`,
|
||||
remove native postgres
|
||||
(superseded by [`restic-backup.sh`](https://git.lutran.fr/julien/scripts/src/branch/main/restic-backup.sh)) —
|
||||
keep for reference, note the retirement in their headers
|
||||
- [ ] `login` container cleanup: the native postgres still holds the
|
||||
pre-migration `outline` DB — `dropdb outline`, remove the
|
||||
package (noted in
|
||||
[plakar-s3-data.md](../ks4/plakar-s3-data.md) §3)
|
||||
- [ ] Mark the ks2 line in `plan.md` done, date it
|
||||
|
||||
+1
-1
@@ -29,7 +29,7 @@ Notes:
|
||||
## Cron (after the seed)
|
||||
|
||||
Add to nuc's root crontab, offset from the 03:30 local nucbackup copy
|
||||
and ks4's own 01:00/04:30 jobs:
|
||||
and ks4's own 01:00/05:00 jobs:
|
||||
|
||||
```cron
|
||||
0 5 * * * /root/scripts/incus-copy.sh -r ks4 -s ks4backup >> /var/log/incus-copy-ks4.log 2>&1
|
||||
|
||||
+29
-27
@@ -4,20 +4,23 @@ Goal: release `ks2.lutran.fr` (`164.132.173.57`, SSH :2233, host
|
||||
`ns3247221`) once ks4 has a real 3-2-1 backup without it.
|
||||
**Deadline: rental ends Sep 30, 2026.**
|
||||
|
||||
## What ks2 does today (updated 2026-08-22)
|
||||
## What ks2 does today (updated 2026-08-28)
|
||||
|
||||
One remaining role, fed nightly by a root cron **on ks4**:
|
||||
**Nothing is written to ks2 any more.** Both feeds are retired: the
|
||||
`01:00` replica push (replaced by the local leg on ks4's sdb5 pool,
|
||||
[local-backup-cron.md](../ks4/local-backup-cron.md)) and the `04:00`
|
||||
`incus-backup.sh` rsync (replaced by restic → S3,
|
||||
[restic-backup.md](../ks4/restic-backup.md)).
|
||||
|
||||
- `04:00` `incus-backup.sh … -d 164.132.173.57` — DB dumps + selected
|
||||
paths rsync'd to ks2 `/backup/ns3061243` on pool `backup` (1.59 T,
|
||||
88 % full), snapshotted daily on ks2 (`zfs-auto-snapshot.sh`,
|
||||
2-month expiry). **Kept as a stopgap** until the plakar S3 leg
|
||||
replaces it.
|
||||
What remains on the box is **cold history**: instance replicas on pool
|
||||
`data` (last refreshed 2026-08-09) and the rsync tree
|
||||
`/backup/ns3061243` on pool `backup` (last refreshed 2026-08-28),
|
||||
snapshotted daily there (`zfs-auto-snapshot.sh`, 2-month expiry).
|
||||
|
||||
Retired: the `01:00` replica push to ks2 pool `data` — replaced by
|
||||
the local leg on ks4's sdb5 pool
|
||||
([local-backup-cron.md](../ks4/local-backup-cron.md)); `biwiki` and `spot`
|
||||
deleted from ks2.
|
||||
⚠️ Consequence while the nuc leg waits for FTTH: **instances have no
|
||||
fresh off-site copy** — only the ks4-local sdb replicas plus ks2's
|
||||
ageing ones. Either re-enable the interim push (below) or accept the
|
||||
gap knowingly until FTTH.
|
||||
|
||||
## Inventory findings (2026-08-22)
|
||||
|
||||
@@ -25,8 +28,9 @@ deleted from ks2.
|
||||
project — **decision: both abandoned**, safe to delete from backups.
|
||||
- `incus-backup.db` is stale (lists `spot`, misses `livetrail`,
|
||||
`outline`, `login`, …) — forgotten-manifest drift is exactly what
|
||||
the plakar-based replacement must eliminate (back up *all*
|
||||
instances by default, opt-out instead of opt-in).
|
||||
the replacement had to eliminate — restic's drivers back up *all*
|
||||
instances and auto-discover *all* databases, opt-out instead of
|
||||
opt-in.
|
||||
- `livetrail` (running on ks4) is **absent from ks4's local `backup`
|
||||
project** — the local leg was run manually once and never again;
|
||||
automating its cron fixes this.
|
||||
@@ -38,17 +42,17 @@ deleted from ks2.
|
||||
`qcm`) — those instances still exist stopped on ks4, so nothing
|
||||
unique expected there; spot-check before wiping.
|
||||
- Losing ks2 also loses its 2 months of rsync-snapshot history —
|
||||
acceptable once plakar has built equivalent retention.
|
||||
acceptable: restic keeps 14 daily / 8 weekly / 6 monthly.
|
||||
|
||||
## Target architecture (3-2-1 for ks4)
|
||||
|
||||
| Leg | Mechanism | Status |
|
||||
|---|---|---|
|
||||
| local, 2nd disk | `incus-copy.sh -p backup -s backup` → sdb5 pool | works, **cron missing** |
|
||||
| off-site, nuc | nuc pulls `ks4:*` → pool `ks4backup` over WG | waiting FTTH seed |
|
||||
| off-site, S3 (data+DB) | **restic** on ks4 → S3 (plakar ran it first, being replaced — see [restic-backup.md](../ks4/restic-backup.md)) | migrating |
|
||||
| off-site, S3 (instances) | restic over `incus file mount` of the `backup`-project replicas | designed |
|
||||
| off-site, ks2 (interim) | 02:00 replica push until the nuc leg seeds (FTTH < Sep 30) | to re-enable |
|
||||
| local, 2nd disk | `incus-copy.sh -p backup -s backup` → sdb5 pool | **live** (01:00) |
|
||||
| off-site, S3 (data+DB) | **restic** → bucket `restic-data` ([restic-backup.md](../ks4/restic-backup.md)) | **live** (05:00) |
|
||||
| off-site, S3 (instances) | restic over `incus file mount` of the `backup`-project replicas | written, **not seeded** |
|
||||
| off-site, nuc | nuc pulls `ks4:*` → pool `ks4backup` over WG ([nuc-seed.md](nuc-seed.md)) | waiting FTTH (< Sep 30) |
|
||||
| off-site, ks2 (optional interim) | `incus-copy.sh -d ks2 -m push` at 02:00 until the nuc leg seeds | **not enabled** — decide |
|
||||
|
||||
## Actions (backup work documented in [`../ks4/`](../ks4/), ks2-only tasks here)
|
||||
|
||||
@@ -57,13 +61,11 @@ deleted from ks2.
|
||||
2. ~~[local-backup-cron.md](../ks4/local-backup-cron.md)~~ — **done
|
||||
2026-08-22**: leg 1 cron re-enabled, ks2 rsync kept as stopgap;
|
||||
`livetrail` verified present in the `backup` project
|
||||
3. [plakar-s3-data.md](../ks4/plakar-s3-data.md) — **seeding S3
|
||||
2026-08-22** (direct-to-S3 kloset `lutran-ks4-plakar-data`, driver
|
||||
script `scripts/plakar-backup.sh`, pipeline validated locally);
|
||||
next: restore test + cron
|
||||
4. [plakar-incus-integration.md](../ks4/plakar-incus-integration.md) —
|
||||
design seeded 2026-08-22: per-file importer over the incus sftp
|
||||
API, instances → S3
|
||||
3. ~~plakar S3 leg~~ → **replaced by restic**
|
||||
([restic-backup.md](../ks4/restic-backup.md), live 2026-08-28;
|
||||
plakar's own doc kept as history)
|
||||
4. instance leg — restic over `incus file mount`
|
||||
([restic-backup.md](../ks4/restic-backup.md) §6): **seed pending**
|
||||
5. [nuc-seed.md](nuc-seed.md) — **prepared**; after FTTH: seed nuc
|
||||
pull leg, verify all instances, test-restore one
|
||||
6. [decommission.md](decommission.md) — **prepared**; cut flows,
|
||||
@@ -74,6 +76,6 @@ deleted from ks2.
|
||||
- [x] biwiki + spot consciously abandoned (2026-08-22)
|
||||
- [ ] local leg cron running ≥ a few days, all instances present
|
||||
- [ ] nuc leg fully seeded **and** one instance test-restored
|
||||
- [ ] plakar S3 data/DB backups running daily **and** one DB + one
|
||||
- [ ] restic S3 data/DB backups running daily **and** one DB + one
|
||||
file tree test-restored
|
||||
- [ ] ks4 crons pointing at ks2 disabled
|
||||
|
||||
@@ -0,0 +1,55 @@
|
||||
# Prod: ks4
|
||||
|
||||
Incus host at OVH — public-facing self-hosted services.
|
||||
|
||||
- Debian 13, Xeon D-1521 / 32 GiB, 2× 6 TB (OS on mdraid RAID1,
|
||||
ZFS pool `data` on `sda5`) — setup & rebuild:
|
||||
[install.md](install.md)
|
||||
- SSH: `ssh -i id_rsa_claude -p 2233 julien@193.70.35.17`
|
||||
(`julien` has passwordless `sudo incus …`)
|
||||
- Instances are on a NAT bridge `incusbr0` (192.168.1.0/24); public
|
||||
ingress via Incus proxy devices. `gateway` (nginx) is the sole HTTP/S
|
||||
entry point and fans out to the app containers.
|
||||
- Services: gateway, mail, nextcloud, seafile, git, freshrss, bitwarden,
|
||||
outline, login (SSO), wireguard, + more — full table in
|
||||
[install.md](install.md).
|
||||
- ⚠️ The ZFS `data` pool is single-disk (not mirrored); durability
|
||||
rests on nightly cron jobs — 01:00 `incus copy --refresh` of all
|
||||
instances to the local `backup` pool on sdb5, then the instance leg
|
||||
to S3; 05:00 restic (DB dumps + data trees) to S3; nuc pulls the
|
||||
replicas over WireGuard. Full picture and restore procedures:
|
||||
**[backup-strategy.md](../backup-strategy.md)**
|
||||
([local-backup-cron.md](local-backup-cron.md),
|
||||
[incus-copy.md](incus-copy.md),
|
||||
[restic-backup.md](restic-backup.md)).
|
||||
|
||||
## Network flows (nuc ↔ ks4)
|
||||
|
||||
```
|
||||
nuc — home LAN 192.168.0.0/24 ks4 — OVH 193.70.35.17
|
||||
+-----------------------------------+ +-------------------------------------+
|
||||
| | | |
|
||||
| host: wg-ks4 (10.8.0.20) | | [wireguard] 192.168.1.18 |
|
||||
| incus remote "ks4" ------+--WG-->| wg0 10.8.0.1/24, udp 51845 |
|
||||
| pull ks4:* -> pool ks4backup | udp | | masquerade -> eth0 |
|
||||
| on usb4t [pending FTTH seed] | 51845 | | |
|
||||
| | | +-> incus API 192.168.1.1:8443 |
|
||||
| [transmission-bt] wg0 (10.8.0.21) | | | (ufw: only from .18) |
|
||||
| full tunnel 0.0.0.0/0 ------+--WG-->| | |
|
||||
| kill switch: no default route | udp | +-> WAN egress: torrents + |
|
||||
| downloads -> /srv/media | 51845 | apt of transmission-bt |
|
||||
| (usb4t/media, read by jellyfin) | | exit as 193.70.35.17 |
|
||||
| | | |
|
||||
| 03:00 instance snapshots | | 03:00 instance snapshots |
|
||||
| 03:30 incus-copy: all instances | | 01:00 incus-copy: all instances |
|
||||
| -> project backup, pool | | -> project backup, zpool sdb5 |
|
||||
| nucbackup (usb4t/backup/nuc) | | then restic instance leg -> S3 |
|
||||
| 05:00 pull ks4:* -> ks4backup | | 05:00 restic: DB dumps + data |
|
||||
| [pending FTTH] | | trees -> S3 (restic-data) |
|
||||
| 05:30 apt upgrade all containers | | Sun 14:00 restic maintenance |
|
||||
+-----------------------------------+ +-------------------------------------+
|
||||
phones/laptops: WG peers 10.8.0.2-3 reach 192.168.1.x through the same endpoint
|
||||
```
|
||||
|
||||
Both tunnels initiate **from** nuc (home NAT, dynamic IP) toward ks4's
|
||||
fixed endpoint; ks4's incus API is never exposed to the internet.
|
||||
+5
-3
@@ -1,7 +1,7 @@
|
||||
# incus-copy — ks4 instance replication
|
||||
|
||||
Incremental replication of all ks4 Incus instances via
|
||||
[`scripts/incus-copy.sh`](../scripts/incus-copy.sh)
|
||||
[`scripts/incus-copy.sh`](https://git.lutran.fr/julien/scripts/src/branch/main/incus-copy.sh)
|
||||
(`incus copy --refresh`: ZFS-incremental, produces ready-to-start
|
||||
replicas instead of tarballs). Two legs replace the historical nightly
|
||||
push to `ks2` (decommissioning):
|
||||
@@ -35,7 +35,7 @@ incus-copy.sh -p <dest-project> -s <dest-pool> # local
|
||||
`--refresh-exclude-older` keeps expired snapshots from being re-sent.
|
||||
- Consistency: copies of running instances are crash-consistent;
|
||||
databases may need recovery on restore. Snapshot history mitigates,
|
||||
and [`incus-backup.sh`](../scripts/incus-backup.sh) (proper
|
||||
and [`incus-backup.sh`](https://git.lutran.fr/julien/scripts/src/branch/main/incus-backup.sh) (proper
|
||||
`mariadb-dump`s + selected paths, manifest-driven) covers the DBs.
|
||||
- **VMs**: `--refresh` re-sends a VM's **full block volume** every run
|
||||
unless source and replica share a common snapshot — containers diff
|
||||
@@ -63,7 +63,9 @@ Cron (root on ks4) — replaces both ks2 jobs:
|
||||
|
||||
```cron
|
||||
0 1 * * * /root/scripts/incus-copy.sh -p backup -s backup >> /var/log/incus-copy.log 2>&1
|
||||
0 4 * * * /root/scripts/incus-backup.sh -f /root/scripts/incus-backup.db -s data >> /var/log/incus-backup.log 2>&1
|
||||
# retired 2026-08-28 — dumps + data trees are restic's job (05:00),
|
||||
# see ../backup-strategy.md and restic-backup.md
|
||||
#0 4 * * * /root/scripts/incus-backup.sh -f /root/scripts/incus-backup.db -s data >> /var/log/incus-backup.log 2>&1
|
||||
```
|
||||
|
||||
(`incus-backup.sh` without `-d` now writes locally to
|
||||
|
||||
+22
-12
@@ -142,22 +142,32 @@ All in the `default` project, on the `default` profile
|
||||
|
||||
## Backups
|
||||
|
||||
> **Migration in progress** (2026-08-09): the `ks2` destination below is
|
||||
> being replaced by a local `backup` zpool on `sdb5` + off-site pull
|
||||
> from nuc — plan and setup in [incus-copy.md](incus-copy.md). The
|
||||
> cron blocks here describe what runs **today**; update on cutover.
|
||||
|
||||
Two **root** cron jobs (this repo's [`scripts/`](../scripts/) are the
|
||||
copies that run here) — the primary safety net for the non-mirrored
|
||||
`data` pool:
|
||||
Full picture, schedule and restore procedures:
|
||||
**[../backup-strategy.md](../backup-strategy.md)**. Drivers live in the
|
||||
[scripts repo](https://git.lutran.fr/julien/scripts) and are deployed
|
||||
to `/root/scripts`. Current **root** cron jobs — the safety net for the
|
||||
non-mirrored `data` pool:
|
||||
|
||||
```cron
|
||||
0 1 * * * /root/scripts/incus-copy.sh -d ks2 -m push
|
||||
0 4 * * * /root/scripts/incus-backup.sh -d 164.132.173.57 -u rsyncbackup \
|
||||
-i /root/.ssh/id_rsyncbackup -p 2233 \
|
||||
-f /root/scripts/incus-backup.db -s data
|
||||
0 1 * * * /root/scripts/incus-copy.sh -p backup -s backup >> /var/log/incus-copy.log 2>&1
|
||||
0 5 * * * /root/scripts/restic-backup.sh >> /var/log/restic-backup.log 2>&1
|
||||
0 14 * * 0 /root/scripts/restic-maintenance.sh >> /var/log/restic-maintenance.log 2>&1
|
||||
```
|
||||
|
||||
- **01:00 `incus-copy.sh`** → replicas of every instance in project
|
||||
`backup` on the sdb5 pool ([incus-copy.md](incus-copy.md),
|
||||
[local-backup-cron.md](local-backup-cron.md)); the S3 instance leg
|
||||
chains here once seeded.
|
||||
- **05:00 `restic-backup.sh`** → auto-discovered DB dumps + the data
|
||||
trees listed in
|
||||
[`scripts/restic-paths`](https://git.lutran.fr/julien/scripts/src/branch/main/restic-paths),
|
||||
into the `restic-data` bucket ([restic-backup.md](restic-backup.md)).
|
||||
- **Sun 14:00 `restic-maintenance.sh`** → prune + integrity checks.
|
||||
|
||||
Historical (retired 2026-08): the `ks2` push and `incus-backup.sh`
|
||||
rsync leg, and the plakar data leg
|
||||
([plakar-s3-data.md](plakar-s3-data.md)).
|
||||
|
||||
- **01:00 — `incus-copy.sh` → `ks2`**: differential `incus copy … \
|
||||
--refresh --refresh-exclude-older --mode push` of every *running*
|
||||
instance to remote incus host `ks2` (a near-live replica). `ks2` must
|
||||
|
||||
+10
-15
@@ -1,28 +1,23 @@
|
||||
# local backup cron on ks4 (leg 1) — re-enabled
|
||||
# local backup cron on ks4 (leg 1)
|
||||
|
||||
Done 2026-08-22. Context: **both** ks4 backup crons had been commented
|
||||
out — no backup of any kind was running.
|
||||
Enabled 2026-08-22 (both ks4 backup crons had been commented out — no
|
||||
backup of any kind was running). Overview of all legs:
|
||||
[../backup-strategy.md](../backup-strategy.md).
|
||||
|
||||
## What runs now (root crontab on ks4)
|
||||
|
||||
```cron
|
||||
0 1 * * * /root/scripts/incus-copy.sh -p backup -s backup >> /var/log/incus-copy.log 2>&1
|
||||
0 4 * * * /root/scripts/incus-backup.sh -d 164.132.173.57 -u rsyncbackup -i /root/.ssh/id_rsyncbackup -p 2233 -f /root/scripts/incus-backup.db -s data >> /var/log/incus-backup.log 2>&1
|
||||
```
|
||||
|
||||
- `01:00` — **local leg**: `incus copy --refresh` of all instances
|
||||
into project `backup` on the sdb5 `backup` zpool (replaces the old
|
||||
push to ks2; see [incus-copy.md](incus-copy.md)).
|
||||
- `04:00` — **stopgap**: `incus-backup.sh` rsync to ks2 kept alive
|
||||
(manifest pruned of `spot`) until the plakar S3 leg replaces it —
|
||||
then this line goes away as part of
|
||||
[decommissioning ks2](../ks2/plan.md). ⚠️ keep
|
||||
`scripts/incus-backup.db` in this repo in sync with
|
||||
`/root/scripts/incus-backup.db` on ks4.
|
||||
|
||||
Both were also run once by hand (in a `screen`) on 2026-08-22 to
|
||||
refresh the backups immediately. The old push-to-ks2 copy line stays
|
||||
commented out — replica refreshes to ks2 are retired.
|
||||
push to ks2; see [incus-copy.md](incus-copy.md)). The S3 instance
|
||||
leg ([restic-backup.md](restic-backup.md) §6) chains onto this same
|
||||
line once seeded — it reads the replicas this job just refreshed.
|
||||
- The rsync stopgap to ks2 (`incus-backup.sh`, 04:00) and the plakar
|
||||
data leg (04:30) are both **retired**; S3 backups are restic's job
|
||||
at 05:00.
|
||||
|
||||
## Cleanup done the same day
|
||||
|
||||
|
||||
@@ -1,4 +1,14 @@
|
||||
# plakar incus integration — design notes
|
||||
# plakar incus integration — design notes (RETIRED)
|
||||
|
||||
> **Historical.** The importer works (smoke-tested against ks4's
|
||||
> `backup` project) but production moved to restic, which needs no
|
||||
> plugin: `incus file mount` exposes any instance — including stopped
|
||||
> replicas — over the same per-instance sftp API
|
||||
> ([restic-backup.md](restic-backup.md) §6). Code lives on at
|
||||
> <https://github.com/jlutran/plakar-integration-incus>; the analysis
|
||||
> below (per-file vs image, replica-vs-live consistency, the
|
||||
> seek-bound wall) is what shaped the restic design.
|
||||
|
||||
|
||||
Status: **importer scaffolded and compiling 2026-08-23** in
|
||||
`perso/plakar/integration-incus` (walker modeled on the official sftp
|
||||
|
||||
+13
-2
@@ -1,4 +1,15 @@
|
||||
# plakar data + DB backups on ks4 (leg 3)
|
||||
# plakar data + DB backups on ks4 (leg 3) — SUPERSEDED
|
||||
|
||||
> **Historical.** This leg ran 2026-08-22 → 08-28 and was replaced by
|
||||
> restic: see **[restic-backup.md](restic-backup.md)**. plakar's
|
||||
> incremental cost scaled with tree size rather than churn
|
||||
> ([PlakarKorp/plakar#2338](https://github.com/PlakarKorp/plakar/issues/2338)).
|
||||
> The `plakar-data` kloset and the plakar install are deliberately kept
|
||||
> until that issue concludes (the report offers debug runs against that
|
||||
> repository); the `plakar-incus` bucket can be deleted. Kept here for
|
||||
> the measurements and the design rationale, which the restic doc
|
||||
> builds on.
|
||||
|
||||
|
||||
Status: **live since 2026-08-24** — S3 kloset
|
||||
(`lutran-ks4-plakar-data`) seeded (dumps + 9 fs sources), restore
|
||||
@@ -19,7 +30,7 @@ leg 1's job. Direct-to-S3 keeps the incremental behavior — the
|
||||
skip-unchanged baseline is the parent snapshot's VFS read from the
|
||||
kloset (plakar caches repository metadata locally; v1.1.4's dirpack
|
||||
prefetcher targets exactly this remote-backend walk). Driver script:
|
||||
[`scripts/plakar-backup.sh`](../../scripts/plakar-backup.sh)
|
||||
[`scripts/plakar-backup.sh`](https://git.lutran.fr/julien/scripts/src/branch/main/plakar-backup.sh)
|
||||
(flock, incus DB dumps, iterates a sources file, prune + maintenance;
|
||||
its `-k` takes any `plakar at` target, so `-k @s3` — a throwaway
|
||||
local kloset, e.g. `-k /backup/plakar-test`, works for testing before
|
||||
|
||||
+23
-20
@@ -1,10 +1,12 @@
|
||||
# restic backups on ks4 — the S3 leg
|
||||
|
||||
Status: **decided 2026-08-25 — restic replaces plakar as the S3
|
||||
backup tool** (evaluation first: parallel run + comparison, see §8).
|
||||
Final architecture: **two tools** — `incus copy` for replication
|
||||
(local sdb, nuc after FTTH, ks2 push as interim), **restic** for
|
||||
everything S3.
|
||||
Status: **live since 2026-08-28** — restic replaced plakar as the S3
|
||||
backup tool (plakar crons off, its kloset kept for
|
||||
[#2338](https://github.com/PlakarKorp/plakar/issues/2338); see §8).
|
||||
Architecture: **two tools** — `incus copy` for replication (local sdb,
|
||||
nuc after FTTH), **restic** for everything S3. Data leg (`restic-data`)
|
||||
runs nightly at 05:00; the instance leg (`restic-incus`, §6) is written
|
||||
and awaits its first seed.
|
||||
|
||||
Why: plakar's incremental cost scales with tree size, not churn
|
||||
(~800 MiB parent metadata re-read from S3 + hours of CPU per night on
|
||||
@@ -58,9 +60,10 @@ property).
|
||||
### Memory (the real constraint on this 23/31 GiB-used box)
|
||||
|
||||
- Index RAM ≈ 200–300 B/blob → combined `restic-data` index
|
||||
~1–1.5 GiB per operation (nextcloud ~1 M blobs + seafile ~2–4 M);
|
||||
prune peaks 2–3×. Accepted with the merged layout (§1) — measure
|
||||
peak RSS during the evaluation week (`/usr/bin/time -v`).
|
||||
~1 GiB per operation; prune peaks 2–3×. The seafile side of that
|
||||
estimate collapsed after its garbage collection (24.9 M → 0.5 M
|
||||
objects, [seafile-gc.md](seafile-gc.md)). Measure peak RSS with
|
||||
`/usr/bin/time -v` if the box ever feels tight.
|
||||
- `GOGC=20` trades CPU for a smaller Go heap — enable if the seafile
|
||||
prune ever pressures the box.
|
||||
|
||||
@@ -140,7 +143,7 @@ for r in restic-data restic-incus; do
|
||||
done
|
||||
```
|
||||
|
||||
## 4. Seed plan
|
||||
## 4. Seed plan (executed 2026-08-25 → 08-28)
|
||||
|
||||
1. **ARC floor first** (helps every walk on the box):
|
||||
```sh
|
||||
@@ -252,18 +255,18 @@ restic -r s3:…/restic-data restore latest --target /backup/restore-test --incl
|
||||
diff -r … && rm -rf /backup/restore-test
|
||||
```
|
||||
|
||||
## 8. Evaluation protocol + plakar retirement
|
||||
## 8. Cutover (done) + what is left
|
||||
|
||||
- **Parallel week**: plakar 04:30 cron keeps running untouched;
|
||||
restic runs at 05:00 on the same data. Same nights, same churn →
|
||||
direct comparison (wall, bytes read from store, bytes written, peak
|
||||
RSS via `/usr/bin/time -v`) against plakar's measured table in
|
||||
[plakar-s3-data.md](plakar-s3-data.md). Restore-speed compared on
|
||||
the same 10 GiB dumps tree (plakar measured: 39 min).
|
||||
- **Cutover** (after the comparison confirms + restore test passes):
|
||||
drop the plakar 04:30 + Sunday-check crons; 01:00 becomes
|
||||
`incus-copy ; restic instances phase` (or stays in the 05:00 run).
|
||||
Update plan.md legs table.
|
||||
- **Cutover 2026-08-28**: plakar's 04:30 and Sunday crons are off;
|
||||
restic's 05:00 data leg is the S3 backup. The formal parallel-week
|
||||
comparison was cut short — the evidence had already settled it
|
||||
(plakar: ~800 MiB metadata re-read and 3–4 h per night on the
|
||||
nextcloud tree, churn-independent; restic: seed at 60–115 MiB/s,
|
||||
local-cache incrementals). plakar's measurements stay in
|
||||
[plakar-s3-data.md](plakar-s3-data.md).
|
||||
- **Left to do**: seed `restic-incus` (§6) and chain it onto the 01:00
|
||||
line; restore test on the complete `restic-data`; then the ks2
|
||||
decommission gates ([../ks2/plan.md](../ks2/plan.md)).
|
||||
- **plakar retirement, one exception**: keep plakar installed and
|
||||
keep the `plakar-data` kloset until
|
||||
[#2338](https://github.com/PlakarKorp/plakar/issues/2338) concludes
|
||||
|
||||
+11
-5
@@ -21,8 +21,8 @@ current state → new library → delete old → GC `-r`, which works for
|
||||
takes the dataset to ~600 k inodes, at which point per-file backup of
|
||||
seafile becomes ordinary.
|
||||
|
||||
All crontab backup jobs on ks4 remain commented out until restart +
|
||||
validation completes.
|
||||
(The ks4 backup crons were paused during the GC window and are back
|
||||
on since 2026-08-27.)
|
||||
|
||||
## Root cause of the 24.9 M-inode / 933 G dataset
|
||||
|
||||
@@ -73,9 +73,15 @@ setting (the existing per-library 30 d row stays). Users can still
|
||||
override per library in the web UI if
|
||||
`ENABLE_REPO_HISTORY_SETTING = True` in `seahub_settings.py`.
|
||||
|
||||
### 2. GC — offline, maintenance window
|
||||
### 2. GC — maintenance window
|
||||
|
||||
CE's GC requires seafile stopped. On this pathological store expect
|
||||
⚠️ Superseded by experience: with the **MySQL backend seafile runs
|
||||
*online* GC** (the runs print "use online GC"), so stopping the
|
||||
services is not required — the recurring cron below does not. The
|
||||
2026-08-26 run stopped them anyway, out of caution. What follows is
|
||||
that run's procedure.
|
||||
|
||||
CE's GC traditionally requires seafile stopped. On this pathological store expect
|
||||
**hours** (it traverses fs/commits on a cold HDD); the dry-run sizes
|
||||
the window. Inside the container:
|
||||
|
||||
@@ -115,7 +121,7 @@ du -sh .../seafile/rootfs/opt/seafile/seafile-data/storage/blocks # was 867 G
|
||||
0 6 15 1,4,7,10 * incus exec seafile -- sh -c "cd /opt/seafile/seafile-server-latest && ./seaf-gc.sh && ./seaf-gc.sh -r && ./seaf-gc.sh -R" >> /var/log/seafile-gc.log 2>&1
|
||||
```
|
||||
At the post-cleanup scale (~120 k live files) the full sequence is
|
||||
minutes, not hours. Keep it clear of the 01:00/04:30/05:00 backup
|
||||
minutes, not hours. Keep it clear of the 01:00/05:00 backup
|
||||
slots.
|
||||
- 2026-08-27 follow-ups executed: 8 old libraries deleted by user →
|
||||
trash emptied → `seaf-gc -r` purged their stores (18 m, blocks now
|
||||
|
||||
+4
-4
@@ -75,7 +75,7 @@ them forces a full re-send of its whole disk every night.
|
||||
|
||||
**Manual, one line:** data that lives in the filesystem (uploads,
|
||||
repositories, mail spools…) must be listed in
|
||||
`/root/scripts/restic-paths`, one absolute host path per line:
|
||||
[`restic-paths`](https://git.lutran.fr/julien/scripts/src/branch/main/restic-paths), one absolute host path per line:
|
||||
|
||||
```
|
||||
/var/lib/incus/storage-pools/data/containers/myservice/rootfs/var/lib/myservice
|
||||
@@ -84,7 +84,7 @@ repositories, mail spools…) must be listed in
|
||||
(The container's `/x` is `…/containers/<name>/rootfs/x` on the host.)
|
||||
No comments in that file — every line is read as a path. Regenerable
|
||||
caches can be skipped by adding a pattern to
|
||||
`/root/scripts/restic-exclude`. Both files live in the `scripts` git
|
||||
[`restic-exclude`](https://git.lutran.fr/julien/scripts/src/branch/main/restic-exclude). Both files live in the `scripts` git
|
||||
repo: edit there, commit, `git pull` on ks4.
|
||||
|
||||
For the database auto-discovery to work, keep the defaults:
|
||||
@@ -98,8 +98,8 @@ For the database auto-discovery to work, keep the defaults:
|
||||
"postgres" (official images do).
|
||||
|
||||
A database you do *not* want dumped (huge, static, re-importable)
|
||||
goes into `/root/scripts/plakar-db-exclude` as `<instance>/<db>` —
|
||||
every skip is logged, so the list cannot rot silently.
|
||||
goes into `/root/scripts/db-exclude` as `<instance>/<db>` — every
|
||||
skip is logged, so the list cannot rot silently.
|
||||
|
||||
## 5. Verify, once
|
||||
|
||||
|
||||
@@ -0,0 +1,28 @@
|
||||
# Homelab: nuc
|
||||
|
||||
Incus host on the LAN.
|
||||
|
||||
- Debian 13, Intel Alder Lake-N (iGPU `i915`, shared by both Jellyfin containers) —
|
||||
bare-metal reinstall: [nuc-install.md](nuc-install.md)
|
||||
- Instances are bridged onto the LAN (192.168.0.0/24)
|
||||
- USB 4 TB WD Red: ZFS pool `usb4t` — `usb4t/backup` → `/backup`
|
||||
(incus exports, `nuc/` + `ks4/` subdatasets, 1 TB quota) and
|
||||
`usb4t/media` → `/srv/media` (media library, shared into containers
|
||||
via `shift=true` disk devices; works because ZFS ≥ 2.2 supports
|
||||
idmapped mounts)
|
||||
- Backups: all local instances replicated to the USB pool
|
||||
(`/root/scripts/incus-copy.sh -p backup -s nucbackup`; replicas
|
||||
stopped, autostart off) — see
|
||||
[nuc-install.md](nuc-install.md); ks4 replicas pulled into
|
||||
pool `ks4backup` — see [ks4/incus-copy.md](../ks4/incus-copy.md)
|
||||
|
||||
## Instances
|
||||
|
||||
| Name | IP | Doc | Features |
|
||||
|---|---|---|---|
|
||||
| [jellyfin-server](jellyfin-server.md) | 192.168.0.5 | ✅ | unprivileged, autostart; iGPU render node (`gpu` device, render gid) for QSV/VAAPI transcoding; `/srv/media` disk device (`shift=true`); proxy device → host :8096 |
|
||||
| [jellyfin-client](jellyfin-client.md) | 192.168.0.6 | ✅ | **privileged**, autostart; full iGPU (`gpu` device, gid 44) → HDMI kiosk (cage + Jellyfin Media Player); custom `raw.lxc` (bind `/dev/snd`, `/dev/input`, host `/run/udev`); Pioneer USB audio as ALSA default; FR keymap; go-librespot Spotify Connect ("Pioneer A-70") |
|
||||
| [transmission-bt](transmission-bt.md) | 192.168.0.7 | ✅ | unprivileged, autostart; always-on WireGuard full tunnel → ks4 (egress = 193.70.35.17, kill switch: no default route); `/srv/media` disk device (`shift=true`), downloads to `/media/downloads`; web UI :9091 (LAN only) |
|
||||
| blocky | 192.168.0.254 | — | unprivileged, autostart; DNS ad-blocker |
|
||||
| privoxy | 192.168.0.11 | — | unprivileged, autostart; filtering HTTP proxy |
|
||||
| homeassistant | (stopped) | — | **virtual machine**, 50 GiB root disk on pool `data` |
|
||||
+1
-1
@@ -19,7 +19,7 @@ How to rebuild the Incus host from scratch if `/dev/sda` (512 GB SSD,
|
||||
The ZFS pool `data` lives on `sda5` → **all instance root disks are lost**
|
||||
with the OS. The USB pool `usb4t` (backups + media) survives — all
|
||||
instances are replicated onto it with
|
||||
[`scripts/incus-copy.sh`](../scripts/incus-copy.sh) (deployed at
|
||||
[`scripts/incus-copy.sh`](https://git.lutran.fr/julien/scripts/src/branch/main/incus-copy.sh) (deployed at
|
||||
`/root/scripts/` on nuc; set up 2026-08-09):
|
||||
|
||||
```sh
|
||||
|
||||
Reference in New Issue
Block a user