c8d759c claimed the ~125 Mbit/s seed rate was ks4's OVH uplink. That was
wrong, and measuring it says so: ks4 uploads at 609 Mbit/s, downloads at
900, and nas pulls 670 from OVH's network. WireGuard is not it either —
zero UdpRcvbufErrors, wg-crypt kworkers at ~2%, both hosts ~85% idle.
The limit is `data` sitting on sdb5, one 7200 rpm HGST 6 TB disk: during
the send it does 109 r/s at 14 MB/s with ~131 KB requests and a queue
depth of ~1.0, which is the random-IOPS ceiling of a single HDD reading a
fragmented 1.42 TiB dataset. 14 MB/s is ~112 Mbit/s on the wire, exactly
what we see, while the network sits idle.
Parallelism is the only lever and it works by disk queue depth, not
bandwidth: a second concurrent copy takes sdb from 14 to 21 MB/s and the
request size from 131 KB to 514 KB, lifting the tunnel from 113 to 153
Mbit/s. Recorded, along with the decision to keep the seed sequential —
one-off pass, incremental refreshes after, and incus-copy.sh is shared by
all three legs.
Also corrects the set size to the actual 1.42 TiB (~29 h, not ~31 h).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
FTTH is up, so the leg nas-seed.md had been holding since 2026-08-30 is
now built: wg-ks4 on nas (10.8.0.22), nas peered on ks4's wireguard
container, incus remote over the tunnel, 05:00 cron, and the first pass
seeding under a systemd-run unit.
Three corrections the runbook needed, all found by running it:
- nas had no wireguard-tools at all. transmission-bt carries its own
tunnel inside the container, so the host never needed them.
- Every ks4 instance has an instance-level eth0 pinned to incusbr0 with
a static 192.168.1.x, so each copy failed in under a second with
"Cannot use manually specified ipv4.address when using unmanaged
parent bridge". nas now runs a managed incusbr0 on 192.168.1.254/24 —
deliberately not .1, which must keep resolving over wg-ks4.
- The seed command was missing -p backup, which the doc's own
verification step already assumed.
Also records the measured rate: ~125 Mbit/s, ks4's OVH uplink rather
than the home downlink, so ~31 h for the first pass — during which the
shared incus-copy lock suppresses the 04:00 nasbackup job.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The plan gated nuc's tunnel teardown on the nas leg being seeded, on the
assumption nuc stayed a viable fallback target. It is not one: its
ks4backup pool was deleted when the disk moved, and its data pool is a
512 GB SSD against a ~1.75 TiB replica set. So the tunnel was doing
nothing except re-establishing a keepalive'd link to ks4 on every boot of
a machine that is now powered off between uses.
Disabled 2026-08-31 (wg-quick@wg-ks4 disabled, interface down, ks4 incus
remote removed from nuc). The config and key are deliberately kept, so it
is one systemctl away if ever needed — deleting them would mean
regenerating keys and re-peering on ks4.
transmission-bt is unaffected: its tunnel is in-container and a separate
peer (10.8.0.21), verified still handshaking with egress 193.70.35.17.
Remaining: drop nuc's now-unused peer on ks4's wireguard container.
Harmless to leave, safe to do any time, recorded with the pubkey.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
backup-strategy (the entry point) still described nuc pulling the ks4
replicas: the leg, its WireGuard peer and the target pool now live on
nas with the 4 TB on direct SATA. Also: ks4 README flow chart redrawn
for the new topology, incus-copy leg 2 retargeted, usb4t-dropouts
marked RESOLVED (kept for the diagnosis method and the alerting gap),
and ks2/plan records that the interim push was deliberately not
re-enabled.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
nuc-seed.md -> nas-seed.md. The leg was designed around nuc's USB pool,
which is exactly the device it must not depend on. Target pool ks4backup
now lives on tank; nas becomes WG peer 10.8.0.22 and nuc's tunnel retires
once seeded — nuc no longer needs one at all, since transmission-bt (the
only other user) moved to nas with its own in-container tunnel.
ks4 needs no change: traffic arrives masqueraded as the wireguard
container whichever peer sent it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Builds the box that ends the usb4t dropouts: the JMicron bridge was the
least reliable device in the setup and it held the intended off-site copy
of ks4. The 4 TB now sits on direct SATA as pool `tank`.
- OS on mdraid RAID1 across two 120 GB SSDs (Intel 330 + Toshiba Q300),
both ESPs bootable; `incus` ZFS mirror on their tails, ~22% left
unallocated as over-provisioning
- media at /export/media, exported read-only over NFSv4 to nuc
- transmission-bt moves here (its WireGuard tunnel is in-container, so
ks4 needed no change) and writes to the dataset locally
- backup pools nucbackup / ks4backup / nasbackup
- monitoring live: msmtp (submission+auth, verified 250), zed with
NOTIFY_DATA, zpool-health.sh every 15 min, smartd on all three disks
Traps recorded because none of them point at their own cause: booting
with the display active kills the i915 probe and wedges incus; d-i picks
grub-pc vs grub-efi from how the installer booted; `incus storage create`
hangs forever on a mountpoint=none dataset; the BMC is deliberately never
cabled, so there is no out-of-band console.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
18/18 instances replicated, restic seed complete and restore-tested
(tree identical to live; dump loads into a scratch MariaDB, 12/12
tables). Remaining ks2 gates are the nuc leg after FTTH.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
backup-strategy.md now leads with the three target legs (local incus
push to sdb, nuc pull over WireGuard, restic to S3 for DBs and
selected trees) and carries the plakar/ks2 story in a closing History
section. restic-backup.md, install.md, local-backup-cron.md and the
ks2 docs describe the predecessor only as 'the first S3
implementation', with one operational note that plakar is still
installed for issue #2338 reproduction.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Decided 2026-08-28: instances are protected by replication (sdb now,
nuc after FTTH, ks2 push in between), their data/DBs/configs by
restic-data — a third copy of instance filesystems in S3 was not
worth the moving parts. backup-strategy.md now explains the
replication-vs-backup split and the rebuild path when no replica
survives; restic-backup.md §6 is marked SHELVED (script and empty
repo kept).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- nuc/README.md and ks4/README.md carry the host sections (+ network
flows) that lived in the top-level README; links rebased
- top README: repo links (doc/scripts on git.lutran.fr), index points
at the new per-host pages
- cross-repo references now use https://git.lutran.fr/julien/scripts
instead of relative ../scripts paths that resolve nowhere
- plakar-s3-data.md and plakar-incus-integration.md marked SUPERSEDED
/ RETIRED with pointers to restic-backup.md; their measurements and
rationale kept
- install.md, local-backup-cron.md, incus-copy.md: crontab sections
updated to the live schedule (01:00 replicas, 05:00 restic, Sun
maintenance); retired legs labelled as such
- restic-backup.md: status live, cutover recorded, post-GC memory
estimate, seed plan dated
- seafile-gc.md: online GC noted, stale 'crons commented out' removed
- ks2/: what-ks2-does-today rewritten (nothing writes to it any more),
legs table and gates reflect restic, decommission steps updated
- db-exclude replaces the plakar-era config name (script keeps a
fallback)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Julien LutranandClaude Opus 5
Julien LutranandClaude Opus 5