FTTH is up, so the leg nas-seed.md had been holding since 2026-08-30 is
now built: wg-ks4 on nas (10.8.0.22), nas peered on ks4's wireguard
container, incus remote over the tunnel, 05:00 cron, and the first pass
seeding under a systemd-run unit.
Three corrections the runbook needed, all found by running it:
- nas had no wireguard-tools at all. transmission-bt carries its own
tunnel inside the container, so the host never needed them.
- Every ks4 instance has an instance-level eth0 pinned to incusbr0 with
a static 192.168.1.x, so each copy failed in under a second with
"Cannot use manually specified ipv4.address when using unmanaged
parent bridge". nas now runs a managed incusbr0 on 192.168.1.254/24 —
deliberately not .1, which must keep resolving over wg-ks4.
- The seed command was missing -p backup, which the doc's own
verification step already assumed.
Also records the measured rate: ~125 Mbit/s, ks4's OVH uplink rather
than the home downlink, so ~31 h for the first pass — during which the
shared incus-copy lock suppresses the 04:00 nasbackup job.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Follow-up to a7f1ac6. The Archer C7's DHCP reservations did not carry
over to the FTTH box, so anything still on DHCP was one lease renew away
from moving — as blocky already demonstrated by taking LAN DNS with it.
jellyfin-server and jellyfin-client are now static too; homeassistant
stays dynamic on purpose (it never had a reservation).
Records three things that cost time to find:
- jellyfin-client can never use netplan: the kiosk raw.lxc bind-mounts
the host /run/udev read-only, so netplan generate fails and config
silently does not regenerate at boot. It uses systemd-networkd now.
- LAPTOP719974 and patate also lost their reserved addresses.
- ks4 cannot serve as an IPv6 exit: it has a global v6 address and a
default v6 route but no working v6 egress, so extending the torrent
tunnel to ::/0 is not an option. IPv6 stays disabled in the container.
Also notes the C7 + LTE box kept as fallback: only the gateway differs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Archer C7 at .2 is gone; the FTTH box at .1 is the gateway. Every
static holdout (nas host, nuc host, privoxy, transmission-bt) pointed at
the dead .2 and had no internet — the reported symptom was privoxy.
Also record two things the migration broke that were not obvious:
- DHCP reservations did not carry over. blocky held .254 by reservation
on the C7; a lease renew on the FTTH box moved it and took LAN DNS
down. It is static now. jellyfin-* are still DHCP on stale leases.
- The FTTH box advertises native IPv6. transmission-bt's tunnel is
AllowedIPs = 0.0.0.0/0, so v6 egressed around the kill switch on the
home address. IPv6 is now disabled in that container.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
nas ran none: the script was deployed but never scheduled, while nuc
(lower stakes) upgraded nightly. Kept clear of the future 05:00 ks4
pull.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Standardised 2026-08-31 on regular crontabs: nuc's push moved off its
systemd timer (units kept disabled on disk), and the cron.d files on
nuc and nas were folded into root's crontab. Notes the consequence
accepted for nuc: a night with the box powered off is skipped rather
than caught up after boot.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
backup-strategy (the entry point) still described nuc pulling the ks4
replicas: the leg, its WireGuard peer and the target pool now live on
nas with the 4 TB on direct SATA. Also: ks4 README flow chart redrawn
for the new topology, incus-copy leg 2 retargeted, usb4t-dropouts
marked RESOLVED (kept for the diagnosis method and the alerting gap),
and ks2/plan records that the interim push was deliberately not
re-enabled.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Aug 29 16:22 the enclosure dropped again despite the new cable, pool
suspended, and ZED sent nothing (suspended pools keep the vdev ONLINE;
data-class events were muted). Enclosure replacement is now the prime
candidate and a gate before the pool hosts the ks4 off-site leg.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The metadata/file errors were artefacts of the interrupted writes, not
corruption. The 2.5h scrub doubled as a soak test of the new cable.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
18/18 instances replicated, restic seed complete and restore-tested
(tree identical to live; dump loads into a scratch MariaDB, 12/12
tables). Remaining ks2 gates are the nuc leg after FTTH.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Records the working chain and the two dead ends: port 25 gets the mail
rspamd-rejected (dynamic home IP, no SPF alignment), and the missing
PTR delays the 587 greeting enough that short-timeout probes look like
a firewall block. Credentials live only in /etc/msmtprc (600).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Nightly replication had been failing since Aug 17 behind a misleading
incus error; root cause is a JMicron USB bridge dropping off the bus
(61 disconnects in 30 days) and suspending the pool — the disk itself
is SMART-clean. Records the recovery (clear/scrub), the damage, the
ordered fixes, and why nothing alerted: zed was running without an MTA
on the host.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
backup-strategy.md now leads with the three target legs (local incus
push to sdb, nuc pull over WireGuard, restic to S3 for DBs and
selected trees) and carries the plakar/ks2 story in a closing History
section. restic-backup.md, install.md, local-backup-cron.md and the
ks2 docs describe the predecessor only as 'the first S3
implementation', with one operational note that plakar is still
installed for issue #2338 reproduction.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Decided 2026-08-28: instances are protected by replication (sdb now,
nuc after FTTH, ks2 push in between), their data/DBs/configs by
restic-data — a third copy of instance filesystems in S3 was not
worth the moving parts. backup-strategy.md now explains the
replication-vs-backup split and the rebuild path when no replica
survives; restic-backup.md §6 is marked SHELVED (script and empty
repo kept).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- nuc/README.md and ks4/README.md carry the host sections (+ network
flows) that lived in the top-level README; links rebased
- top README: repo links (doc/scripts on git.lutran.fr), index points
at the new per-host pages
- cross-repo references now use https://git.lutran.fr/julien/scripts
instead of relative ../scripts paths that resolve nowhere
- plakar-s3-data.md and plakar-incus-integration.md marked SUPERSEDED
/ RETIRED with pointers to restic-backup.md; their measurements and
rationale kept
- install.md, local-backup-cron.md, incus-copy.md: crontab sections
updated to the live schedule (01:00 replicas, 05:00 restic, Sun
maintenance); retired legs labelled as such
- restic-backup.md: status live, cutover recorded, post-GC memory
estimate, seed plan dated
- seafile-gc.md: online GC noted, stale 'crons commented out' removed
- ks2/: what-ks2-does-today rewritten (nothing writes to it any more),
legs table and gates reflect restic, decommission steps updated
- db-exclude replaces the plakar-era config name (script keeps a
fallback)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
backup-strategy.md: the two-tool architecture (incus copy + restic),
ASCII map of the legs (ks4 sdb / nuc pull / two S3 buckets), the
schedule table, a 30-second health check, and restore recipes for a
file, a database and a whole instance.
new-container.md: conventions for adding a service — one container
per service, no Docker (incus runs OCI natively, incus-compose for
stacks), gateway/proxy ingress, leave the snapshot schedule alone,
what the backups pick up automatically vs the one line to add to
restic-paths, DB-discovery requirements (incl. the .my.cnf trap), a
verification run, and the pitfalls (tmpfs /tmp, tiny-file swarms,
cron PATH).
README: start-here links, ks4 durability bullet and flow chart
updated to the restic era.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Documents library identified as commit-storm source (19M fs objects,
~4.9M commits for 2.9G of data); its commit store survives GC and is
retired by recreating the library after fixing the offending client.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
GC never ran in the server's life: 13 ghost libraries, unlimited
history on 16/17 repos, 867G blocks vs 710G live data, and an
fs/commits object swarm that makes per-file backup walks take days.
Procedure: pre-GC zfs snapshot, keep_days=90 default, offline
seaf-gc (+ --rm-deleted, --rm-fs), measure, quarterly GC schedule,
then re-decide the S3 backup mechanism and the 13.x upgrade.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Giants merged into restic-data (single nightly invocation, one index
load, --group-by host retention); instances leg gets per-instance
mountpoints — restic picks parents by host+path, a shared mountpoint
would cause nightly full re-reads.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Four buckets (data/nextcloud/seafile/instances), instances leg via
incus file mount (no plugin architecture in restic/rustic — mount the
source instead of forking the tool), nightly driver spec, parallel
plakar comparison week, plakar retirement plan (kloset kept for
PlakarKorp/plakar#2338 debugging), interim ks2 replica push until the
FTTH/nuc leg.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
pack-size 64 MiB (object count vs prune amplification), split repos
for index RAM, cache on /backup, read-concurrency vs the seek-bound
walk, nextcloud preview excludes, ARC floor before seeding, weekly
rotating read-data-subset verification, and the analysis of a full
plakar->restic migration (conclusion: the split is a reasonable end
state, gated on PlakarKorp/plakar#2338).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Per-file access to cold, tiny-file trees on HDD is ~275 IOPS whatever
the importer; only zfs send reads sequentially. Options recorded,
direction: per-file for the 16 small instances, zfs send for
nextcloud/seafile.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- status: S3 kloset seeded, restore-tested, nightly cron installed
- restore-test paths moved off /tmp — it is a 16G tmpfs on ks4 and the
10 GiB dumps restore sat in RAM squeezing the ZFS ARC
- seafile-fs note rewritten: cause unknown, local-kloset bisect next;
nextcloud 3-4h/night cost tracked for the upstream growth issue
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Seafile's block store wedged the fs walker for 12h in an lstat
preemption/EINTR livelock on the live pool; coverage moves to the
incus leg reading the sdb replica. Restore test + cron are the
remaining steps of the data leg.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
scp a .torrent into /srv/media/.watchdir and Transmission auto-downloads
it. Documents the settings (watch-dir-force-generic + trash-original),
the correct active config path (--config-dir .../info), and the perms.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Update host NIC/interfaces (gw + DNS to public 1.1.1.1/9.9.9.9, no
container dependency), fix transmission-bt kill-switch routes, and add a
note that DHCP instances self-heal while static ones (privoxy,
transmission-bt) must be updated by hand. Also spell out the macvlan
test-from-inside/external-host rule.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add udev rule + oneshot service + script on nuc that restart the
container kiosk when the Logitech Unifying receiver is replugged, since
hotplug uevents don't cross into the container's netns and cage only
enumerates input at startup.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Applied live to the 5 running containers; install scripts and the
post-install checklist now include the timedatectl step.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
README with instance tables and nuc<->ks4 network flow chart;
per-container install/troubleshooting docs for nuc (jellyfin
server/client, transmission-bt, bare-metal reinstall) and the
ks4 two-leg backup scheme (incus-copy over wireguard).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
