FTTH is up, so the leg nas-seed.md had been holding since 2026-08-30 is
now built: wg-ks4 on nas (10.8.0.22), nas peered on ks4's wireguard
container, incus remote over the tunnel, 05:00 cron, and the first pass
seeding under a systemd-run unit.
Three corrections the runbook needed, all found by running it:
- nas had no wireguard-tools at all. transmission-bt carries its own
tunnel inside the container, so the host never needed them.
- Every ks4 instance has an instance-level eth0 pinned to incusbr0 with
a static 192.168.1.x, so each copy failed in under a second with
"Cannot use manually specified ipv4.address when using unmanaged
parent bridge". nas now runs a managed incusbr0 on 192.168.1.254/24 —
deliberately not .1, which must keep resolving over wg-ks4.
- The seed command was missing -p backup, which the doc's own
verification step already assumed.
Also records the measured rate: ~125 Mbit/s, ks4's OVH uplink rather
than the home downlink, so ~31 h for the first pass — during which the
shared incus-copy lock suppresses the 04:00 nasbackup job.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Follow-up to a7f1ac6. The Archer C7's DHCP reservations did not carry
over to the FTTH box, so anything still on DHCP was one lease renew away
from moving — as blocky already demonstrated by taking LAN DNS with it.
jellyfin-server and jellyfin-client are now static too; homeassistant
stays dynamic on purpose (it never had a reservation).
Records three things that cost time to find:
- jellyfin-client can never use netplan: the kiosk raw.lxc bind-mounts
the host /run/udev read-only, so netplan generate fails and config
silently does not regenerate at boot. It uses systemd-networkd now.
- LAPTOP719974 and patate also lost their reserved addresses.
- ks4 cannot serve as an IPv6 exit: it has a global v6 address and a
default v6 route but no working v6 egress, so extending the torrent
tunnel to ::/0 is not an option. IPv6 stays disabled in the container.
Also notes the C7 + LTE box kept as fallback: only the gateway differs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Archer C7 at .2 is gone; the FTTH box at .1 is the gateway. Every
static holdout (nas host, nuc host, privoxy, transmission-bt) pointed at
the dead .2 and had no internet — the reported symptom was privoxy.
Also record two things the migration broke that were not obvious:
- DHCP reservations did not carry over. blocky held .254 by reservation
on the C7; a lease renew on the FTTH box moved it and took LAN DNS
down. It is static now. jellyfin-* are still DHCP on stale leases.
- The FTTH box advertises native IPv6. transmission-bt's tunnel is
AllowedIPs = 0.0.0.0/0, so v6 egressed around the kill switch on the
home address. IPv6 is now disabled in that container.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
nas ran none: the script was deployed but never scheduled, while nuc
(lower stakes) upgraded nightly. Kept clear of the future 05:00 ks4
pull.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Finished downloads stopped appearing in Jellyfin after the 2026-08-30
storage move, and everything looked healthy: the file was on nas, visible
through NFS inside the container, and /media/downloads is a configured
library. The cause is that Jellyfin watches libraries with inotify, which
only reports changes made through the local mount — transmission now
writes on nas while jellyfin-server reads over NFS on nuc, so no event
ever reaches it. EnableRealtimeMonitor is true and SupportsLibraryMonitor
reports true, which is why it looks fine. Previously both shared one local
dataset on nuc and it worked.
transmission now calls Jellyfin's /Library/Refresh via script-torrent-done.
The hook always exits 0 and never blocks (transmission runs it
synchronously; a hanging hook stalls the daemon), and both the success and
missing-key paths are tested. nuc is usually powered off, so a failed
request is expected and logged rather than treated as an error — the
scheduled scan catches up.
Also records that nuc's mount is read-only, so reorganising downloads into
movies/tv-shows must now happen on nas, and the ordered checklist for
"my download is not in Jellyfin" — the first three checks all passed when
this was hit, which is what made it confusing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Standardised 2026-08-31 on regular crontabs: nuc's push moved off its
systemd timer (units kept disabled on disk), and the cron.d files on
nuc and nas were folded into root's crontab. Notes the consequence
accepted for nuc: a night with the box powered off is skipped rather
than caught up after boot.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The plan gated nuc's tunnel teardown on the nas leg being seeded, on the
assumption nuc stayed a viable fallback target. It is not one: its
ks4backup pool was deleted when the disk moved, and its data pool is a
512 GB SSD against a ~1.75 TiB replica set. So the tunnel was doing
nothing except re-establishing a keepalive'd link to ks4 on every boot of
a machine that is now powered off between uses.
Disabled 2026-08-31 (wg-quick@wg-ks4 disabled, interface down, ks4 incus
remote removed from nuc). The config and key are deliberately kept, so it
is one systemctl away if ever needed — deleting them would mean
regenerating keys and re-peering on ks4.
transmission-bt is unaffected: its tunnel is in-container and a separate
peer (10.8.0.21), verified still handshaking with egress 193.70.35.17.
Remaining: drop nuc's now-unused peer on ks4's wireguard container.
Harmless to leave, safe to do any time, recorded with the pubkey.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ef88ef9 updated the headings but left the body describing the old setup,
so the two docs contradicted each other on the same procedure.
The one that would actually have failed: `incus storage create ks4backup
zfs source=usb4t/backup/ks4` — that pool no longer exists, the disk moved
to nas and the pool was renamed to `tank` on import. Also corrects the
WireGuard peer (nas is 10.8.0.22/32, not nuc's 10.8.0.20/32, which is
retired once nas is seeded), the host for the cron and the restore test,
and drops the "replicas live only on the USB drive" framing — direct SATA
was the entire point of the rebuild.
Adds a pointer to ks2/nas-seed.md as the authoritative seed procedure.
Historical notes (the 2026-08-09 verification, the homeassistant VM
measurement) are left as-is: they are dated observations, not steps.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both hosts were powered off and brought back from a cold start, which is
the first real test of everything built on 2026-08-30.
nas: both pools imported from /etc/zfs/zpool.cache, all instances
autostarted, NFS exports republished, 0 failed units. nuc: /dev/dri
present with 0 i915 warnings (booted with the display unplugged), NFS
auto-remounted, VAAPI transcode at 9.5x realtime.
Records the ordering that actually cleared tank's inherited
<metadata>:<0x0> and <0x3d>: a scrub alone found 0 errors and repaired 0B
but left them, and a plain `zpool clear` afterwards did not drop them —
ZFS flushes the persistent error log on a scrub run *after* the clear.
That matters beyond tidiness, because while those entries stand
`zpool status -x` reports the pool unhealthy forever and zpool-health.sh
cannot signal anything new.
Two kiosk corrections, both from observed behaviour:
- the Pioneer DAC being switched off is the most likely cause of
"video, no sound" — asound.conf pins the ALSA default to it by card
name, so `default` fails to open outright and mpv falls back to null
silently. Adds the one-line aplay check.
- hotplugging the display makes cage exit once and Restart=on-failure
recovers it ~5s later. Do NOT restart it by hand; check
ActiveEnterTimestamp against the hotplug time first.
Also flags that the OS mirror is still untested with a disk physically
unplugged — it is a guess until then.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
backup-strategy (the entry point) still described nuc pulling the ks4
replicas: the leg, its WireGuard peer and the target pool now live on
nas with the 4 TB on direct SATA. Also: ks4 README flow chart redrawn
for the new topology, incus-copy leg 2 retargeted, usb4t-dropouts
marked RESOLVED (kept for the diagnosis method and the alerting gap),
and ks2/plan records that the interim push was deliberately not
re-enabled.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
nuc keeps only what needs its iGPU. blocky, privoxy and transmission-bt
moved to nas so the box can be powered off when not watching Jellyfin or
using the Spotify kiosk.
- /srv/media is now an NFSv4 mount from nas; jellyfin-server reads it
with shift=false (idmapped mounts are unsupported on NFS, as the
container doc already noted for CIFS) and readonly=true
- replication to nas is a systemd timer with Persistent=true, not cron —
an on-demand host misses its 03:30 window and cron cannot catch up
Two failures documented in full, both diagnosed from the wrong layer
first:
- booting with the TV connected and powered on kills the i915 probe
(drm_WARN_ON in intel_modeset_setup_hw_state), so /dev/dri never
appears, snd_hda_intel deferred-probes forever holding the PCI device
lock, and incusd blocks in sriov_numvfs_show — no container starts at
all, including LAN DNS. Identical on 6.12.107 and 6.12.105.
- the kiosk input gid mismatch is real but was NOT the cause of the
2026-08-30 outage (flat K400 batteries were); seatd opens input devices
as root, so kiosk group membership is not on that path. Records the
one-line raw capture that settles hardware-vs-software immediately.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
nuc-seed.md -> nas-seed.md. The leg was designed around nuc's USB pool,
which is exactly the device it must not depend on. Target pool ks4backup
now lives on tank; nas becomes WG peer 10.8.0.22 and nuc's tunnel retires
once seeded — nuc no longer needs one at all, since transmission-bt (the
only other user) moved to nas with its own in-container tunnel.
ks4 needs no change: traffic arrives masqueraded as the wireguard
container whichever peer sent it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Builds the box that ends the usb4t dropouts: the JMicron bridge was the
least reliable device in the setup and it held the intended off-site copy
of ks4. The 4 TB now sits on direct SATA as pool `tank`.
- OS on mdraid RAID1 across two 120 GB SSDs (Intel 330 + Toshiba Q300),
both ESPs bootable; `incus` ZFS mirror on their tails, ~22% left
unallocated as over-provisioning
- media at /export/media, exported read-only over NFSv4 to nuc
- transmission-bt moves here (its WireGuard tunnel is in-container, so
ks4 needed no change) and writes to the dataset locally
- backup pools nucbackup / ks4backup / nasbackup
- monitoring live: msmtp (submission+auth, verified 250), zed with
NOTIFY_DATA, zpool-health.sh every 15 min, smartd on all three disks
Traps recorded because none of them point at their own cause: booting
with the display active kills the i915 probe and wedges incus; d-i picks
grub-pc vs grub-efi from how the installer booted; `incus storage create`
hangs forever on a mountpoint=none dataset; the BMC is deliberately never
cabled, so there is no out-of-band console.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Aug 29 16:22 the enclosure dropped again despite the new cable, pool
suspended, and ZED sent nothing (suspended pools keep the vdev ONLINE;
data-class events were muted). Enclosure replacement is now the prime
candidate and a gate before the pool hosts the ks4 off-site leg.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The metadata/file errors were artefacts of the interrupted writes, not
corruption. The 2.5h scrub doubled as a soak test of the new cable.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
18/18 instances replicated, restic seed complete and restore-tested
(tree identical to live; dump loads into a scratch MariaDB, 12/12
tables). Remaining ks2 gates are the nuc leg after FTTH.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Records the working chain and the two dead ends: port 25 gets the mail
rspamd-rejected (dynamic home IP, no SPF alignment), and the missing
PTR delays the 587 greeting enough that short-timeout probes look like
a firewall block. Credentials live only in /etc/msmtprc (600).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Nightly replication had been failing since Aug 17 behind a misleading
incus error; root cause is a JMicron USB bridge dropping off the bus
(61 disconnects in 30 days) and suspending the pool — the disk itself
is SMART-clean. Records the recovery (clear/scrub), the damage, the
ordered fixes, and why nothing alerted: zed was running without an MTA
on the host.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
backup-strategy.md now leads with the three target legs (local incus
push to sdb, nuc pull over WireGuard, restic to S3 for DBs and
selected trees) and carries the plakar/ks2 story in a closing History
section. restic-backup.md, install.md, local-backup-cron.md and the
ks2 docs describe the predecessor only as 'the first S3
implementation', with one operational note that plakar is still
installed for issue #2338 reproduction.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Decided 2026-08-28: instances are protected by replication (sdb now,
nuc after FTTH, ks2 push in between), their data/DBs/configs by
restic-data — a third copy of instance filesystems in S3 was not
worth the moving parts. backup-strategy.md now explains the
replication-vs-backup split and the rebuild path when no replica
survives; restic-backup.md §6 is marked SHELVED (script and empty
repo kept).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- nuc/README.md and ks4/README.md carry the host sections (+ network
flows) that lived in the top-level README; links rebased
- top README: repo links (doc/scripts on git.lutran.fr), index points
at the new per-host pages
- cross-repo references now use https://git.lutran.fr/julien/scripts
instead of relative ../scripts paths that resolve nowhere
- plakar-s3-data.md and plakar-incus-integration.md marked SUPERSEDED
/ RETIRED with pointers to restic-backup.md; their measurements and
rationale kept
- install.md, local-backup-cron.md, incus-copy.md: crontab sections
updated to the live schedule (01:00 replicas, 05:00 restic, Sun
maintenance); retired legs labelled as such
- restic-backup.md: status live, cutover recorded, post-GC memory
estimate, seed plan dated
- seafile-gc.md: online GC noted, stale 'crons commented out' removed
- ks2/: what-ks2-does-today rewritten (nothing writes to it any more),
legs table and gates reflect restic, decommission steps updated
- db-exclude replaces the plakar-era config name (script keeps a
fallback)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
backup-strategy.md: the two-tool architecture (incus copy + restic),
ASCII map of the legs (ks4 sdb / nuc pull / two S3 buckets), the
schedule table, a 30-second health check, and restore recipes for a
file, a database and a whole instance.
new-container.md: conventions for adding a service — one container
per service, no Docker (incus runs OCI natively, incus-compose for
stacks), gateway/proxy ingress, leave the snapshot schedule alone,
what the backups pick up automatically vs the one line to add to
restic-paths, DB-discovery requirements (incl. the .my.cnf trap), a
verification run, and the pitfalls (tmpfs /tmp, tiny-file swarms,
cron PATH).
README: start-here links, ks4 durability bullet and flow chart
updated to the restic era.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Documents library identified as commit-storm source (19M fs objects,
~4.9M commits for 2.9G of data); its commit store survives GC and is
retired by recreating the library after fixing the offending client.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
GC never ran in the server's life: 13 ghost libraries, unlimited
history on 16/17 repos, 867G blocks vs 710G live data, and an
fs/commits object swarm that makes per-file backup walks take days.
Procedure: pre-GC zfs snapshot, keep_days=90 default, offline
seaf-gc (+ --rm-deleted, --rm-fs), measure, quarterly GC schedule,
then re-decide the S3 backup mechanism and the 13.x upgrade.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Giants merged into restic-data (single nightly invocation, one index
load, --group-by host retention); instances leg gets per-instance
mountpoints — restic picks parents by host+path, a shared mountpoint
would cause nightly full re-reads.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Four buckets (data/nextcloud/seafile/instances), instances leg via
incus file mount (no plugin architecture in restic/rustic — mount the
source instead of forking the tool), nightly driver spec, parallel
plakar comparison week, plakar retirement plan (kloset kept for
PlakarKorp/plakar#2338 debugging), interim ks2 replica push until the
FTTH/nuc leg.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
pack-size 64 MiB (object count vs prune amplification), split repos
for index RAM, cache on /backup, read-concurrency vs the seek-bound
walk, nextcloud preview excludes, ARC floor before seeding, weekly
rotating read-data-subset verification, and the analysis of a full
plakar->restic migration (conclusion: the split is a reasonable end
state, gated on PlakarKorp/plakar#2338).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Per-file access to cold, tiny-file trees on HDD is ~275 IOPS whatever
the importer; only zfs send reads sequentially. Options recorded,
direction: per-file for the 16 small instances, zfs send for
nextcloud/seafile.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- status: S3 kloset seeded, restore-tested, nightly cron installed
- restore-test paths moved off /tmp — it is a 16G tmpfs on ks4 and the
10 GiB dumps restore sat in RAM squeezing the ZFS ARC
- seafile-fs note rewritten: cause unknown, local-kloset bisect next;
nextcloud 3-4h/night cost tracked for the upstream growth issue
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Seafile's block store wedged the fs walker for 12h in an lstat
preemption/EINTR livelock on the live pool; coverage moves to the
incus leg reading the sdb replica. Restore test + cron are the
remaining steps of the data leg.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
