Standardised 2026-08-31 on regular crontabs: nuc's push moved off its
systemd timer (units kept disabled on disk), and the cron.d files on
nuc and nas were folded into root's crontab. Notes the consequence
accepted for nuc: a night with the box powered off is skipped rather
than caught up after boot.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The plan gated nuc's tunnel teardown on the nas leg being seeded, on the
assumption nuc stayed a viable fallback target. It is not one: its
ks4backup pool was deleted when the disk moved, and its data pool is a
512 GB SSD against a ~1.75 TiB replica set. So the tunnel was doing
nothing except re-establishing a keepalive'd link to ks4 on every boot of
a machine that is now powered off between uses.
Disabled 2026-08-31 (wg-quick@wg-ks4 disabled, interface down, ks4 incus
remote removed from nuc). The config and key are deliberately kept, so it
is one systemctl away if ever needed — deleting them would mean
regenerating keys and re-peering on ks4.
transmission-bt is unaffected: its tunnel is in-container and a separate
peer (10.8.0.21), verified still handshaking with egress 193.70.35.17.
Remaining: drop nuc's now-unused peer on ks4's wireguard container.
Harmless to leave, safe to do any time, recorded with the pubkey.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ef88ef9 updated the headings but left the body describing the old setup,
so the two docs contradicted each other on the same procedure.
The one that would actually have failed: `incus storage create ks4backup
zfs source=usb4t/backup/ks4` — that pool no longer exists, the disk moved
to nas and the pool was renamed to `tank` on import. Also corrects the
WireGuard peer (nas is 10.8.0.22/32, not nuc's 10.8.0.20/32, which is
retired once nas is seeded), the host for the cron and the restore test,
and drops the "replicas live only on the USB drive" framing — direct SATA
was the entire point of the rebuild.
Adds a pointer to ks2/nas-seed.md as the authoritative seed procedure.
Historical notes (the 2026-08-09 verification, the homeassistant VM
measurement) are left as-is: they are dated observations, not steps.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both hosts were powered off and brought back from a cold start, which is
the first real test of everything built on 2026-08-30.
nas: both pools imported from /etc/zfs/zpool.cache, all instances
autostarted, NFS exports republished, 0 failed units. nuc: /dev/dri
present with 0 i915 warnings (booted with the display unplugged), NFS
auto-remounted, VAAPI transcode at 9.5x realtime.
Records the ordering that actually cleared tank's inherited
<metadata>:<0x0> and <0x3d>: a scrub alone found 0 errors and repaired 0B
but left them, and a plain `zpool clear` afterwards did not drop them —
ZFS flushes the persistent error log on a scrub run *after* the clear.
That matters beyond tidiness, because while those entries stand
`zpool status -x` reports the pool unhealthy forever and zpool-health.sh
cannot signal anything new.
Two kiosk corrections, both from observed behaviour:
- the Pioneer DAC being switched off is the most likely cause of
"video, no sound" — asound.conf pins the ALSA default to it by card
name, so `default` fails to open outright and mpv falls back to null
silently. Adds the one-line aplay check.
- hotplugging the display makes cage exit once and Restart=on-failure
recovers it ~5s later. Do NOT restart it by hand; check
ActiveEnterTimestamp against the hotplug time first.
Also flags that the OS mirror is still untested with a disk physically
unplugged — it is a guess until then.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
backup-strategy (the entry point) still described nuc pulling the ks4
replicas: the leg, its WireGuard peer and the target pool now live on
nas with the 4 TB on direct SATA. Also: ks4 README flow chart redrawn
for the new topology, incus-copy leg 2 retargeted, usb4t-dropouts
marked RESOLVED (kept for the diagnosis method and the alerting gap),
and ks2/plan records that the interim push was deliberately not
re-enabled.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
nuc keeps only what needs its iGPU. blocky, privoxy and transmission-bt
moved to nas so the box can be powered off when not watching Jellyfin or
using the Spotify kiosk.
- /srv/media is now an NFSv4 mount from nas; jellyfin-server reads it
with shift=false (idmapped mounts are unsupported on NFS, as the
container doc already noted for CIFS) and readonly=true
- replication to nas is a systemd timer with Persistent=true, not cron —
an on-demand host misses its 03:30 window and cron cannot catch up
Two failures documented in full, both diagnosed from the wrong layer
first:
- booting with the TV connected and powered on kills the i915 probe
(drm_WARN_ON in intel_modeset_setup_hw_state), so /dev/dri never
appears, snd_hda_intel deferred-probes forever holding the PCI device
lock, and incusd blocks in sriov_numvfs_show — no container starts at
all, including LAN DNS. Identical on 6.12.107 and 6.12.105.
- the kiosk input gid mismatch is real but was NOT the cause of the
2026-08-30 outage (flat K400 batteries were); seatd opens input devices
as root, so kiosk group membership is not on that path. Records the
one-line raw capture that settles hardware-vs-software immediately.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
nuc-seed.md -> nas-seed.md. The leg was designed around nuc's USB pool,
which is exactly the device it must not depend on. Target pool ks4backup
now lives on tank; nas becomes WG peer 10.8.0.22 and nuc's tunnel retires
once seeded — nuc no longer needs one at all, since transmission-bt (the
only other user) moved to nas with its own in-container tunnel.
ks4 needs no change: traffic arrives masqueraded as the wireguard
container whichever peer sent it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Builds the box that ends the usb4t dropouts: the JMicron bridge was the
least reliable device in the setup and it held the intended off-site copy
of ks4. The 4 TB now sits on direct SATA as pool `tank`.
- OS on mdraid RAID1 across two 120 GB SSDs (Intel 330 + Toshiba Q300),
both ESPs bootable; `incus` ZFS mirror on their tails, ~22% left
unallocated as over-provisioning
- media at /export/media, exported read-only over NFSv4 to nuc
- transmission-bt moves here (its WireGuard tunnel is in-container, so
ks4 needed no change) and writes to the dataset locally
- backup pools nucbackup / ks4backup / nasbackup
- monitoring live: msmtp (submission+auth, verified 250), zed with
NOTIFY_DATA, zpool-health.sh every 15 min, smartd on all three disks
Traps recorded because none of them point at their own cause: booting
with the display active kills the i915 probe and wedges incus; d-i picks
grub-pc vs grub-efi from how the installer booted; `incus storage create`
hangs forever on a mountpoint=none dataset; the BMC is deliberately never
cabled, so there is no out-of-band console.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Aug 29 16:22 the enclosure dropped again despite the new cable, pool
suspended, and ZED sent nothing (suspended pools keep the vdev ONLINE;
data-class events were muted). Enclosure replacement is now the prime
candidate and a gate before the pool hosts the ks4 off-site leg.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The metadata/file errors were artefacts of the interrupted writes, not
corruption. The 2.5h scrub doubled as a soak test of the new cable.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
18/18 instances replicated, restic seed complete and restore-tested
(tree identical to live; dump loads into a scratch MariaDB, 12/12
tables). Remaining ks2 gates are the nuc leg after FTTH.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Records the working chain and the two dead ends: port 25 gets the mail
rspamd-rejected (dynamic home IP, no SPF alignment), and the missing
PTR delays the 587 greeting enough that short-timeout probes look like
a firewall block. Credentials live only in /etc/msmtprc (600).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Nightly replication had been failing since Aug 17 behind a misleading
incus error; root cause is a JMicron USB bridge dropping off the bus
(61 disconnects in 30 days) and suspending the pool — the disk itself
is SMART-clean. Records the recovery (clear/scrub), the damage, the
ordered fixes, and why nothing alerted: zed was running without an MTA
on the host.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
backup-strategy.md now leads with the three target legs (local incus
push to sdb, nuc pull over WireGuard, restic to S3 for DBs and
selected trees) and carries the plakar/ks2 story in a closing History
section. restic-backup.md, install.md, local-backup-cron.md and the
ks2 docs describe the predecessor only as 'the first S3
implementation', with one operational note that plakar is still
installed for issue #2338 reproduction.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Decided 2026-08-28: instances are protected by replication (sdb now,
nuc after FTTH, ks2 push in between), their data/DBs/configs by
restic-data — a third copy of instance filesystems in S3 was not
worth the moving parts. backup-strategy.md now explains the
replication-vs-backup split and the rebuild path when no replica
survives; restic-backup.md §6 is marked SHELVED (script and empty
repo kept).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- nuc/README.md and ks4/README.md carry the host sections (+ network
flows) that lived in the top-level README; links rebased
- top README: repo links (doc/scripts on git.lutran.fr), index points
at the new per-host pages
- cross-repo references now use https://git.lutran.fr/julien/scripts
instead of relative ../scripts paths that resolve nowhere
- plakar-s3-data.md and plakar-incus-integration.md marked SUPERSEDED
/ RETIRED with pointers to restic-backup.md; their measurements and
rationale kept
- install.md, local-backup-cron.md, incus-copy.md: crontab sections
updated to the live schedule (01:00 replicas, 05:00 restic, Sun
maintenance); retired legs labelled as such
- restic-backup.md: status live, cutover recorded, post-GC memory
estimate, seed plan dated
- seafile-gc.md: online GC noted, stale 'crons commented out' removed
- ks2/: what-ks2-does-today rewritten (nothing writes to it any more),
legs table and gates reflect restic, decommission steps updated
- db-exclude replaces the plakar-era config name (script keeps a
fallback)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
backup-strategy.md: the two-tool architecture (incus copy + restic),
ASCII map of the legs (ks4 sdb / nuc pull / two S3 buckets), the
schedule table, a 30-second health check, and restore recipes for a
file, a database and a whole instance.
new-container.md: conventions for adding a service — one container
per service, no Docker (incus runs OCI natively, incus-compose for
stacks), gateway/proxy ingress, leave the snapshot schedule alone,
what the backups pick up automatically vs the one line to add to
restic-paths, DB-discovery requirements (incl. the .my.cnf trap), a
verification run, and the pitfalls (tmpfs /tmp, tiny-file swarms,
cron PATH).
README: start-here links, ks4 durability bullet and flow chart
updated to the restic era.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Documents library identified as commit-storm source (19M fs objects,
~4.9M commits for 2.9G of data); its commit store survives GC and is
retired by recreating the library after fixing the offending client.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
GC never ran in the server's life: 13 ghost libraries, unlimited
history on 16/17 repos, 867G blocks vs 710G live data, and an
fs/commits object swarm that makes per-file backup walks take days.
Procedure: pre-GC zfs snapshot, keep_days=90 default, offline
seaf-gc (+ --rm-deleted, --rm-fs), measure, quarterly GC schedule,
then re-decide the S3 backup mechanism and the 13.x upgrade.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Giants merged into restic-data (single nightly invocation, one index
load, --group-by host retention); instances leg gets per-instance
mountpoints — restic picks parents by host+path, a shared mountpoint
would cause nightly full re-reads.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Four buckets (data/nextcloud/seafile/instances), instances leg via
incus file mount (no plugin architecture in restic/rustic — mount the
source instead of forking the tool), nightly driver spec, parallel
plakar comparison week, plakar retirement plan (kloset kept for
PlakarKorp/plakar#2338 debugging), interim ks2 replica push until the
FTTH/nuc leg.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
pack-size 64 MiB (object count vs prune amplification), split repos
for index RAM, cache on /backup, read-concurrency vs the seek-bound
walk, nextcloud preview excludes, ARC floor before seeding, weekly
rotating read-data-subset verification, and the analysis of a full
plakar->restic migration (conclusion: the split is a reasonable end
state, gated on PlakarKorp/plakar#2338).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Per-file access to cold, tiny-file trees on HDD is ~275 IOPS whatever
the importer; only zfs send reads sequentially. Options recorded,
direction: per-file for the 16 small instances, zfs send for
nextcloud/seafile.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- status: S3 kloset seeded, restore-tested, nightly cron installed
- restore-test paths moved off /tmp — it is a 16G tmpfs on ks4 and the
10 GiB dumps restore sat in RAM squeezing the ZFS ARC
- seafile-fs note rewritten: cause unknown, local-kloset bisect next;
nextcloud 3-4h/night cost tracked for the upstream growth issue
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Seafile's block store wedged the fs walker for 12h in an lstat
preemption/EINTR livelock on the live pool; coverage moves to the
incus leg reading the sdb replica. Restore test + cron are the
remaining steps of the data leg.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
scp a .torrent into /srv/media/.watchdir and Transmission auto-downloads
it. Documents the settings (watch-dir-force-generic + trash-original),
the correct active config path (--config-dir .../info), and the perms.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Update host NIC/interfaces (gw + DNS to public 1.1.1.1/9.9.9.9, no
container dependency), fix transmission-bt kill-switch routes, and add a
note that DHCP instances self-heal while static ones (privoxy,
transmission-bt) must be updated by hand. Also spell out the macvlan
test-from-inside/external-host rule.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
