doc: record the real ks4 pull bottleneck — source disk, not the link

c8d759c claimed the ~125 Mbit/s seed rate was ks4's OVH uplink. That was
wrong, and measuring it says so: ks4 uploads at 609 Mbit/s, downloads at
900, and nas pulls 670 from OVH's network. WireGuard is not it either —
zero UdpRcvbufErrors, wg-crypt kworkers at ~2%, both hosts ~85% idle.

The limit is `data` sitting on sdb5, one 7200 rpm HGST 6 TB disk: during
the send it does 109 r/s at 14 MB/s with ~131 KB requests and a queue
depth of ~1.0, which is the random-IOPS ceiling of a single HDD reading a
fragmented 1.42 TiB dataset. 14 MB/s is ~112 Mbit/s on the wire, exactly
what we see, while the network sits idle.

Parallelism is the only lever and it works by disk queue depth, not
bandwidth: a second concurrent copy takes sdb from 14 to 21 MB/s and the
request size from 131 KB to 514 KB, lifting the tunnel from 113 to 153
Mbit/s. Recorded, along with the decision to keep the seed sequential —
one-off pass, incremental refreshes after, and incus-copy.sh is shared by
all three legs.

Also corrects the set size to the actual 1.42 TiB (~29 h, not ~31 h).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

backup-strategy.md

@@ -9,7 +9,7 @@ docs; this page is the map.
 | leg | mechanism | protects |
 |---|---|---|
 | **local replication** | `incus copy --refresh` → pool `backup` on ks4's second disk (sdb) | instances, against losing the `data` pool |
-| **remote replication** | nuc pulls the same replicas over WireGuard → pool `ks4backup` | instances, against losing ks4 or the site |
+| **remote replication** | **nas** pulls the same replicas over WireGuard → pool `ks4backup` (on `tank`, direct SATA) | instances, against losing ks4 or the site |
 | **remote backup** | **restic** → S3 bucket `restic-data`: database dumps + selected filesystem trees | the data itself, versioned and encrypted, independent of every disk above (a nightly run walks 1.1 M files / 1.24 TiB in under two minutes) |
 
 Only two tools are involved: `incus copy` (ZFS-incremental, native to
@@ -18,15 +18,22 @@ encryption, local metadata cache, so a night costs only the churn —
 [ks4/restic-backup.md](ks4/restic-backup.md)).
 
 The old backup server **ks2 is being retired** (decommission by
-Sep 30, 2026 — [ks2/plan.md](ks2/plan.md)); until FTTH enables the nuc
-leg it still receives an interim replica push.
+Sep 30, 2026 — [ks2/plan.md](ks2/plan.md)); nothing is written to it
+any more.
 
-**Status 2026-08-28**: local replication and the S3 backup leg are
+**Status 2026-08-31**: local replication and the S3 backup leg are
 live, and the S3 leg has been **restore-tested** (a file tree came
 back identical to the live one; a database dump loaded into a scratch
-server with all its tables). The **nuc pull leg waits for FTTH**
-(expected before end of September); until then the ks2 push stands
-in. Backing up whole
+server with all its tables). The **nas pull leg waits for FTTH**
+(expected before end of September) — it moved off nuc on 2026-08-30,
+onto a host where the 4 TB disk is on direct SATA rather than a USB
+bridge that suspended the pool weekly
+([nas/README.md](nas/README.md), [ks2/nas-seed.md](ks2/nas-seed.md)).
+In the meantime instances have no *fresh* off-site copy: the ks2 push
+was deliberately not re-enabled (a 3-week-old replica set on a
+94 %-full pool that is about to be wiped), so off-site protection
+rests on `restic-data`, which holds the data, the databases and the
+incus configuration needed to rebuild. Backing up whole
 instance *images* to S3 was considered and left out — S3 holds the
 data, the databases and the incus configuration, which is what a
 rebuild needs.
@@ -39,10 +46,10 @@ rebuild needs.
  │  live instances       pool `data` (sda)    │
  │  nextcloud, seafile, mail, git, ...        │
  │        │                                   │
- │        │ 01:00  incus copy --refresh       │     nuc (home LAN, via WireGuard)
+ │        │ 01:00  incus copy --refresh       │     nas (home LAN, via WireGuard)
  │        ▼                                   │     ┌───────────────────────────┐
  │  replicas (stopped)   pool `backup` (sdb)  │────▶│ 05:00 incus copy (pull)   │
- │  project `backup`     ready to start       │     │ pool `ks4backup` (USB)    │
+ │  project `backup`     ready to start       │     │ ks4backup on `tank` (SATA)│
  │                                            │     └───────────────────────────┘
  │  ────────────────────────────────────────  │
  │                                            │     OVH Object Storage (S3, sbg)
@@ -59,7 +66,7 @@ rebuild needs.
 Two different kinds of protection, on purpose:
 
 - **instances** (the running systems) are protected by *replication* —
-  a ready-to-start copy on ks4's second disk and, after FTTH, on nuc.
+  a ready-to-start copy on ks4's second disk and, after FTTH, on nas.
   Restoring one is `incus copy` + `incus start`.
 - **the data inside them** (files, databases, incus configuration) is
   protected by *backup* — encrypted, deduplicated, versioned on S3,
@@ -77,10 +84,9 @@ instances two homes.
 | when | what | log |
 |---|---|---|
 | 01:00 daily | `incus-copy.sh -p backup -s backup` — refresh all replicas onto sdb | `/var/log/incus-copy.log` |
-| 02:00 daily | `incus-copy.sh -d ks2 -m push` — interim off-site replicas, until the nuc leg replaces it | `/var/log/incus-copy.log` |
 | 03:00 daily | instance snapshots (incus profile, 7-day expiry) — what keeps the refreshes incremental | `incus info <inst>` |
 | 05:00 daily | `restic-backup.sh` — dumps + data trees → S3 | `/var/log/restic-backup.log` |
-| 05:00 daily (nuc) | nuc pulls all replicas over WireGuard | nuc: `/var/log/incus-copy-ks4.log` |
+| 05:00 daily (**nas**) | nas pulls all ks4 replicas over WireGuard → `ks4backup` (after FTTH) | nas: `/var/log/incus-copy-ks4.log` |
 | Sun 14:00 | `restic-maintenance.sh` — prune (capped), structure check, 1/52 data verification | `/var/log/restic-maintenance.log` |
 | 1st of month 06:00 / quarterly 15th | seafile GC dry-run report / real GC ([ks4/seafile-gc.md](ks4/seafile-gc.md)) | `/var/log/seafile-gc.log` |
 
@@ -101,8 +107,8 @@ never silently.
 
 - **A whole instance, fast (same box)**: `incus copy backup:<inst>`
   style — copy the replica from project `backup` back into `default`
-  ([ks4/incus-copy.md](ks4/incus-copy.md)); from nuc the same via the
-  remote.
+  ([ks4/incus-copy.md](ks4/incus-copy.md)); from nas the same via the
+  incus remote.
 - **A file or directory** (any date within retention):
   `restic -r s3:s3.sbg.io.cloud.ovh.net/restic-data restore <snapshot> --target /backup/restore-x --include <path>`
   (never restore into `/tmp` — it is RAM). Env: `. /root/.restic-env`.

ks2/nas-seed.md

@@ -1,6 +1,9 @@
 # ks4 pull leg — seed after FTTH
 
-Status: **prepared, waiting on the FTTH link.**
+Status: **built and seeding — 2026-09-16.** FTTH is up (gateway
+`192.168.0.1`, see [nuc/nuc-install.md](../nuc/nuc-install.md)); the
+tunnel, the incus remote and the cron are in place and the first pass is
+running. Remaining: let the seed finish, then the test-restore below.
 
 ⚠️ **Changed 2026-08-30: this leg lands on `nas`, not on nuc.** It was
 originally designed for nuc's USB pool `usb4t`, but that pool proved to
@@ -14,30 +17,47 @@ SATA on the new host `nas` (`192.168.0.4`,
 Consequences versus the original plan:
 
 - Target pool `ks4backup` is now backed by `tank/backup/ks4` on nas.
-- The **WireGuard tunnel moves too**: nas becomes peer `10.8.0.22`;
-  nuc's `wg-ks4` (`10.8.0.20`) is retired once this works. nuc no longer
-  needs a tunnel at all — `transmission-bt`, its only other user, now
-  runs on nas and carries its own in-container tunnel (`10.8.0.21`,
-  unchanged, ks4 needs no edit for it).
+- The **WireGuard tunnel moves too**: nas becomes peer `10.8.0.22`.
+  `transmission-bt`, the only other user, moved to nas and carries its
+  own in-container tunnel (`10.8.0.21`, unchanged — ks4 needs no edit
+  for it).
+- ⚠️ **nuc's `wg-ks4` was already disabled on 2026-08-31**, *before* the
+  seed, not after. The original plan retired it only once nas was
+  seeded, on the assumption nuc could serve as a fallback target — it
+  cannot: its `ks4backup` pool was deleted and its `data` pool is a
+  512 GB SSD, far too small for the ~1.75 TiB replica set. Keeping a
+  keepalive'd tunnel alive on a machine that is now powered off between
+  uses bought nothing. `wg-quick@wg-ks4` is `disabled`, and the `ks4`
+  incus remote was removed from nuc.
+  `/etc/wireguard/wg-ks4.conf` and its key are **kept**, so it is one
+  `systemctl enable --now wg-quick@wg-ks4` away if ever needed.
 - ks4's ufw rule is unchanged: traffic arrives masqueraded as the
   `wireguard` container (`192.168.1.18`) whichever peer sent it.
 
 ## Prerequisites
 
-- [ ] `tank` running on SATA for ≥ 7 days with zero pool suspensions —
-      the gate that replaces "fix the USB enclosure"
-- [ ] FTTH up (the first pass moves ~1.75 TiB)
+- [x] `tank` running on SATA for ≥ 7 days with zero pool suspensions —
+      the gate that replaces "fix the USB enclosure".
+      Verified 2026-09-16: `ONLINE`, scrub clean 2026-09-14, uptime 2 w 2 d,
+      zero suspensions in the journal.
+- [x] FTTH up (the first pass moves ~1.75 TiB)
 
 ## Setup (root on nas)
 
 ```sh
-# 1. peer nas on ks4's wireguard container
-#    (run on ks4) — <nas-pubkey> from /etc/wireguard/wg-ks4.key on nas
+# 0. the nas host has no wireguard-tools — transmission-bt carries its own
+#    tunnel *inside* the container, so the host never needed them
+apt-get install -y wireguard-tools
+
+# 1. key on nas, then peer it on ks4's wireguard container
+umask 077; wg genkey > /etc/wireguard/wg-ks4.key
+wg pubkey < /etc/wireguard/wg-ks4.key          # -> <nas-pubkey>
+#    (run on ks4)
 incus exec wireguard -- wg set wg0 peer <nas-pubkey> allowed-ips 10.8.0.22/32
 incus exec wireguard -- wg-quick save wg0
 
 # 2. tunnel on nas: /etc/wireguard/wg-ks4.conf, modelled on nuc's
-#    Address = 10.8.0.22/32, peer pubkey TVs6d7…,
+#    Address = 10.8.0.22/24, peer pubkey TVs6d7…,
 #    Endpoint = 193.70.35.17:51845,
 #    AllowedIPs = 10.8.0.0/24, 192.168.1.1/32, keepalive 25
 systemctl enable --now wg-quick@wg-ks4
@@ -47,14 +67,100 @@ incus remote add ks4 https://192.168.1.1:8443 --accept-certificate --token '…'
 incus list ks4: | head          # sanity: remote reachable
 ```
 
-## Seed
+⚠️ **nas needs a managed `incusbr0` of its own, or every copy fails
+instantly.** All 18 ks4 instances carry an *instance-level* `eth0`
+(`nictype: bridged`, `parent: incusbr0`, `ipv4.address: 192.168.1.x`).
+nas has no such bridge, so instance creation dies with:
+
+```
+Device validation failed for "eth0": Cannot use manually specified
+ipv4.address when using unmanaged parent bridge
+```
+
+Create a managed network of the same name — but **give it `.254`, never
+`.1`**: `192.168.1.1` must keep resolving over `wg-ks4` to ks4's incus
+API, and a local address always beats a route.
 
 ```sh
-# full pull of every ks4 instance into pool ks4backup (tmux — first pass
-# moves ~1.75 TiB through the WG tunnel)
-/root/scripts/incus-copy.sh -r ks4 -s ks4backup 2>&1 | tee -a /var/log/incus-copy-ks4.log
+incus network create incusbr0 \
+    ipv4.address=192.168.1.254/24 ipv4.nat=false ipv6.address=none
+ip route get 192.168.1.1        # must still say: dev wg-ks4
 ```
 
+The bridge stays inert — replicas are never started here.
+
+## Seed
+
+⚠️ **`-p backup` is required**, exactly as for the nuc leg
+([nas/nas-install.md](../nas/nas-install.md) §9a) — and as the
+verification command below already assumed. Without it the 18 replicas
+land in `default` alongside nas's live instances.
+
+Run it detached rather than in a shell that can drop — the first pass is
+long: **~1.42 TiB at ~14 MB/s ≈ 29 h** (see the bottleneck section below
+for why it is 14 MB/s and not more).
+
+```sh
+systemd-run --unit=ks4-seed --collect \
+  /bin/bash -c '/root/scripts/incus-copy.sh -r ks4 -s ks4backup -p backup \
+                >> /var/log/incus-copy-ks4.log 2>&1'
+
+systemctl is-active ks4-seed          # progress:
+tail -f /var/log/incus-copy-ks4.log
+```
+
+⚠️ **The seed suppresses the 04:00 `nasbackup` job while it runs.**
+`incus-copy.sh` takes a single `/run/lock/incus-copy.lock` for every
+shape, so any run starting while the seed holds it aborts with
+`another incus-copy run holds …`. Over a ~29 h seed that skips one or two
+nights of the nas-local copy — accepted; those replicas are small,
+same-host, and rebuildable. The 03:30 nuc push is unaffected (it runs on
+nuc, with nuc's own lock).
+
+### Bottleneck: ks4's source disk, not the network
+
+Measured 2026-09-16, because ~112 Mbit/s looked far too slow for a 2 Gbit/s
+FTTH line. **It is not the link, and not WireGuard.** Do not go looking for
+a network fix:
+
+| Path | Measured |
+|---|---|
+| ks4 upload → internet | 609 Mbit/s |
+| ks4 download ← internet | 900 Mbit/s |
+| nas download ← OVH network | 670 Mbit/s |
+| ks4 → nas through `wg-ks4` | **~112 Mbit/s** |
+
+WireGuard was ruled out too: `UdpRcvbufErrors` 0 on nas (so the default
+`net.core.rmem_max` of 208 KB is *not* dropping packets), wg-crypt
+kworkers ~2 %, both hosts ~85 % idle.
+
+The limit is **`data` living on `sdb5`, a single 7200 rpm HGST 6 TB
+spinning disk**. During the send `iostat` showed sdb at **109 r/s /
+14 MB/s, ~131 KB average request, queue depth ~1.0** — the random-IOPS
+ceiling of one HDD walking a fragmented 1.42 TiB dataset. 14 MB/s is
+~112 Mbit/s on the wire, which is exactly the observed rate. The network
+is idle the whole time.
+
+**Parallelism is the only real lever, and it is a disk-queue effect**, not
+a bandwidth one. Running a second instance copy alongside the seed:
+
+| | sequential | + 1 parallel copy |
+|---|---|---|
+| sdb read | 14 MB/s | **21 MB/s** |
+| avg request size | 131 KB | **514 KB** |
+| tunnel | 113 Mbit/s | **153 Mbit/s** |
+
+With two senders queued, ZFS issues larger, more sequential reads instead
+of seeking one request at a time. Three or four concurrent copies would
+plausibly reach 25–30 MB/s and roughly halve the seed.
+
+**Decision 2026-09-16: keep it sequential.** The first pass is a one-off,
+later refreshes are ZFS-incremental and tiny, and `incus-copy.sh` is
+shared by all three legs — parallelising means either reworking the script
+or running copies outside its flock, i.e. two jobs contending for the same
+dataset. Not worth one overnight. If a full reseed is ever needed and the
+wall-clock matters, this is the knob; the disk is the floor either way.
+
 Notes:
 
 - First pass is a full send per instance; later refreshes are
@@ -70,9 +176,12 @@ Add to nas's root crontab, offset from the 03:30 nuc→nas push, the
 04:00 nas→nuc push and ks4's own 01:00/05:00 jobs:
 
 ```cron
-0 5 * * * /root/scripts/incus-copy.sh -r ks4 -s ks4backup >> /var/log/incus-copy-ks4.log 2>&1
+0 5 * * * /root/scripts/incus-copy.sh -r ks4 -s ks4backup -p backup >> /var/log/incus-copy-ks4.log 2>&1
 ```
 
+`/etc/logrotate.d/incus-copy` already lists `incus-copy-ks4.log`, so
+nothing to add there.
+
 ## Verification (release gate for ks2)
 
 ```sh
@@ -84,13 +193,18 @@ incus start solar-restoretest && incus exec solar-restoretest -- systemctl is-sy
 incus delete -f solar-restoretest
 ```
 
-Once verified, tick the nas gate in the [ks2 plan](plan.md) and retire
-nuc's tunnel:
+Once verified, tick the nas gate in the [ks2 plan](plan.md).
+
+Only one piece of nuc's retirement is still outstanding — dropping its
+now-unused peer on ks4. Harmless to leave (an unused peer costs nothing)
+and safe to do at any time, since nuc's tunnel is already down:
 
 ```sh
-# on nuc
-systemctl disable --now wg-quick@wg-ks4 && rm /etc/wireguard/wg-ks4.conf
-# on ks4
-incus exec wireguard -- wg set wg0 peer <nuc-pubkey> remove
+# on ks4 — nuc's pubkey is 31Tlgloc…
+incus exec wireguard -- wg set wg0 peer 31TlglocNJyooDVAO8HWEC0lyCykhbaFIWVWFUCOrmQ= remove
 incus exec wireguard -- wg-quick save wg0
 ```
+
+Deleting `/etc/wireguard/wg-ks4.conf` + `.key` on nuc is deliberately
+**not** done: they cost nothing and regenerating keys would mean
+re-peering on ks4.

ks2/plan.md

@@ -18,10 +18,14 @@ What remains on the box is **cold history**: instance replicas on pool
 snapshotted daily there (`zfs-auto-snapshot.sh`, 2-month expiry).
 
 ⚠️ While the nas leg waits for FTTH, instances have no *fresh*
-off-site copy — the ks2 push is to be re-enabled as soon as the
-initial restic sync finishes (decided 2026-08-28), and retired again
-when nas takes over. (The leg moved from nuc to the new host `nas`
-on 2026-08-30 — see [nas-seed.md](nas-seed.md).)
+off-site copy. The ks2 push was **deliberately not re-enabled**
+(2026-08-30): ks2's replicas are three weeks old, its `data` pool is
+94 % full, no common snapshot survives ks4's 7-day expiry, and the box
+is wiped within the month — so a full ~1.5 T re-send buys four weeks
+of freshness on hardware already scheduled for destruction. Off-site
+protection meanwhile rests on `restic-data` (data, databases and the
+incus configuration — enough to rebuild). (The leg moved from nuc to
+the new host `nas` on 2026-08-30 — see [nas-seed.md](nas-seed.md).)
 
 ## Inventory findings (2026-08-22)
 
@@ -53,7 +57,7 @@ on 2026-08-30 — see [nas-seed.md](nas-seed.md).)
 | off-site, S3 (data+DB) | **restic** → bucket `restic-data` ([restic-backup.md](../ks4/restic-backup.md)) | **live** (05:00) |
 | off-site, S3 (instances) | restic over `incus file mount` of the `backup`-project replicas | **shelved** 2026-08-28 (replication covers instances) |
 | off-site, nas | **nas** pulls `ks4:*` → pool `ks4backup` over WG ([nas-seed.md](nas-seed.md)) | waiting FTTH (< Sep 30); moved off nuc 2026-08-30 |
-| off-site, ks2 (interim) | `incus-copy.sh -d ks2 -m push` at 02:00 until the nas leg seeds | to re-enable once the restic seed finishes |
+| off-site, ks2 (interim) | `incus-copy.sh -d ks2 -m push` | **not re-enabled** 2026-08-30 — see above |
 
 ## Actions (backup work documented in [`../ks4/`](../ks4/), ks2-only tasks here)
 

ks4/README.md

@@ -21,40 +21,43 @@ Incus host at OVH — public-facing self-hosted services.
 - ⚠️ The ZFS `data` pool is single-disk (not mirrored); durability
   rests on nightly cron jobs — 01:00 `incus copy --refresh` of all
   instances to the local `backup` pool on sdb5, then the instance leg
-  to S3; 05:00 restic (DB dumps + data trees) to S3; nuc pulls the
-  replicas over WireGuard. Full picture and restore procedures:
+  to S3; 05:00 restic (DB dumps + data trees) to S3; **nas** pulls the
+  replicas over WireGuard (after FTTH). Full picture and restore procedures:
   **[backup-strategy.md](../backup-strategy.md)**
   ([local-backup-cron.md](local-backup-cron.md),
   [incus-copy.md](incus-copy.md),
   [restic-backup.md](restic-backup.md)).
 
-## Network flows (nuc ↔ ks4)
+## Network flows (home ↔ ks4)
 
 ```
-  nuc — home LAN 192.168.0.0/24                    ks4 — OVH 193.70.35.17
+  nas — home LAN 192.168.0.4                       ks4 — OVH 193.70.35.17
 +-----------------------------------+       +-------------------------------------+
 |                                   |       |                                     |
-| host: wg-ks4 (10.8.0.20)          |       |  [wireguard] 192.168.1.18           |
+| host: wg-ks4 (10.8.0.22)          |       |  [wireguard] 192.168.1.18           |
 |   incus remote "ks4"        ------+--WG-->|   wg0 10.8.0.1/24, udp 51845        |
 |   pull ks4:* -> pool ks4backup    | udp   |    | masquerade -> eth0             |
-|   on usb4t  [pending FTTH seed]   | 51845 |    |                                |
+|   on tank (SATA) [pending FTTH]   | 51845 |    |                                |
 |                                   |       |    +-> incus API 192.168.1.1:8443   |
 | [transmission-bt] wg0 (10.8.0.21) |       |    |   (ufw: only from .18)         |
 |   full tunnel 0.0.0.0/0     ------+--WG-->|    |                                |
 |   kill switch: no default route   | udp   |    +-> WAN egress: torrents +       |
-|   downloads -> /srv/media         | 51845 |        apt of transmission-bt       |
-|   (usb4t/media, read by jellyfin) |       |        exit as 193.70.35.17         |
+|   downloads -> /export/media      | 51845 |        apt of transmission-bt       |
+|   (NFS-exported to nuc)           |       |        exit as 193.70.35.17         |
 |                                   |       |                                     |
-| 03:00 instance snapshots          |       | 03:00 instance snapshots            |
-| 03:30 incus-copy: all instances   |       | 01:00 incus-copy: all instances     |
-|   -> project backup, pool         |       |   -> project backup, zpool sdb5     |
-|   nucbackup (usb4t/backup/nuc)    |       |   then restic instance leg -> S3    |
+| 03:30 nuc pushes its instances    |       | 03:00 instance snapshots            |
+|   -> nucbackup on tank            |       | 01:00 incus-copy: all instances     |
+| 04:00 nas replicates its own      |       |   -> project backup, zpool sdb5     |
+|   -> nasbackup on tank            |       |                                     |
 | 05:00 pull ks4:* -> ks4backup     |       | 05:00 restic: DB dumps + data       |
 |   [pending FTTH]                  |       |   trees -> S3 (restic-data)         |
-| 05:30 apt upgrade all containers  |       | Sun 14:00 restic maintenance        |
+|                                   |       | Sun 14:00 restic maintenance        |
 +-----------------------------------+       +-------------------------------------+
    phones/laptops: WG peers 10.8.0.2-3 reach 192.168.1.x through the same endpoint
+   nuc (on-demand media box) mounts /export/media from nas over NFSv4
 ```
 
-Both tunnels initiate **from** nuc (home NAT, dynamic IP) toward ks4's
+Both tunnels initiate **from home** (NAT, dynamic IP) toward ks4's
 fixed endpoint; ks4's incus API is never exposed to the internet.
+The pull leg and its tunnel moved from nuc to nas on 2026-08-30
+([nas/README.md](../nas/README.md)).

ks4/incus-copy.md

@@ -8,8 +8,8 @@ push to `ks2` (decommissioning):
 
 1. **local** — replicas + dumps on a dedicated `backup` zpool on ks4's
    second disk (`sdb5`), survives `sda` death
-2. **off-site** — replicas pulled by **nuc** into pool `ks4backup`
-   (dataset `usb4t/backup/ks4`), survives losing ks4 entirely
+2. **off-site** — replicas pulled by **nas** into pool `ks4backup`
+   (dataset `tank/backup/ks4`), survives losing ks4 entirely
 
 ## The script
 
@@ -74,24 +74,31 @@ Cron (root on ks4) — replaces both ks2 jobs:
 ⚠️ Replicas in the `backup` project must stay **stopped** — they keep
 the live containers' static `192.168.1.x` addresses.
 
-## Leg 2 — off-site pull from nuc
+## Leg 2 — off-site pull from nas (was nuc until 2026-08-30)
 
-Storage pool on nuc (done 2026-08-09): `ks4backup`, backed by the
-dataset `usb4t/backup/ks4` on the USB 4 TB pool (quota on `usb4t/backup`
-removed 2026-08-09 — full replica set is ~1.75 TiB):
+Storage pool on **nas** (recreated 2026-08-30): `ks4backup`, backed by
+the dataset `tank/backup/ks4` — the same 4 TB disk, now on **direct
+SATA** instead of the USB enclosure whose bridge kept suspending the pool
+([nuc/usb4t-dropouts.md](../nuc/usb4t-dropouts.md)). Full replica set is
+~1.75 TiB.
 
 ```sh
-incus storage create ks4backup zfs source=usb4t/backup/ks4
+incus storage create ks4backup zfs source=tank/backup/ks4
 ```
 
-Replicas live only on the USB drive — if it fails, only backups are
-lost; nuc's own instances (pool `data` on the SSD) are unaffected.
+⚠️ Originally this lived on nuc as `usb4t/backup/ks4`. That pool no
+longer exists — the disk moved to nas and the pool was renamed on import
+([nas/nas-install.md](../nas/nas-install.md) §5b). Following the old
+command fails with "no such pool".
 
-**Direction: nuc pulls, through the WireGuard tunnel.** Verified
+`tank` is a single vdev, so if the disk fails only backups are lost —
+nas's own instances live on the `incus` SSD mirror and are unaffected.
+
+**Direction: nas pulls, through the WireGuard tunnel.** Verified
 2026-08-09: ks4's API listens on wildcard `:8443` (so it answers on
 `192.168.1.1`, the `incusbr0` host address) but is **firewalled from
 the internet** — the VPN path keeps it that way, needs no inbound port
-at home, and doesn't care that nuc's public IP is dynamic. The
+at home, and doesn't care that the home public IP is dynamic. The
 `wireguard` container (`192.168.1.18`, `wg0` `10.8.0.1/24`) is exposed
 via a proxy device on public UDP `51845`.
 
@@ -99,13 +106,16 @@ Setup (✅ **done 2026-08-09**, verified end-to-end with
 `incus list ks4:` from nuc):
 
 - **wireguard container** (ks4): forwards + masquerades wg0→eth0
-  (pre-existing); nuc added as peer `10.8.0.20/32`
-  (`wg set wg0 peer 31Tlgloc… allowed-ips 10.8.0.20/32` +
-  `wg-quick save wg0`).
+  (pre-existing). **nas** is the peer for this leg —
+  `wg set wg0 peer <nas-pubkey> allowed-ips 10.8.0.22/32` +
+  `wg-quick save wg0`. (`10.8.0.20/32` was nuc's peer for the same leg
+  and is retired once nas is seeded; `10.8.0.21` is transmission-bt's
+  own in-container tunnel and is unrelated.)
 - **ufw** (ks4): `ufw allow in on incusbr0 from 192.168.1.18 to any
   port 8443 proto tcp` — the API stays firewalled from the internet
   and the connection arrives masqueraded as the WG container.
-- **nuc**: `/etc/wireguard/wg-ks4.conf` (`wg-quick@wg-ks4` enabled;
+- **nas**: `/etc/wireguard/wg-ks4.conf` (`wg-quick@wg-ks4` enabled,
+  `Address = 10.8.0.22/32`;
   peer = container pubkey `TVs6d7…`, endpoint `193.70.35.17:51845`,
   `AllowedIPs = 10.8.0.0/24, 192.168.1.1/32`, keepalive 25s) and
   `incus remote add ks4 https://192.168.1.1:8443 --accept-certificate
@@ -117,7 +127,8 @@ container over its own tunnel is fine (crash-consistent, tiny, no
 interruption); if the tunnel is down the cron job fails loudly instead
 of hanging.
 
-Then cron (root on nuc) — stagger after ks4's local leg:
+Then cron (root on **nas**) — stagger after ks4's local leg and after
+nas's own 04:00 local replication:
 
 ```cron
 30 2 * * * /root/scripts/incus-copy.sh -r ks4 -s ks4backup >> /var/log/incus-copy.log 2>&1
@@ -130,7 +141,7 @@ cron once it completes.
 ## Cutover checklist (then kill ks2)
 
 1. First full cycle of all three jobs clean (logs above).
-2. Restore test: on nuc, start a small replica (e.g. `freshrss`) with
+2. Restore test: on nas, start a small replica (e.g. `freshrss`) with
    its NIC detached, check app data, then stop it.
 3. Remove both ks2 cron lines on ks4, `incus remote remove ks2`,
    cancel the server (`164.132.173.57` = ks2, rsync target of the old
@@ -139,11 +150,15 @@ cron once it completes.
 ## Restore
 
 ```sh
-# from nuc (off-site replica):
+# from nas (off-site replica):
 incus copy <instance> ks4:<instance> --mode push
 # from the local backup project (sda replaced, pool data rebuilt):
 incus copy <instance> <instance> --project backup --target-project default -s data
 ```
 
+The step-by-step seed procedure, including the WireGuard move and the
+gate it depends on, is [ks2/nas-seed.md](../ks2/nas-seed.md) — that is
+the authoritative version for this leg.
+
 Remember replicas have `boot.autostart=false`; re-enable after a real
 failover, and re-check it after copying back to ks4.

ks4/restic-backup.md

@@ -292,6 +292,11 @@ deleted.
   3–4 h per night for *one* of those trees. This is the local-cache
   cost model working as intended: the walk is stat-only, and only the
   churn is read, chunked and uploaded.
+- **First maintenance run 2026-08-30**: `prune` 0 blobs / 0 B removed,
+  `check` and the rotating `--read-data-subset` slice both clean, 7 min,
+  `rc=0`. Nothing to reclaim because the killed seed's packs were
+  *adopted* by the successful seed's dedup rather than orphaned — worth
+  knowing before assuming an interrupted backup wastes storage.
 - **Left**: the nuc leg after FTTH, then the ks2 decommission
   ([../ks2/plan.md](../ks2/plan.md)). The instance leg (§6) is
   shelved, not pending.

nas/README.md

@@ -33,15 +33,29 @@ Storage + backup host on the LAN, added 2026-08.
 | Name | IP | Doc | Features |
 |---|---|---|---|
 | blocky | 192.168.0.254 | — | unprivileged, autostart; DNS ad-blocker for the LAN. Moved from nuc 2026-08-30 so it survives nuc being powered off |
-| privoxy | 192.168.0.11 | — | unprivileged, autostart; filtering HTTP proxy, listens on **:3128** (not privoxy's default 8118); static config in `/etc/systemd/network/eth0.network`, `DNS=192.168.0.254`. Moved from nuc 2026-08-30 |
-| [transmission-bt](transmission-bt.md) | 192.168.0.7 | ✅ | unprivileged, autostart; always-on WireGuard full tunnel → ks4 (egress = 193.70.35.17, kill switch: no default route); `/export/media` disk device (`shift=true`), downloads to `/media/downloads`; web UI :9091 (LAN only). Moved from nuc 2026-08-30 |
+| privoxy | 192.168.0.11 | — | unprivileged, autostart; filtering HTTP proxy, listens on **:3128** (not privoxy's default 8118); static config in `/etc/systemd/network/eth0.network` (`Gateway=192.168.0.1`), `DNS=192.168.0.254`. Moved from nuc 2026-08-30 |
+| [transmission-bt](transmission-bt.md) | 192.168.0.7 | ✅ | unprivileged, autostart; always-on WireGuard full tunnel → ks4 (egress = 193.70.35.17, kill switch: no default route in `main`, wg-quick's `fwmark`/`suppress_prefixlength` rules send traffic to table 51820 — **`netplan apply` wipes those rules, so always `systemctl restart wg-quick@wg0` after it**); **IPv6 disabled** (`/etc/sysctl.d/99-no-ipv6.conf`) since the tunnel is `AllowedIPs = 0.0.0.0/0` only and the FTTH box's native IPv6 RA bypassed the kill switch entirely. Extending the tunnel to `::/0` is **not currently possible**: ks4 has a global v6 address and a default v6 route but **no working v6 egress** (verified 2026-09-16 — both ICMP and TCP to the v6 internet fail while v4 is fine), so it cannot act as a v6 exit. Fix OVH v6 on ks4 first if v6 peers are ever wanted; `/export/media` disk device (`shift=true`), downloads to `/media/downloads`; web UI :9091 (LAN only). Moved from nuc 2026-08-30 |
+
+## Host tunnel
+
+`wg-ks4` — `10.8.0.22/24`, peer = the `wireguard` container on ks4,
+endpoint `193.70.35.17:51845`, `AllowedIPs = 10.8.0.0/24, 192.168.1.1/32`.
+It exists only to reach ks4's incus API at `192.168.1.1:8443` for the
+05:00 pull. Key at `/etc/wireguard/wg-ks4.key`, unit
+`wg-quick@wg-ks4` (enabled).
+
+⚠️ nas also runs a **managed `incusbr0` on `192.168.1.254/24`** — no
+uplink, nothing attached, inert. It exists purely so the ks4 replicas'
+instance-level `eth0` (`parent: incusbr0`, static `192.168.1.x`) passes
+validation on arrival. It must never take `192.168.1.1`: that address has
+to keep resolving over `wg-ks4`, and a local address beats a route.
 
 ## Backup pools hosted here
 
 | incus pool | dataset | receives |
 |---|---|---|
 | `nucbackup` | `tank/backup/nuc` | nuc's instances (pushed nightly, 03:30) |
-| `ks4backup` | `tank/backup/ks4` | ks4's instances (pulled over WG, 05:00 — after FTTH) |
+| `ks4backup` | `tank/backup/ks4` | ks4's instances (pulled over `wg-ks4`, 05:00; built 2026-09-16, see [ks2/nas-seed.md](../ks2/nas-seed.md)) |
 | `nasbackup` | `tank/backup/nas` | **nas's own** instances (local copy, 04:00) |
 
 nas's own instances are replicated **locally** rather than to nuc: nuc is

nas/nas-install.md

@@ -324,11 +324,11 @@ allow-hotplug enp0s20f0
 iface enp0s20f0 inet static
     address 192.168.0.4
     netmask 255.255.255.0
-    gateway 192.168.0.2
+    gateway 192.168.0.1
     dns-nameservers 1.1.1.1 9.9.9.9
 ```
 
-(Gateway is **`192.168.0.2`** — `.1` is gone since 2026-08. The host
+(Gateway is **`192.168.0.1`** — the FTTH box, since 2026-09. The host
 uses public resolvers, never blocky, to avoid a bootstrap loop.
 Interface name is a guess until the board is up — check `ip -br link`.)
 
@@ -416,7 +416,30 @@ instances depend on that host surviving.
 
 Once `tank` has run a week on SATA with **zero** pool suspensions and no
 CRC errors, the `usb4t-dropouts` gate is cleared — record that in
-[ks2/plan.md](../ks2/plan.md).
+[ks2/plan.md](../ks2/plan.md). Day 1 was clean (2026-08-31).
+
+### Clearing the inherited `<metadata>` errors — order matters
+
+The pool imported carrying `<metadata>:<0x0>` and `<metadata>:<0x3d>` from
+the 2026-08-29 USB dropout. A scrub found **0 errors and repaired 0B**,
+yet the entries stayed, and a plain `zpool clear` afterwards did not drop
+them either. ZFS flushes its persistent error log on a scrub that runs
+**after** the clear — so the working order is:
+
+```sh
+zpool clear tank
+zpool scrub tank        # this is the run that flushes the log
+```
+
+Result 2026-08-31: `scrub repaired 0B in 02:34:24 with 0 errors`,
+`errors: No known data errors`, `all pools are healthy`. They were
+artefacts of interrupted writes, not corruption — matching the
+[2026-08-28 incident](../nuc/usb4t-dropouts.md).
+
+⚠️ This matters for monitoring, not just tidiness: while those entries
+stand, `zpool status -x` reports the pool unhealthy permanently, so
+`zpool-health.sh` sits in the alarm state and **cannot signal a new
+problem**. Clear them before trusting the watchdog.
 
 ## 6. Incus
 
@@ -504,7 +527,7 @@ Its WireGuard tunnel is **entirely inside the container** (`wg0`,
 `10.8.0.21`, `wg-quick@wg0`, `BindsTo=` on the daemon), so the container
 carries its own keys and **ks4 needs no change at all** — the peer stays
 `10.8.0.21/32`. The kill-switch `/32` route points at the gateway
-`192.168.0.2`, which is the same from here.
+`192.168.0.1`, which is the same from here.
 
 ```sh
 # on nuc — remote already added in §9a
@@ -592,43 +615,26 @@ The driver is unchanged —
 [`incus-copy.sh`](https://git.lutran.fr/julien/scripts/src/branch/main/incus-copy.sh),
 deployed to `/root/scripts` as everywhere else.
 
-### 9a. nuc's instances -> nas (systemd timer, not cron)
+### 9a. nuc's instances -> nas
 
-⚠️ **nuc is an on-demand media box** (see [nuc/README.md](../nuc/README.md)):
-since 2026-08-30 it only runs when watching Jellyfin or using the Spotify
-kiosk, so it is usually powered off at 03:30. Cron cannot catch up a
-missed window; a **systemd timer with `Persistent=true`** runs the job
-shortly after the next boot. That is the whole reason this is a timer.
+Root crontab on nuc:
 
-`/etc/systemd/system/incus-copy.service` on nuc:
-
-```ini
-[Unit]
-Description=Replicate all nuc instances to nas (project backup, pool nucbackup)
-After=network-online.target incus.service
-Wants=network-online.target
-Requires=incus.service
-
-[Service]
-Type=oneshot
-ExecStart=/bin/sh -c "/root/scripts/incus-copy.sh -d nas -m push -s nucbackup -p backup >> /var/log/incus-copy.log 2>&1"
+```cron
+30 3 * * * /root/scripts/incus-copy.sh -d nas -m push -s nucbackup -p backup >> /var/log/incus-copy.log 2>&1
 ```
 
-`/etc/systemd/system/incus-copy.timer`:
-
-```ini
-[Timer]
-OnCalendar=03:30
-Persistent=true
-RandomizedDelaySec=300
-
-[Install]
-WantedBy=timers.target
-```
-
-```sh
-systemctl enable --now incus-copy.timer && systemctl list-timers incus-copy
-```
+⚠️ **nuc is an on-demand media box** (see
+[nuc/README.md](../nuc/README.md)): since 2026-08-30 it only runs when
+watching Jellyfin or using the Spotify kiosk, so it is often powered
+off at 03:30 and **that night's push is simply skipped** — cron does
+not catch up missed windows. Accepted deliberately (2026-08-31):
+nuc's instances change rarely and the next time it is up the refresh
+is incremental anyway.
+This ran briefly as a `systemd` timer with `Persistent=true` (which
+*does* catch up after boot); the units are still on disk, disabled, at
+`/etc/systemd/system/incus-copy.{service,timer}` if that behaviour is
+ever wanted back:
+`systemctl enable --now incus-copy.timer` (and remove the cron line).
 
 ⚠️ **`-p backup` is not optional.** Without it the replicas land in
 `default` on nas and collide with nas's *live* instances — both hosts are
@@ -651,7 +657,7 @@ incus storage create nasbackup zfs source=tank/backup/nas
 ```
 
 ```cron
-# nas, /etc/cron.d/incus-copy
+# nas, root crontab
 0 4 * * * root /root/scripts/incus-copy.sh -p backup -s nasbackup >> /var/log/incus-copy.log 2>&1
 ```
 
@@ -704,11 +710,20 @@ is now here and carries its own).
   remove `/etc/wireguard/wg-ks4.conf`, and drop the `10.8.0.20/32` peer
   on ks4.
 
-Seed manually in tmux — the first pass is **~1.75 TiB** over the home
-downlink and only makes sense once FTTH is up:
+Built 2026-09-16 — full runbook, corrections and gotchas:
+[ks2/nas-seed.md](../ks2/nas-seed.md). Two things that block the copy if
+missed: nas needs `wireguard-tools` installed and a **managed `incusbr0`
+on `192.168.1.254/24`** (never `.1`), and the seed needs **`-p backup`**.
+
+The pull runs at ~14 MB/s (~112 Mbit/s) and that is **ks4's single
+spinning source disk, not the link or the tunnel** — measured, with the
+numbers, in [ks2/nas-seed.md](../ks2/nas-seed.md) §Bottleneck. Nothing to
+fix on the network side.
 
 ```sh
-/root/scripts/incus-copy.sh -r ks4 -s ks4backup 2>&1 | tee -a /var/log/incus-copy-ks4.log
+systemd-run --unit=ks4-seed --collect \
+  /bin/bash -c '/root/scripts/incus-copy.sh -r ks4 -s ks4backup -p backup \
+                >> /var/log/incus-copy-ks4.log 2>&1'
 ```
 
 Then test-restore one instance before ticking the gate in
@@ -719,10 +734,11 @@ Then test-restore one instance before ticking the gate in
 | When | Host | What |
 |---|---|---|
 | 03:00 | nuc, nas | instance snapshots (profile) |
-| 03:30 | nuc | push all instances → `nas:nucbackup` (**systemd timer**, `Persistent=true` — catches up after boot) |
+| 03:30 | nuc | push all instances → `nas:nucbackup` (root crontab; skipped when nuc is off) |
 | 04:00 | nas | local copy of nas instances → `nasbackup` (`tank/backup/nas`) |
 | 05:00 | nas | pull `ks4:*` → `ks4backup` (after FTTH) |
 | 05:30 | nuc | apt upgrade all containers |
+| 06:00 | nas | apt upgrade all containers (`incus-container-upgrade.sh`, added 2026-09-01; also refreshes `user.os`) |
 | Mon ~00:12 | nas | `zfs-scrub-weekly@tank.timer` / `@incus.timer` (systemd, not cron) |
 
 Staggered around ks4's own 01:00 / 05:00 jobs.
@@ -756,7 +772,7 @@ host had no MTA).
   `ZED_NOTIFY_VERBOSE=1`, **`ZED_NOTIFY_DATA=1`**,
   `ZED_NOTIFY_INTERVAL_SECS=3600`.
 - **[`zpool-health.sh`](https://git.lutran.fr/julien/scripts/src/branch/main/zpool-health.sh)**
-  every 15 min via `/etc/cron.d/zpool-health` — zed does *not* report a
+  every 15 min from root's crontab — zed does *not* report a
   suspended pool (the vdev stays `ONLINE`, so `statechange-notify.sh`
   never fires). That watchdog is the only thing that catches the exact
   failure this box was built to prevent.
@@ -804,11 +820,15 @@ trace can be recut against the meter's series afterwards.
 
 - [x] IPMI **left unplugged by decision** (2026-08-30) — no BMC on the
       LAN, therefore no out-of-band console either
-- [ ] `cat /proc/mdstat` — all three arrays `[UU]`; GRUB written to **both**
-      ESPs (`efibootmgr -v` shows two entries),
-      and boot tested with one disk unplugged
-- [ ] `zpool status` healthy for `incus` and `tank`; weekly scrubs
-      scheduled; both import cleanly after a reboot (`cachefile` set)
+- [x] `cat /proc/mdstat` — all three arrays `[UU]`; GRUB written to
+      **both** ESPs, both mounted, `debian` + `debian-b` boot entries
+      present (verified across a cold boot 2026-08-31)
+- [ ] boot still untested with **one disk physically unplugged** — the
+      mirror is a guess until that is done
+- [x] `zpool status` healthy for `incus` and `tank`; weekly scrubs
+      scheduled; **cold boot verified 2026-08-31** — both pools imported
+      from `/etc/zfs/zpool.cache`, all instances autostarted, NFS exports
+      republished, 0 failed units
 - [ ] **7 days with zero pool suspensions and zero CRC errors** — the
       gate that closes [usb4t-dropouts.md](../nuc/usb4t-dropouts.md)
 - [x] `smartd` monitoring all 3 disks; `zpool-health.sh -t` mail

nas/transmission-bt.md

@@ -66,17 +66,17 @@ network:
         addresses: [192.168.0.254]
       routes:
         - to: 193.70.35.17/32
-          via: 192.168.0.2
+          via: 192.168.0.1
 EOF
 chmod 600 /etc/netplan/10-lxc.yaml
 netplan apply'
 
 # packages need a temporary default route (removed right after)
-incus exec "$CNAME" -- ip route add default via 192.168.0.2
+incus exec "$CNAME" -- ip route add default via 192.168.0.1
 incus exec "$CNAME" -- apt-get update
 incus exec "$CNAME" -- apt-get install -y --no-install-recommends \
     transmission-daemon wireguard-tools iptables curl
-incus exec "$CNAME" -- ip route del default via 192.168.0.2
+incus exec "$CNAME" -- ip route del default via 192.168.0.1
 
 # WireGuard full tunnel (generate key, print pubkey for the ks4 side)
 incus exec "$CNAME" -- bash -c 'umask 077
@@ -199,3 +199,70 @@ incus exec transmission-bt -- bash -c "ping -c1 -W2 8.8.8.8 || echo kill-switch
 - The image server check can make `incus launch` hang on slow WAN —
   launching from the cached image fingerprint (`incus image list`)
   bypasses it.
+
+## Jellyfin library scan on completion (2026-08-31)
+
+Jellyfin cannot notice finished downloads by itself any more. It watches
+libraries with **inotify**, but since the media moved to nas the writer
+(transmission, here) and the reader (`jellyfin-server` on nuc, over NFS)
+are on different machines — an inotify event never crosses that. Before
+the move both shared one local dataset on nuc, so it just worked.
+
+So transmission tells Jellyfin explicitly, via
+`script-torrent-done`:
+
+```json
+"script-torrent-done-enabled": true,
+"script-torrent-done-filename": "/usr/local/bin/jellyfin-scan.sh"
+```
+
+The hook POSTs to Jellyfin's `/Library/Refresh`:
+
+```sh
+#!/bin/sh
+KEY_FILE=/etc/jellyfin-scan.key
+JF=http://192.168.0.5:8096
+NAME="${TR_TORRENT_NAME:-unknown}"
+[ -r "$KEY_FILE" ] || { logger -t jellyfin-scan "no readable key file; skipped ($NAME)"; exit 0; }
+KEY=$(tr -d " \t\r\n" < "$KEY_FILE")
+if curl -fsS -m 15 -X POST -H "X-Emby-Token: $KEY" "$JF/Library/Refresh" >/dev/null 2>&1; then
+    logger -t jellyfin-scan "library scan requested after: $NAME"
+else
+    logger -t jellyfin-scan "library scan request FAILED (nuc off?) after: $NAME"
+fi
+exit 0
+```
+
+Design points, each of which matters:
+
+- **Always `exit 0`, never block.** transmission runs the hook
+  synchronously; a hanging or failing hook stalls the daemon. Both paths
+  are tested — success and unreadable-key both exit 0.
+- **nuc is usually powered off.** The request then fails, logs
+  `FAILED (nuc off?)`, and Jellyfin picks the file up on its next
+  scheduled scan. Not an error worth alerting on.
+- **The key file is `640 root:debian-transmission`** — the hook runs as
+  `debian-transmission`, so it must be group-readable, and nothing wider.
+- Reachable despite the kill switch: `192.168.0.5` is on the directly
+  connected LAN, so it needs no default route.
+
+Verify:
+
+```sh
+incus exec transmission-bt -- su -s /bin/sh debian-transmission \
+    -c 'TR_TORRENT_NAME=selftest /usr/local/bin/jellyfin-scan.sh'
+incus exec transmission-bt -- journalctl -t jellyfin-scan -n 3
+```
+
+**Rotating the key**: create a new one in Jellyfin (Dashboard → API Keys),
+then
+
+```sh
+printf %s '<new-key>' | incus exec transmission-bt -- sh -c \
+  'umask 027; cat > /etc/jellyfin-scan.key; chown root:debian-transmission /etc/jellyfin-scan.key'
+```
+
+⚠️ Edit `settings.json` only while the daemon is **stopped** —
+transmission rewrites the whole file on shutdown and will silently
+discard changes made underneath it.
+

nuc/jellyfin-client.md

@@ -447,6 +447,23 @@ incus exec jellyfin-client -- udevadm info /dev/input/event0    # udev db visibl
   visible (#2); check `run-udev.mount` is active.
 - **Video OK, no sound; JMP log shows `AO: [null]`** → ALSA default broken
   (#3); check `/etc/asound.conf` and the card name in `aplay -l`.
+  **Most common cause: the Pioneer DAC is simply switched off.**
+  `/etc/asound.conf` pins the default to it *by card name* (`Device`), so
+  with the amp off the name does not exist and `default` fails to open —
+  mpv then falls back to null silently: picture, no sound, no error.
+  One-line check before blaming anything else:
+  ```sh
+  incus exec jellyfin-client -- su -s /bin/bash kiosk -c 'aplay -D default -d 1 /usr/share/sounds/alsa/Front_Center.wav'
+  ```
+  `audio open error: No such device` = amp is off. Power it on; no
+  restart needed, JMP opens the device per playback.
+- **Display hotplugged after boot → cage exits once, then recovers by
+  itself.** If the kiosk started with no outputs, plugging the HDMI in
+  makes cage fail (`Failed with result 'exit-code'`); the unit's
+  `Restart=on-failure` / `RestartSec=5` restarts it ~5 s later, this time
+  with the display present. **Do not restart it by hand** — check
+  `systemctl show jellyfin-kiosk -p ActiveEnterTimestamp` first and only
+  intervene if the timestamp predates the hotplug. Verified 2026-08-31.
 - **Keyboard plugged in after boot isn't seen** — the host udev db is live
   through the bind, but udev hotplug *events* don't cross the container's
   network namespace, so cage only enumerates at startup. Re-plugging the

nuc/jellyfin-server.md

@@ -116,6 +116,46 @@ incus start jellyfin-server
 the dataset locally ([nas/transmission-bt.md](../nas/transmission-bt.md)),
 so nuc's mount is read-only and there is exactly one writer.
 
+## ⚠️ Real-time monitoring does not work over NFS (2026-08-31)
+
+Libraries have `EnableRealtimeMonitor=true` and Jellyfin reports
+`SupportsLibraryMonitor: true`, but **new files never appear on their
+own**. Jellyfin watches with inotify, which only reports changes made
+through the local mount; transmission writes them on **nas**, so nuc's
+NFS client sees nothing. Jellyfin looks healthy and silently misses
+everything until a scan.
+
+This is a regression from the 2026-08-30 storage move — before it,
+transmission and jellyfin-server shared one local dataset on nuc and
+inotify fired normally.
+
+**Fix in place:** transmission calls Jellyfin's `/Library/Refresh` when a
+download completes — see
+[nas/transmission-bt.md](../nas/transmission-bt.md). Downloads appear
+within seconds; if nuc is powered off the request fails harmlessly and
+the scheduled scan catches up.
+
+Manual scan (UI): Dashboard → Scheduled Tasks → **Scan Media Library**.
+By API:
+
+```sh
+curl -X POST -H "X-Emby-Token: <key>" http://192.168.0.5:8096/Library/Refresh   # expect 204
+```
+
+Diagnosing "my download is not in Jellyfin", in order — the first three
+were all fine when this was hit, which is what made it confusing:
+
+```sh
+ls /export/media/downloads/                                   # on nas: file there?
+incus exec jellyfin-server -- ls /media/downloads/            # visible through NFS?
+incus exec jellyfin-server -- find /var/lib/jellyfin/root -name '*.mblink' -exec cat {} +   # in a library path?
+incus exec jellyfin-server -- cat /var/lib/jellyfin/data/ScheduledTasks/*.js | grep -o '"Name":"Scan Media Library".*'  # when did it last scan?
+```
+
+⚠️ **nuc's mount is read-only.** Reorganising finished downloads into
+`/media/movies` or `/media/tv-shows` can no longer be done from nuc — do
+it on nas under `/export/media/`.
+
 ## First-run configuration
 
 1. Run the setup wizard; add libraries pointing at `/media/...`.

nuc/nuc-install.md

@@ -12,7 +12,7 @@ How to rebuild the Incus host from scratch if `/dev/sda` (512 GB SSD,
   `usb4t/media` → `/srv/media` (media library)
 - USB: Pioneer USB audio (`08e4:0176`), Logitech Unifying receiver (K400),
   CSCTEK USB Audio and HID
-- NIC: `enp1s0` (static `192.168.0.3/24`, gw `192.168.0.2`)
+- NIC: `enp1s0` (static `192.168.0.3/24`, gw `192.168.0.1`)
 
 ## ⚠️ What dies with sda
 
@@ -31,7 +31,7 @@ incus project create backup -c features.images=false -c features.profiles=false
 /root/scripts/incus-copy.sh -p backup -s nucbackup
 ```
 
-Runs nightly via `/etc/cron.d/incus-copy` at **03:30** (30 min after
+Runs nightly from root's crontab at **03:30** (30 min after
 the profile-scheduled 03:00 instance snapshots, so VM refreshes stay
 incremental), logging to `/var/log/incus-copy.log` (logrotate:
 `/etc/logrotate.d/incus-copy`). Note the script's `flock` is global:
@@ -87,7 +87,7 @@ allow-hotplug enp1s0
 iface enp1s0 inet static
     address 192.168.0.3
     netmask 255.255.255.0
-    gateway 192.168.0.2
+    gateway 192.168.0.1
     dns-nameservers 1.1.1.1 9.9.9.9
 ```
 
@@ -165,13 +165,56 @@ hosts reach them normally. (So test a container's LAN service from inside
 the container or from an external LAN host — never by pinging its IP from
 the nuc or a sibling container; that always fails by design.)
 
-LAN gateway note: the router/gateway is **`192.168.0.2`** (migrated from
-`192.168.0.1`, 2026-08 — `.1` is gone). DHCP-configured instances pick the
-new gateway up automatically; **statically-configured ones must be updated
-by hand.** Current static holdouts: privoxy
-(`/etc/systemd/network/eth0.network`, `Gateway=`) and transmission-bt
-(netplan `routes: via:` + the WG kill-switch `/32`). Symptom of a missed
-one: the service is up and its port answers, but nothing it fetches works.
+LAN gateway note: the router/gateway is **`192.168.0.1`** — the FTTH box,
+since 2026-09 (it was `.2`, the Archer C7, from 2026-08; and `.1` before
+that). **Every host and instance is statically configured, so each one
+must be updated by hand.** Symptom of a missed one: the service is up and
+its port answers, but nothing it fetches works.
+
+Every LAN host and instance is now **statically configured** (verified
+2026-09-16) — nothing on this LAN depends on a DHCP reservation any more.
+On a gateway change, update all of these by hand:
+
+| Where | File | Address |
+|---|---|---|
+| nas host | `/etc/network/interfaces`, `gateway` | `.4` |
+| nuc host | `/etc/network/interfaces`, `gateway` | `.3` |
+| blocky | `/etc/systemd/network/eth0.network`, `Gateway=` | `.254` |
+| privoxy | `/etc/systemd/network/eth0.network`, `Gateway=` | `.11` |
+| transmission-bt | netplan `routes: via:` (WG kill-switch `/32`) | `.7` |
+| jellyfin-server | netplan `routes: - to: default / via:` | `.5` |
+| jellyfin-client | `/etc/systemd/network/10-eth0.network`, `Gateway=` | `.6` |
+
+`homeassistant` (a HAOS **VM**, NetworkManager, normally stopped) is
+deliberately left on DHCP — it never had a reservation and nothing
+addresses it by IP.
+
+⚠️ **Why everything is static now: DHCP reservations did not survive the
+FTTH migration.** They lived in the Archer C7's `dhcp.@host[-1]` list
+(the `add_host` block in
+[../archer-c7/upgrade-openwrt-25.12.md](../archer-c7/upgrade-openwrt-25.12.md)),
+and the FTTH box did not inherit them. blocky held `192.168.0.254` that
+way; renewing its lease handed it a pool address and took LAN DNS down
+with it. Static config removes the dependency entirely.
+
+Two **non-container** hosts also lost their reservations and are still
+dynamic — harmless, nothing addresses them by IP, but the old fixed
+addresses are gone: `LAPTOP719974` (was `.20`) and `patate` (was `.21`).
+
+⚠️ **`jellyfin-client` cannot use netplan at all.** It is a privileged
+kiosk whose `raw.lxc` bind-mounts the host's `/run/udev` read-only, so
+`netplan generate` dies with `cannot create directory /run/udev/rules.d`
+— which means netplan changes there **silently fail to regenerate at
+boot**. It is configured with plain systemd-networkd
+(`/etc/systemd/network/10-eth0.network`); its old netplan yaml is parked
+at `/root/10-lxc.yaml.netplan-disabled-ftth`. Always verify a network
+change in that container with `incus restart jellyfin-client`, not just
+`netplan apply`.
+
+**Fallback hardware:** the Archer C7 and the LTE box are kept on the
+shelf. Their addressing does not clash with the current LAN — **the
+gateway is the only thing that differs**, so failing back means walking
+the table above and setting `.2` (C7) instead of `.1`.
 
 Let `julien` run harmless incus commands (list/info/config/show…)
 without a password — mutating ones (`exec`, `start/stop`, `delete`)
@@ -245,8 +288,8 @@ once `usb4t` is imported.)
 | When  | What | Where |
 |-------|------|-------|
 | 03:00 | instance snapshots (`snapshots.schedule` on the default profile, expiry 7d) | incus |
-| 03:30 | replicate all instances to the USB pool (`incus-copy.sh -p backup -s nucbackup`) | `/etc/cron.d/incus-copy` → `/var/log/incus-copy.log` |
-| 05:00 | apt dist-upgrade all running containers (`incus-container-upgrade.sh`; VMs and non-apt containers skipped; jellyfin pinned to the 10.11 series in-container) | `/etc/cron.d/incus-container-upgrade` → `/var/log/incus-container-upgrade.log` |
+| 03:30 | replicate all instances to the USB pool (`incus-copy.sh -p backup -s nucbackup`) | root crontab → `/var/log/incus-copy.log` |
+| 05:00 | apt dist-upgrade all running containers (`incus-container-upgrade.sh`; VMs and non-apt containers skipped; jellyfin pinned to the 10.11 series in-container) | root crontab → `/var/log/incus-container-upgrade.log` |
 
 The ordering is deliberate: snapshot → backup → upgrade, so a broken
 upgrade is always one snapshot-restore away and the replicas predate it.

nuc/usb4t-dropouts.md

@@ -1,7 +1,16 @@
-# usb4t: USB dropouts suspend the pool (2026-08)
+# usb4t: USB dropouts suspend the pool (2026-08) — RESOLVED
+
+> **Outcome (2026-08-30): the disk moved off USB entirely.** It now
+> runs on **direct SATA** in the new host `nas`
+> ([nas/README.md](../nas/README.md)), as pool `tank`, and the ks4
+> pull leg plus its WireGuard tunnel moved with it
+> ([ks2/nas-seed.md](../ks2/nas-seed.md)). Everything below is the
+> investigation that led there — worth keeping for the diagnosis
+> method and for the alerting gap it exposed, which applies to any
+> host.
 
 Symptom seen first in the nightly backup log
-(`/var/log/incus-copy.log`, job in `/etc/cron.d/incus-copy`): every
+(`/var/log/incus-copy.log`, job in root's crontab): every
 instance fails with
 
 ```
@@ -131,7 +140,7 @@ before). Two reasons:
 
 Fixed 2026-08-30 by adding
 [`zpool-health.sh`](https://git.lutran.fr/julien/scripts/src/branch/main/zpool-health.sh),
-run every 15 min from `/etc/cron.d/zpool-health`. It mails only on
+run every 15 min from root's crontab. It mails only on
 `healthy <-> problem` **transitions**, so it is silent in normal
 operation and cannot spam; `-t` sends a test. Worth deploying on ks4
 too — its `data` pool is single-disk and has the same blind spot.