doc: record the real ks4 pull bottleneck — source disk, not the link

c8d759c claimed the ~125 Mbit/s seed rate was ks4's OVH uplink. That was
wrong, and measuring it says so: ks4 uploads at 609 Mbit/s, downloads at
900, and nas pulls 670 from OVH's network. WireGuard is not it either —
zero UdpRcvbufErrors, wg-crypt kworkers at ~2%, both hosts ~85% idle.

The limit is `data` sitting on sdb5, one 7200 rpm HGST 6 TB disk: during
the send it does 109 r/s at 14 MB/s with ~131 KB requests and a queue
depth of ~1.0, which is the random-IOPS ceiling of a single HDD reading a
fragmented 1.42 TiB dataset. 14 MB/s is ~112 Mbit/s on the wire, exactly
what we see, while the network sits idle.

Parallelism is the only lever and it works by disk queue depth, not
bandwidth: a second concurrent copy takes sdb from 14 to 21 MB/s and the
request size from 131 KB to 514 KB, lifting the tunnel from 113 to 153
Mbit/s. Recorded, along with the decision to keep the seed sequential —
one-off pass, incremental refreshes after, and incus-copy.sh is shared by
all three legs.

Also corrects the set size to the actual 1.42 TiB (~29 h, not ~31 h).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

ks2/nas-seed.md

@@ -1,6 +1,9 @@
 # ks4 pull leg — seed after FTTH
 
-Status: **prepared, waiting on the FTTH link.**
+Status: **built and seeding — 2026-09-16.** FTTH is up (gateway
+`192.168.0.1`, see [nuc/nuc-install.md](../nuc/nuc-install.md)); the
+tunnel, the incus remote and the cron are in place and the first pass is
+running. Remaining: let the seed finish, then the test-restore below.
 
 ⚠️ **Changed 2026-08-30: this leg lands on `nas`, not on nuc.** It was
 originally designed for nuc's USB pool `usb4t`, but that pool proved to
@@ -33,20 +36,28 @@ Consequences versus the original plan:
 
 ## Prerequisites
 
-- [ ] `tank` running on SATA for ≥ 7 days with zero pool suspensions —
+- [x] `tank` running on SATA for ≥ 7 days with zero pool suspensions —
-      the gate that replaces "fix the USB enclosure"
+      the gate that replaces "fix the USB enclosure".
-- [ ] FTTH up (the first pass moves ~1.75 TiB)
+      Verified 2026-09-16: `ONLINE`, scrub clean 2026-09-14, uptime 2 w 2 d,
+      zero suspensions in the journal.
+- [x] FTTH up (the first pass moves ~1.75 TiB)
 
 ## Setup (root on nas)
 
 ```sh
-# 1. peer nas on ks4's wireguard container
+# 0. the nas host has no wireguard-tools — transmission-bt carries its own
-#    (run on ks4) — <nas-pubkey> from /etc/wireguard/wg-ks4.key on nas
+#    tunnel *inside* the container, so the host never needed them
+apt-get install -y wireguard-tools
+
+# 1. key on nas, then peer it on ks4's wireguard container
+umask 077; wg genkey > /etc/wireguard/wg-ks4.key
+wg pubkey < /etc/wireguard/wg-ks4.key          # -> <nas-pubkey>
+#    (run on ks4)
 incus exec wireguard -- wg set wg0 peer <nas-pubkey> allowed-ips 10.8.0.22/32
 incus exec wireguard -- wg-quick save wg0
 
 # 2. tunnel on nas: /etc/wireguard/wg-ks4.conf, modelled on nuc's
-#    Address = 10.8.0.22/32, peer pubkey TVs6d7…,
+#    Address = 10.8.0.22/24, peer pubkey TVs6d7…,
 #    Endpoint = 193.70.35.17:51845,
 #    AllowedIPs = 10.8.0.0/24, 192.168.1.1/32, keepalive 25
 systemctl enable --now wg-quick@wg-ks4
@@ -56,14 +67,100 @@ incus remote add ks4 https://192.168.1.1:8443 --accept-certificate --token '…'
 incus list ks4: | head          # sanity: remote reachable
 ```
 
-## Seed
+⚠️ **nas needs a managed `incusbr0` of its own, or every copy fails
+instantly.** All 18 ks4 instances carry an *instance-level* `eth0`
+(`nictype: bridged`, `parent: incusbr0`, `ipv4.address: 192.168.1.x`).
+nas has no such bridge, so instance creation dies with:
+
+```
+Device validation failed for "eth0": Cannot use manually specified
+ipv4.address when using unmanaged parent bridge
+```
+
+Create a managed network of the same name — but **give it `.254`, never
+`.1`**: `192.168.1.1` must keep resolving over `wg-ks4` to ks4's incus
+API, and a local address always beats a route.
 
 ```sh
-# full pull of every ks4 instance into pool ks4backup (tmux — first pass
+incus network create incusbr0 \
-# moves ~1.75 TiB through the WG tunnel)
+    ipv4.address=192.168.1.254/24 ipv4.nat=false ipv6.address=none
-/root/scripts/incus-copy.sh -r ks4 -s ks4backup 2>&1 | tee -a /var/log/incus-copy-ks4.log
+ip route get 192.168.1.1        # must still say: dev wg-ks4
 ```
 
+The bridge stays inert — replicas are never started here.
+
+## Seed
+
+⚠️ **`-p backup` is required**, exactly as for the nuc leg
+([nas/nas-install.md](../nas/nas-install.md) §9a) — and as the
+verification command below already assumed. Without it the 18 replicas
+land in `default` alongside nas's live instances.
+
+Run it detached rather than in a shell that can drop — the first pass is
+long: **~1.42 TiB at ~14 MB/s ≈ 29 h** (see the bottleneck section below
+for why it is 14 MB/s and not more).
+
+```sh
+systemd-run --unit=ks4-seed --collect \
+  /bin/bash -c '/root/scripts/incus-copy.sh -r ks4 -s ks4backup -p backup \
+                >> /var/log/incus-copy-ks4.log 2>&1'
+
+systemctl is-active ks4-seed          # progress:
+tail -f /var/log/incus-copy-ks4.log
+```
+
+⚠️ **The seed suppresses the 04:00 `nasbackup` job while it runs.**
+`incus-copy.sh` takes a single `/run/lock/incus-copy.lock` for every
+shape, so any run starting while the seed holds it aborts with
+`another incus-copy run holds …`. Over a ~29 h seed that skips one or two
+nights of the nas-local copy — accepted; those replicas are small,
+same-host, and rebuildable. The 03:30 nuc push is unaffected (it runs on
+nuc, with nuc's own lock).
+
+### Bottleneck: ks4's source disk, not the network
+
+Measured 2026-09-16, because ~112 Mbit/s looked far too slow for a 2 Gbit/s
+FTTH line. **It is not the link, and not WireGuard.** Do not go looking for
+a network fix:
+
+| Path | Measured |
+|---|---|
+| ks4 upload → internet | 609 Mbit/s |
+| ks4 download ← internet | 900 Mbit/s |
+| nas download ← OVH network | 670 Mbit/s |
+| ks4 → nas through `wg-ks4` | **~112 Mbit/s** |
+
+WireGuard was ruled out too: `UdpRcvbufErrors` 0 on nas (so the default
+`net.core.rmem_max` of 208 KB is *not* dropping packets), wg-crypt
+kworkers ~2 %, both hosts ~85 % idle.
+
+The limit is **`data` living on `sdb5`, a single 7200 rpm HGST 6 TB
+spinning disk**. During the send `iostat` showed sdb at **109 r/s /
+14 MB/s, ~131 KB average request, queue depth ~1.0** — the random-IOPS
+ceiling of one HDD walking a fragmented 1.42 TiB dataset. 14 MB/s is
+~112 Mbit/s on the wire, which is exactly the observed rate. The network
+is idle the whole time.
+
+**Parallelism is the only real lever, and it is a disk-queue effect**, not
+a bandwidth one. Running a second instance copy alongside the seed:
+
+| | sequential | + 1 parallel copy |
+|---|---|---|
+| sdb read | 14 MB/s | **21 MB/s** |
+| avg request size | 131 KB | **514 KB** |
+| tunnel | 113 Mbit/s | **153 Mbit/s** |
+
+With two senders queued, ZFS issues larger, more sequential reads instead
+of seeking one request at a time. Three or four concurrent copies would
+plausibly reach 25–30 MB/s and roughly halve the seed.
+
+**Decision 2026-09-16: keep it sequential.** The first pass is a one-off,
+later refreshes are ZFS-incremental and tiny, and `incus-copy.sh` is
+shared by all three legs — parallelising means either reworking the script
+or running copies outside its flock, i.e. two jobs contending for the same
+dataset. Not worth one overnight. If a full reseed is ever needed and the
+wall-clock matters, this is the knob; the disk is the floor either way.
+
 Notes:
 
 - First pass is a full send per instance; later refreshes are
@@ -79,9 +176,12 @@ Add to nas's root crontab, offset from the 03:30 nuc→nas push, the
 04:00 nas→nuc push and ks4's own 01:00/05:00 jobs:
 
 ```cron
-0 5 * * * /root/scripts/incus-copy.sh -r ks4 -s ks4backup >> /var/log/incus-copy-ks4.log 2>&1
+0 5 * * * /root/scripts/incus-copy.sh -r ks4 -s ks4backup -p backup >> /var/log/incus-copy-ks4.log 2>&1
 ```
 
+`/etc/logrotate.d/incus-copy` already lists `incus-copy-ks4.log`, so
+nothing to add there.
+
 ## Verification (release gate for ks2)
 
 ```sh

nas/README.md

@@ -33,15 +33,29 @@ Storage + backup host on the LAN, added 2026-08.
 | Name | IP | Doc | Features |
 |---|---|---|---|
 | blocky | 192.168.0.254 | — | unprivileged, autostart; DNS ad-blocker for the LAN. Moved from nuc 2026-08-30 so it survives nuc being powered off |
-| privoxy | 192.168.0.11 | — | unprivileged, autostart; filtering HTTP proxy, listens on **:3128** (not privoxy's default 8118); static config in `/etc/systemd/network/eth0.network`, `DNS=192.168.0.254`. Moved from nuc 2026-08-30 |
+| privoxy | 192.168.0.11 | — | unprivileged, autostart; filtering HTTP proxy, listens on **:3128** (not privoxy's default 8118); static config in `/etc/systemd/network/eth0.network` (`Gateway=192.168.0.1`), `DNS=192.168.0.254`. Moved from nuc 2026-08-30 |
-| [transmission-bt](transmission-bt.md) | 192.168.0.7 | ✅ | unprivileged, autostart; always-on WireGuard full tunnel → ks4 (egress = 193.70.35.17, kill switch: no default route); `/export/media` disk device (`shift=true`), downloads to `/media/downloads`; web UI :9091 (LAN only). Moved from nuc 2026-08-30 |
+| [transmission-bt](transmission-bt.md) | 192.168.0.7 | ✅ | unprivileged, autostart; always-on WireGuard full tunnel → ks4 (egress = 193.70.35.17, kill switch: no default route in `main`, wg-quick's `fwmark`/`suppress_prefixlength` rules send traffic to table 51820 — **`netplan apply` wipes those rules, so always `systemctl restart wg-quick@wg0` after it**); **IPv6 disabled** (`/etc/sysctl.d/99-no-ipv6.conf`) since the tunnel is `AllowedIPs = 0.0.0.0/0` only and the FTTH box's native IPv6 RA bypassed the kill switch entirely. Extending the tunnel to `::/0` is **not currently possible**: ks4 has a global v6 address and a default v6 route but **no working v6 egress** (verified 2026-09-16 — both ICMP and TCP to the v6 internet fail while v4 is fine), so it cannot act as a v6 exit. Fix OVH v6 on ks4 first if v6 peers are ever wanted; `/export/media` disk device (`shift=true`), downloads to `/media/downloads`; web UI :9091 (LAN only). Moved from nuc 2026-08-30 |
+
+## Host tunnel
+
+`wg-ks4` — `10.8.0.22/24`, peer = the `wireguard` container on ks4,
+endpoint `193.70.35.17:51845`, `AllowedIPs = 10.8.0.0/24, 192.168.1.1/32`.
+It exists only to reach ks4's incus API at `192.168.1.1:8443` for the
+05:00 pull. Key at `/etc/wireguard/wg-ks4.key`, unit
+`wg-quick@wg-ks4` (enabled).
+
+⚠️ nas also runs a **managed `incusbr0` on `192.168.1.254/24`** — no
+uplink, nothing attached, inert. It exists purely so the ks4 replicas'
+instance-level `eth0` (`parent: incusbr0`, static `192.168.1.x`) passes
+validation on arrival. It must never take `192.168.1.1`: that address has
+to keep resolving over `wg-ks4`, and a local address beats a route.
 
 ## Backup pools hosted here
 
 | incus pool | dataset | receives |
 |---|---|---|
 | `nucbackup` | `tank/backup/nuc` | nuc's instances (pushed nightly, 03:30) |
-| `ks4backup` | `tank/backup/ks4` | ks4's instances (pulled over WG, 05:00 — after FTTH) |
+| `ks4backup` | `tank/backup/ks4` | ks4's instances (pulled over `wg-ks4`, 05:00; built 2026-09-16, see [ks2/nas-seed.md](../ks2/nas-seed.md)) |
 | `nasbackup` | `tank/backup/nas` | **nas's own** instances (local copy, 04:00) |
 
 nas's own instances are replicated **locally** rather than to nuc: nuc is

nas/nas-install.md

@@ -324,11 +324,11 @@ allow-hotplug enp0s20f0
 iface enp0s20f0 inet static
     address 192.168.0.4
     netmask 255.255.255.0
-    gateway 192.168.0.2
+    gateway 192.168.0.1
     dns-nameservers 1.1.1.1 9.9.9.9
 ```
 
-(Gateway is **`192.168.0.2`** — `.1` is gone since 2026-08. The host
+(Gateway is **`192.168.0.1`** — the FTTH box, since 2026-09. The host
 uses public resolvers, never blocky, to avoid a bootstrap loop.
 Interface name is a guess until the board is up — check `ip -br link`.)
 
@@ -527,7 +527,7 @@ Its WireGuard tunnel is **entirely inside the container** (`wg0`,
 `10.8.0.21`, `wg-quick@wg0`, `BindsTo=` on the daemon), so the container
 carries its own keys and **ks4 needs no change at all** — the peer stays
 `10.8.0.21/32`. The kill-switch `/32` route points at the gateway
-`192.168.0.2`, which is the same from here.
+`192.168.0.1`, which is the same from here.
 
 ```sh
 # on nuc — remote already added in §9a
@@ -710,11 +710,20 @@ is now here and carries its own).
   remove `/etc/wireguard/wg-ks4.conf`, and drop the `10.8.0.20/32` peer
   on ks4.
 
-Seed manually in tmux — the first pass is **~1.75 TiB** over the home
+Built 2026-09-16 — full runbook, corrections and gotchas:
-downlink and only makes sense once FTTH is up:
+[ks2/nas-seed.md](../ks2/nas-seed.md). Two things that block the copy if
+missed: nas needs `wireguard-tools` installed and a **managed `incusbr0`
+on `192.168.1.254/24`** (never `.1`), and the seed needs **`-p backup`**.
+
+The pull runs at ~14 MB/s (~112 Mbit/s) and that is **ks4's single
+spinning source disk, not the link or the tunnel** — measured, with the
+numbers, in [ks2/nas-seed.md](../ks2/nas-seed.md) §Bottleneck. Nothing to
+fix on the network side.
 
 ```sh
-/root/scripts/incus-copy.sh -r ks4 -s ks4backup 2>&1 | tee -a /var/log/incus-copy-ks4.log
+systemd-run --unit=ks4-seed --collect \
+  /bin/bash -c '/root/scripts/incus-copy.sh -r ks4 -s ks4backup -p backup \
+                >> /var/log/incus-copy-ks4.log 2>&1'
 ```
 
 Then test-restore one instance before ticking the gate in

nas/transmission-bt.md

@@ -66,17 +66,17 @@ network:
         addresses: [192.168.0.254]
       routes:
         - to: 193.70.35.17/32
-          via: 192.168.0.2
+          via: 192.168.0.1
 EOF
 chmod 600 /etc/netplan/10-lxc.yaml
 netplan apply'
 
 # packages need a temporary default route (removed right after)
-incus exec "$CNAME" -- ip route add default via 192.168.0.2
+incus exec "$CNAME" -- ip route add default via 192.168.0.1
 incus exec "$CNAME" -- apt-get update
 incus exec "$CNAME" -- apt-get install -y --no-install-recommends \
     transmission-daemon wireguard-tools iptables curl
-incus exec "$CNAME" -- ip route del default via 192.168.0.2
+incus exec "$CNAME" -- ip route del default via 192.168.0.1
 
 # WireGuard full tunnel (generate key, print pubkey for the ks4 side)
 incus exec "$CNAME" -- bash -c 'umask 077

nuc/nuc-install.md

@@ -12,7 +12,7 @@ How to rebuild the Incus host from scratch if `/dev/sda` (512 GB SSD,
   `usb4t/media` → `/srv/media` (media library)
 - USB: Pioneer USB audio (`08e4:0176`), Logitech Unifying receiver (K400),
   CSCTEK USB Audio and HID
-- NIC: `enp1s0` (static `192.168.0.3/24`, gw `192.168.0.2`)
+- NIC: `enp1s0` (static `192.168.0.3/24`, gw `192.168.0.1`)
 
 ## ⚠️ What dies with sda
 
@@ -87,7 +87,7 @@ allow-hotplug enp1s0
 iface enp1s0 inet static
     address 192.168.0.3
     netmask 255.255.255.0
-    gateway 192.168.0.2
+    gateway 192.168.0.1
     dns-nameservers 1.1.1.1 9.9.9.9
 ```
 
@@ -165,13 +165,56 @@ hosts reach them normally. (So test a container's LAN service from inside
 the container or from an external LAN host — never by pinging its IP from
 the nuc or a sibling container; that always fails by design.)
 
-LAN gateway note: the router/gateway is **`192.168.0.2`** (migrated from
+LAN gateway note: the router/gateway is **`192.168.0.1`** — the FTTH box,
-`192.168.0.1`, 2026-08 — `.1` is gone). DHCP-configured instances pick the
+since 2026-09 (it was `.2`, the Archer C7, from 2026-08; and `.1` before
-new gateway up automatically; **statically-configured ones must be updated
+that). **Every host and instance is statically configured, so each one
-by hand.** Current static holdouts: privoxy
+must be updated by hand.** Symptom of a missed one: the service is up and
-(`/etc/systemd/network/eth0.network`, `Gateway=`) and transmission-bt
+its port answers, but nothing it fetches works.
-(netplan `routes: via:` + the WG kill-switch `/32`). Symptom of a missed
+
-one: the service is up and its port answers, but nothing it fetches works.
+Every LAN host and instance is now **statically configured** (verified
+2026-09-16) — nothing on this LAN depends on a DHCP reservation any more.
+On a gateway change, update all of these by hand:
+
+| Where | File | Address |
+|---|---|---|
+| nas host | `/etc/network/interfaces`, `gateway` | `.4` |
+| nuc host | `/etc/network/interfaces`, `gateway` | `.3` |
+| blocky | `/etc/systemd/network/eth0.network`, `Gateway=` | `.254` |
+| privoxy | `/etc/systemd/network/eth0.network`, `Gateway=` | `.11` |
+| transmission-bt | netplan `routes: via:` (WG kill-switch `/32`) | `.7` |
+| jellyfin-server | netplan `routes: - to: default / via:` | `.5` |
+| jellyfin-client | `/etc/systemd/network/10-eth0.network`, `Gateway=` | `.6` |
+
+`homeassistant` (a HAOS **VM**, NetworkManager, normally stopped) is
+deliberately left on DHCP — it never had a reservation and nothing
+addresses it by IP.
+
+⚠️ **Why everything is static now: DHCP reservations did not survive the
+FTTH migration.** They lived in the Archer C7's `dhcp.@host[-1]` list
+(the `add_host` block in
+[../archer-c7/upgrade-openwrt-25.12.md](../archer-c7/upgrade-openwrt-25.12.md)),
+and the FTTH box did not inherit them. blocky held `192.168.0.254` that
+way; renewing its lease handed it a pool address and took LAN DNS down
+with it. Static config removes the dependency entirely.
+
+Two **non-container** hosts also lost their reservations and are still
+dynamic — harmless, nothing addresses them by IP, but the old fixed
+addresses are gone: `LAPTOP719974` (was `.20`) and `patate` (was `.21`).
+
+⚠️ **`jellyfin-client` cannot use netplan at all.** It is a privileged
+kiosk whose `raw.lxc` bind-mounts the host's `/run/udev` read-only, so
+`netplan generate` dies with `cannot create directory /run/udev/rules.d`
+— which means netplan changes there **silently fail to regenerate at
+boot**. It is configured with plain systemd-networkd
+(`/etc/systemd/network/10-eth0.network`); its old netplan yaml is parked
+at `/root/10-lxc.yaml.netplan-disabled-ftth`. Always verify a network
+change in that container with `incus restart jellyfin-client`, not just
+`netplan apply`.
+
+**Fallback hardware:** the Archer C7 and the LTE box are kept on the
+shelf. Their addressing does not clash with the current LAN — **the
+gateway is the only thing that differs**, so failing back means walking
+the table above and setting `.2` (C7) instead of `.1`.
 
 Let `julien` run harmless incus commands (list/info/config/show…)
 without a password — mutating ones (`exec`, `start/stop`, `delete`)