# ks4 pull leg — seed after FTTH Status: **built and seeding — 2026-09-16.** FTTH is up (gateway `192.168.0.1`, see [nuc/nuc-install.md](../nuc/nuc-install.md)); the tunnel, the incus remote and the cron are in place and the first pass is running. Remaining: let the seed finish, then the test-restore below. ⚠️ **Changed 2026-08-30: this leg lands on `nas`, not on nuc.** It was originally designed for nuc's USB pool `usb4t`, but that pool proved to be the least reliable device in the setup ([nuc/usb4t-dropouts.md](../nuc/usb4t-dropouts.md)) — which is exactly what an off-site copy of ks4 must not be. The 4 TB disk moved to direct SATA on the new host `nas` (`192.168.0.4`, [nas/nas-install.md](../nas/nas-install.md)), and the target pool `ks4backup` moved with it. Consequences versus the original plan: - Target pool `ks4backup` is now backed by `tank/backup/ks4` on nas. - The **WireGuard tunnel moves too**: nas becomes peer `10.8.0.22`. `transmission-bt`, the only other user, moved to nas and carries its own in-container tunnel (`10.8.0.21`, unchanged — ks4 needs no edit for it). - ⚠️ **nuc's `wg-ks4` was already disabled on 2026-08-31**, *before* the seed, not after. The original plan retired it only once nas was seeded, on the assumption nuc could serve as a fallback target — it cannot: its `ks4backup` pool was deleted and its `data` pool is a 512 GB SSD, far too small for the ~1.75 TiB replica set. Keeping a keepalive'd tunnel alive on a machine that is now powered off between uses bought nothing. `wg-quick@wg-ks4` is `disabled`, and the `ks4` incus remote was removed from nuc. `/etc/wireguard/wg-ks4.conf` and its key are **kept**, so it is one `systemctl enable --now wg-quick@wg-ks4` away if ever needed. - ks4's ufw rule is unchanged: traffic arrives masqueraded as the `wireguard` container (`192.168.1.18`) whichever peer sent it. ## Prerequisites - [x] `tank` running on SATA for ≥ 7 days with zero pool suspensions — the gate that replaces "fix the USB enclosure". Verified 2026-09-16: `ONLINE`, scrub clean 2026-09-14, uptime 2 w 2 d, zero suspensions in the journal. - [x] FTTH up (the first pass moves ~1.75 TiB) ## Setup (root on nas) ```sh # 0. the nas host has no wireguard-tools — transmission-bt carries its own # tunnel *inside* the container, so the host never needed them apt-get install -y wireguard-tools # 1. key on nas, then peer it on ks4's wireguard container umask 077; wg genkey > /etc/wireguard/wg-ks4.key wg pubkey < /etc/wireguard/wg-ks4.key # -> # (run on ks4) incus exec wireguard -- wg set wg0 peer allowed-ips 10.8.0.22/32 incus exec wireguard -- wg-quick save wg0 # 2. tunnel on nas: /etc/wireguard/wg-ks4.conf, modelled on nuc's # Address = 10.8.0.22/24, peer pubkey TVs6d7…, # Endpoint = 193.70.35.17:51845, # AllowedIPs = 10.8.0.0/24, 192.168.1.1/32, keepalive 25 systemctl enable --now wg-quick@wg-ks4 # 3. incus remote over the tunnel incus remote add ks4 https://192.168.1.1:8443 --accept-certificate --token '…' incus list ks4: | head # sanity: remote reachable ``` ⚠️ **nas needs a managed `incusbr0` of its own, or every copy fails instantly.** All 18 ks4 instances carry an *instance-level* `eth0` (`nictype: bridged`, `parent: incusbr0`, `ipv4.address: 192.168.1.x`). nas has no such bridge, so instance creation dies with: ``` Device validation failed for "eth0": Cannot use manually specified ipv4.address when using unmanaged parent bridge ``` Create a managed network of the same name — but **give it `.254`, never `.1`**: `192.168.1.1` must keep resolving over `wg-ks4` to ks4's incus API, and a local address always beats a route. ```sh incus network create incusbr0 \ ipv4.address=192.168.1.254/24 ipv4.nat=false ipv6.address=none ip route get 192.168.1.1 # must still say: dev wg-ks4 ``` The bridge stays inert — replicas are never started here. ## Seed ⚠️ **`-p backup` is required**, exactly as for the nuc leg ([nas/nas-install.md](../nas/nas-install.md) §9a) — and as the verification command below already assumed. Without it the 18 replicas land in `default` alongside nas's live instances. Run it detached rather than in a shell that can drop — the first pass is long: **~1.42 TiB at ~14 MB/s ≈ 29 h** (see the bottleneck section below for why it is 14 MB/s and not more). ```sh systemd-run --unit=ks4-seed --collect \ /bin/bash -c '/root/scripts/incus-copy.sh -r ks4 -s ks4backup -p backup \ >> /var/log/incus-copy-ks4.log 2>&1' systemctl is-active ks4-seed # progress: tail -f /var/log/incus-copy-ks4.log ``` ⚠️ **The seed suppresses the 04:00 `nasbackup` job while it runs.** `incus-copy.sh` takes a single `/run/lock/incus-copy.lock` for every shape, so any run starting while the seed holds it aborts with `another incus-copy run holds …`. Over a ~29 h seed that skips one or two nights of the nas-local copy — accepted; those replicas are small, same-host, and rebuildable. The 03:30 nuc push is unaffected (it runs on nuc, with nuc's own lock). ### Bottleneck: ks4's source disk, not the network Measured 2026-09-16, because ~112 Mbit/s looked far too slow for a 2 Gbit/s FTTH line. **It is not the link, and not WireGuard.** Do not go looking for a network fix: | Path | Measured | |---|---| | ks4 upload → internet | 609 Mbit/s | | ks4 download ← internet | 900 Mbit/s | | nas download ← OVH network | 670 Mbit/s | | ks4 → nas through `wg-ks4` | **~112 Mbit/s** | WireGuard was ruled out too: `UdpRcvbufErrors` 0 on nas (so the default `net.core.rmem_max` of 208 KB is *not* dropping packets), wg-crypt kworkers ~2 %, both hosts ~85 % idle. The limit is **`data` living on `sdb5`, a single 7200 rpm HGST 6 TB spinning disk**. During the send `iostat` showed sdb at **109 r/s / 14 MB/s, ~131 KB average request, queue depth ~1.0** — the random-IOPS ceiling of one HDD walking a fragmented 1.42 TiB dataset. 14 MB/s is ~112 Mbit/s on the wire, which is exactly the observed rate. The network is idle the whole time. **Parallelism is the only real lever, and it is a disk-queue effect**, not a bandwidth one. Running a second instance copy alongside the seed: | | sequential | + 1 parallel copy | |---|---|---| | sdb read | 14 MB/s | **21 MB/s** | | avg request size | 131 KB | **514 KB** | | tunnel | 113 Mbit/s | **153 Mbit/s** | With two senders queued, ZFS issues larger, more sequential reads instead of seeking one request at a time. Three or four concurrent copies would plausibly reach 25–30 MB/s and roughly halve the seed. **Decision 2026-09-16: keep it sequential.** The first pass is a one-off, later refreshes are ZFS-incremental and tiny, and `incus-copy.sh` is shared by all three legs — parallelising means either reworking the script or running copies outside its flock, i.e. two jobs contending for the same dataset. Not worth one overnight. If a full reseed is ever needed and the wall-clock matters, this is the knob; the disk is the floor either way. Notes: - First pass is a full send per instance; later refreshes are ZFS-incremental **as long as they run at least every `snapshots.expiry` (7 d on ks4)** — same caveat as [ks4's local leg](../ks4/local-backup-cron.md). - Replicas arrive stopped with `boot.autostart=false` (the script does this) — they must never come up on the LAN with ks4's proxy devices. ## Cron (after the seed) Add to nas's root crontab, offset from the 03:30 nuc→nas push, the 04:00 nas→nuc push and ks4's own 01:00/05:00 jobs: ```cron 0 5 * * * /root/scripts/incus-copy.sh -r ks4 -s ks4backup -p backup >> /var/log/incus-copy-ks4.log 2>&1 ``` `/etc/logrotate.d/incus-copy` already lists `incus-copy-ks4.log`, so nothing to add there. ## Verification (release gate for ks2) ```sh incus list --project backup -c ns -f csv # all ks4 instances present # test-restore one instance: copy a replica to the local pool, # start it isolated, check the service answers, then delete it incus copy solar solar-restoretest -s incus incus start solar-restoretest && incus exec solar-restoretest -- systemctl is-system-running incus delete -f solar-restoretest ``` Once verified, tick the nas gate in the [ks2 plan](plan.md). Only one piece of nuc's retirement is still outstanding — dropping its now-unused peer on ks4. Harmless to leave (an unused peer costs nothing) and safe to do at any time, since nuc's tunnel is already down: ```sh # on ks4 — nuc's pubkey is 31Tlgloc… incus exec wireguard -- wg set wg0 peer 31TlglocNJyooDVAO8HWEC0lyCykhbaFIWVWFUCOrmQ= remove incus exec wireguard -- wg-quick save wg0 ``` Deleting `/etc/wireguard/wg-ks4.conf` + `.key` on nuc is deliberately **not** done: they cost nothing and regenerating keys would mean re-peering on ks4.