doc/ks2/nas-seed.md

ks4 pull leg — seed after FTTH

Status: built and seeding — 2026-09-16. FTTH is up (gateway 192.168.0.1, see nuc/nuc-install.md); the tunnel, the incus remote and the cron are in place and the first pass is running. Remaining: let the seed finish, then the test-restore below.

⚠️ Changed 2026-08-30: this leg lands on nas, not on nuc. It was originally designed for nuc's USB pool usb4t, but that pool proved to be the least reliable device in the setup (nuc/usb4t-dropouts.md) — which is exactly what an off-site copy of ks4 must not be. The 4 TB disk moved to direct SATA on the new host nas (192.168.0.4, nas/nas-install.md), and the target pool ks4backup moved with it.

Consequences versus the original plan:

• Target pool ks4backup is now backed by tank/backup/ks4 on nas.
• The WireGuard tunnel moves too: nas becomes peer 10.8.0.22. transmission-bt, the only other user, moved to nas and carries its own in-container tunnel (10.8.0.21, unchanged — ks4 needs no edit for it).
• ⚠️ nuc's wg-ks4 was already disabled on 2026-08-31, before the seed, not after. The original plan retired it only once nas was seeded, on the assumption nuc could serve as a fallback target — it cannot: its ks4backup pool was deleted and its data pool is a 512 GB SSD, far too small for the ~1.75 TiB replica set. Keeping a keepalive'd tunnel alive on a machine that is now powered off between uses bought nothing. wg-quick@wg-ks4 is disabled, and the ks4 incus remote was removed from nuc. /etc/wireguard/wg-ks4.conf and its key are kept, so it is one systemctl enable --now wg-quick@wg-ks4 away if ever needed.
• ks4's ufw rule is unchanged: traffic arrives masqueraded as the wireguard container (192.168.1.18) whichever peer sent it.

Prerequisites

• tank running on SATA for ≥ 7 days with zero pool suspensions — the gate that replaces "fix the USB enclosure". Verified 2026-09-16: ONLINE, scrub clean 2026-09-14, uptime 2 w 2 d, zero suspensions in the journal.
• FTTH up (the first pass moves ~1.75 TiB)

Setup (root on nas)

# 0. the nas host has no wireguard-tools — transmission-bt carries its own
#    tunnel *inside* the container, so the host never needed them
apt-get install -y wireguard-tools

# 1. key on nas, then peer it on ks4's wireguard container
umask 077; wg genkey > /etc/wireguard/wg-ks4.key
wg pubkey < /etc/wireguard/wg-ks4.key          # -> <nas-pubkey>
#    (run on ks4)
incus exec wireguard -- wg set wg0 peer <nas-pubkey> allowed-ips 10.8.0.22/32
incus exec wireguard -- wg-quick save wg0

# 2. tunnel on nas: /etc/wireguard/wg-ks4.conf, modelled on nuc's
#    Address = 10.8.0.22/24, peer pubkey TVs6d7…,
#    Endpoint = 193.70.35.17:51845,
#    AllowedIPs = 10.8.0.0/24, 192.168.1.1/32, keepalive 25
systemctl enable --now wg-quick@wg-ks4

# 3. incus remote over the tunnel
incus remote add ks4 https://192.168.1.1:8443 --accept-certificate --token '…'
incus list ks4: | head          # sanity: remote reachable

⚠️ nas needs a managed incusbr0 of its own, or every copy fails instantly. All 18 ks4 instances carry an instance-level eth0 (nictype: bridged, parent: incusbr0, ipv4.address: 192.168.1.x). nas has no such bridge, so instance creation dies with:

Device validation failed for "eth0": Cannot use manually specified
ipv4.address when using unmanaged parent bridge

Create a managed network of the same name — but give it .254, never .1: 192.168.1.1 must keep resolving over wg-ks4 to ks4's incus API, and a local address always beats a route.

incus network create incusbr0 \
    ipv4.address=192.168.1.254/24 ipv4.nat=false ipv6.address=none
ip route get 192.168.1.1        # must still say: dev wg-ks4

The bridge stays inert — replicas are never started here.

Seed

⚠️ -p backup is required, exactly as for the nuc leg (nas/nas-install.md §9a) — and as the verification command below already assumed. Without it the 18 replicas land in default alongside nas's live instances.

Run it detached rather than in a shell that can drop — the first pass is long: ~1.42 TiB at ~14 MB/s ≈ 29 h (see the bottleneck section below for why it is 14 MB/s and not more).

systemd-run --unit=ks4-seed --collect \
  /bin/bash -c '/root/scripts/incus-copy.sh -r ks4 -s ks4backup -p backup \
                >> /var/log/incus-copy-ks4.log 2>&1'

systemctl is-active ks4-seed          # progress:
tail -f /var/log/incus-copy-ks4.log

⚠️ The seed suppresses the 04:00 nasbackup job while it runs. incus-copy.sh takes a single /run/lock/incus-copy.lock for every shape, so any run starting while the seed holds it aborts with another incus-copy run holds …. Over a ~29 h seed that skips one or two nights of the nas-local copy — accepted; those replicas are small, same-host, and rebuildable. The 03:30 nuc push is unaffected (it runs on nuc, with nuc's own lock).

Bottleneck: ks4's source disk, not the network

Measured 2026-09-16, because ~112 Mbit/s looked far too slow for a 2 Gbit/s FTTH line. It is not the link, and not WireGuard. Do not go looking for a network fix:

Path	Measured
ks4 upload → internet	609 Mbit/s
ks4 download ← internet	900 Mbit/s
nas download ← OVH network	670 Mbit/s
ks4 → nas through wg-ks4	~112 Mbit/s

WireGuard was ruled out too: UdpRcvbufErrors 0 on nas (so the default net.core.rmem_max of 208 KB is not dropping packets), wg-crypt kworkers ~2 %, both hosts ~85 % idle.

The limit is data living on sdb5, a single 7200 rpm HGST 6 TB spinning disk. During the send iostat showed sdb at 109 r/s / 14 MB/s, ~131 KB average request, queue depth ~1.0 — the random-IOPS ceiling of one HDD walking a fragmented 1.42 TiB dataset. 14 MB/s is ~112 Mbit/s on the wire, which is exactly the observed rate. The network is idle the whole time.

Parallelism is the only real lever, and it is a disk-queue effect, not a bandwidth one. Running a second instance copy alongside the seed:

	sequential	+ 1 parallel copy
sdb read	14 MB/s	21 MB/s
avg request size	131 KB	514 KB
tunnel	113 Mbit/s	153 Mbit/s

With two senders queued, ZFS issues larger, more sequential reads instead of seeking one request at a time. Three or four concurrent copies would plausibly reach 25–30 MB/s and roughly halve the seed.

Decision 2026-09-16: keep it sequential. The first pass is a one-off, later refreshes are ZFS-incremental and tiny, and incus-copy.sh is shared by all three legs — parallelising means either reworking the script or running copies outside its flock, i.e. two jobs contending for the same dataset. Not worth one overnight. If a full reseed is ever needed and the wall-clock matters, this is the knob; the disk is the floor either way.

Notes:

• First pass is a full send per instance; later refreshes are ZFS-incremental as long as they run at least every snapshots.expiry (7 d on ks4) — same caveat as ks4's local leg.
• Replicas arrive stopped with boot.autostart=false (the script does this) — they must never come up on the LAN with ks4's proxy devices.

Cron (after the seed)

Add to nas's root crontab, offset from the 03:30 nuc→nas push, the 04:00 nas→nuc push and ks4's own 01:00/05:00 jobs:

0 5 * * * /root/scripts/incus-copy.sh -r ks4 -s ks4backup -p backup >> /var/log/incus-copy-ks4.log 2>&1

/etc/logrotate.d/incus-copy already lists incus-copy-ks4.log, so nothing to add there.

Verification (release gate for ks2)

incus list --project backup -c ns -f csv         # all ks4 instances present
# test-restore one instance: copy a replica to the local pool,
# start it isolated, check the service answers, then delete it
incus copy solar solar-restoretest -s incus
incus start solar-restoretest && incus exec solar-restoretest -- systemctl is-system-running
incus delete -f solar-restoretest

Once verified, tick the nas gate in the ks2 plan.

Only one piece of nuc's retirement is still outstanding — dropping its now-unused peer on ks4. Harmless to leave (an unused peer costs nothing) and safe to do at any time, since nuc's tunnel is already down:

# on ks4 — nuc's pubkey is 31Tlgloc…
incus exec wireguard -- wg set wg0 peer 31TlglocNJyooDVAO8HWEC0lyCykhbaFIWVWFUCOrmQ= remove
incus exec wireguard -- wg-quick save wg0

Deleting /etc/wireguard/wg-ks4.conf + .key on nuc is deliberately not done: they cost nothing and regenerating keys would mean re-peering on ks4.