From ceff4ec0d0c1d0efd61a92626be4d2605a1e3d70 Mon Sep 17 00:00:00 2001 From: Julien Lutran Date: Wed, 16 Sep 2026 14:47:28 +0200 Subject: [PATCH] =?UTF-8?q?doc:=20record=20the=20real=20ks4=20pull=20bottl?= =?UTF-8?q?eneck=20=E2=80=94=20source=20disk,=20not=20the=20link?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit c8d759c claimed the ~125 Mbit/s seed rate was ks4's OVH uplink. That was wrong, and measuring it says so: ks4 uploads at 609 Mbit/s, downloads at 900, and nas pulls 670 from OVH's network. WireGuard is not it either — zero UdpRcvbufErrors, wg-crypt kworkers at ~2%, both hosts ~85% idle. The limit is `data` sitting on sdb5, one 7200 rpm HGST 6 TB disk: during the send it does 109 r/s at 14 MB/s with ~131 KB requests and a queue depth of ~1.0, which is the random-IOPS ceiling of a single HDD reading a fragmented 1.42 TiB dataset. 14 MB/s is ~112 Mbit/s on the wire, exactly what we see, while the network sits idle. Parallelism is the only lever and it works by disk queue depth, not bandwidth: a second concurrent copy takes sdb from 14 to 21 MB/s and the request size from 131 KB to 514 KB, lifting the tunnel from 113 to 153 Mbit/s. Recorded, along with the decision to keep the seed sequential — one-off pass, incremental refreshes after, and incus-copy.sh is shared by all three legs. Also corrects the set size to the actual 1.42 TiB (~29 h, not ~31 h). Co-Authored-By: Claude Opus 5 (1M context) --- ks2/nas-seed.md | 50 +++++++++++++++++++++++++++++++++++++++++++--- nas/nas-install.md | 5 +++++ 2 files changed, 52 insertions(+), 3 deletions(-) diff --git a/ks2/nas-seed.md b/ks2/nas-seed.md index 34dc96c..7277251 100644 --- a/ks2/nas-seed.md +++ b/ks2/nas-seed.md @@ -97,8 +97,8 @@ verification command below already assumed. Without it the 18 replicas land in `default` alongside nas's live instances. Run it detached rather than in a shell that can drop — the first pass is -long (measured **~125 Mbit/s**, which is ks4's OVH *uplink*, not the home -downlink: **~31 h** for the full set): +long: **~1.42 TiB at ~14 MB/s ≈ 29 h** (see the bottleneck section below +for why it is 14 MB/s and not more). ```sh systemd-run --unit=ks4-seed --collect \ @@ -112,11 +112,55 @@ tail -f /var/log/incus-copy-ks4.log ⚠️ **The seed suppresses the 04:00 `nasbackup` job while it runs.** `incus-copy.sh` takes a single `/run/lock/incus-copy.lock` for every shape, so any run starting while the seed holds it aborts with -`another incus-copy run holds …`. Over a ~31 h seed that skips one or two +`another incus-copy run holds …`. Over a ~29 h seed that skips one or two nights of the nas-local copy — accepted; those replicas are small, same-host, and rebuildable. The 03:30 nuc push is unaffected (it runs on nuc, with nuc's own lock). +### Bottleneck: ks4's source disk, not the network + +Measured 2026-09-16, because ~112 Mbit/s looked far too slow for a 2 Gbit/s +FTTH line. **It is not the link, and not WireGuard.** Do not go looking for +a network fix: + +| Path | Measured | +|---|---| +| ks4 upload → internet | 609 Mbit/s | +| ks4 download ← internet | 900 Mbit/s | +| nas download ← OVH network | 670 Mbit/s | +| ks4 → nas through `wg-ks4` | **~112 Mbit/s** | + +WireGuard was ruled out too: `UdpRcvbufErrors` 0 on nas (so the default +`net.core.rmem_max` of 208 KB is *not* dropping packets), wg-crypt +kworkers ~2 %, both hosts ~85 % idle. + +The limit is **`data` living on `sdb5`, a single 7200 rpm HGST 6 TB +spinning disk**. During the send `iostat` showed sdb at **109 r/s / +14 MB/s, ~131 KB average request, queue depth ~1.0** — the random-IOPS +ceiling of one HDD walking a fragmented 1.42 TiB dataset. 14 MB/s is +~112 Mbit/s on the wire, which is exactly the observed rate. The network +is idle the whole time. + +**Parallelism is the only real lever, and it is a disk-queue effect**, not +a bandwidth one. Running a second instance copy alongside the seed: + +| | sequential | + 1 parallel copy | +|---|---|---| +| sdb read | 14 MB/s | **21 MB/s** | +| avg request size | 131 KB | **514 KB** | +| tunnel | 113 Mbit/s | **153 Mbit/s** | + +With two senders queued, ZFS issues larger, more sequential reads instead +of seeking one request at a time. Three or four concurrent copies would +plausibly reach 25–30 MB/s and roughly halve the seed. + +**Decision 2026-09-16: keep it sequential.** The first pass is a one-off, +later refreshes are ZFS-incremental and tiny, and `incus-copy.sh` is +shared by all three legs — parallelising means either reworking the script +or running copies outside its flock, i.e. two jobs contending for the same +dataset. Not worth one overnight. If a full reseed is ever needed and the +wall-clock matters, this is the knob; the disk is the floor either way. + Notes: - First pass is a full send per instance; later refreshes are diff --git a/nas/nas-install.md b/nas/nas-install.md index 3a64eb2..386b70e 100644 --- a/nas/nas-install.md +++ b/nas/nas-install.md @@ -715,6 +715,11 @@ Built 2026-09-16 — full runbook, corrections and gotchas: missed: nas needs `wireguard-tools` installed and a **managed `incusbr0` on `192.168.1.254/24`** (never `.1`), and the seed needs **`-p backup`**. +The pull runs at ~14 MB/s (~112 Mbit/s) and that is **ks4's single +spinning source disk, not the link or the tunnel** — measured, with the +numbers, in [ks2/nas-seed.md](../ks2/nas-seed.md) §Bottleneck. Nothing to +fix on the network side. + ```sh systemd-run --unit=ks4-seed --collect \ /bin/bash -c '/root/scripts/incus-copy.sh -r ks4 -s ks4backup -p backup \