doc: record the real ks4 pull bottleneck — source disk, not the link

c8d759c claimed the ~125 Mbit/s seed rate was ks4's OVH uplink. That was
wrong, and measuring it says so: ks4 uploads at 609 Mbit/s, downloads at
900, and nas pulls 670 from OVH's network. WireGuard is not it either —
zero UdpRcvbufErrors, wg-crypt kworkers at ~2%, both hosts ~85% idle.

The limit is `data` sitting on sdb5, one 7200 rpm HGST 6 TB disk: during
the send it does 109 r/s at 14 MB/s with ~131 KB requests and a queue
depth of ~1.0, which is the random-IOPS ceiling of a single HDD reading a
fragmented 1.42 TiB dataset. 14 MB/s is ~112 Mbit/s on the wire, exactly
what we see, while the network sits idle.

Parallelism is the only lever and it works by disk queue depth, not
bandwidth: a second concurrent copy takes sdb from 14 to 21 MB/s and the
request size from 131 KB to 514 KB, lifting the tunnel from 113 to 153
Mbit/s. Recorded, along with the decision to keep the seed sequential —
one-off pass, incremental refreshes after, and incus-copy.sh is shared by
all three legs.

Also corrects the set size to the actual 1.42 TiB (~29 h, not ~31 h).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Julien Lutran
2026-09-16 14:47:28 +02:00
co-authored by Claude Opus 5
parent c8d759ce9e
commit ceff4ec0d0
2 changed files with 52 additions and 3 deletions
+47 -3
View File
@@ -97,8 +97,8 @@ verification command below already assumed. Without it the 18 replicas
land in `default` alongside nas's live instances.
Run it detached rather than in a shell that can drop — the first pass is
long (measured **~125 Mbit/s**, which is ks4's OVH *uplink*, not the home
downlink: **~31 h** for the full set):
long: **~1.42 TiB at ~14 MB/s ≈ 29 h** (see the bottleneck section below
for why it is 14 MB/s and not more).
```sh
systemd-run --unit=ks4-seed --collect \
@@ -112,11 +112,55 @@ tail -f /var/log/incus-copy-ks4.log
⚠️ **The seed suppresses the 04:00 `nasbackup` job while it runs.**
`incus-copy.sh` takes a single `/run/lock/incus-copy.lock` for every
shape, so any run starting while the seed holds it aborts with
`another incus-copy run holds …`. Over a ~31 h seed that skips one or two
`another incus-copy run holds …`. Over a ~29 h seed that skips one or two
nights of the nas-local copy — accepted; those replicas are small,
same-host, and rebuildable. The 03:30 nuc push is unaffected (it runs on
nuc, with nuc's own lock).
### Bottleneck: ks4's source disk, not the network
Measured 2026-09-16, because ~112 Mbit/s looked far too slow for a 2 Gbit/s
FTTH line. **It is not the link, and not WireGuard.** Do not go looking for
a network fix:
| Path | Measured |
|---|---|
| ks4 upload → internet | 609 Mbit/s |
| ks4 download ← internet | 900 Mbit/s |
| nas download ← OVH network | 670 Mbit/s |
| ks4 → nas through `wg-ks4` | **~112 Mbit/s** |
WireGuard was ruled out too: `UdpRcvbufErrors` 0 on nas (so the default
`net.core.rmem_max` of 208 KB is *not* dropping packets), wg-crypt
kworkers ~2 %, both hosts ~85 % idle.
The limit is **`data` living on `sdb5`, a single 7200 rpm HGST 6 TB
spinning disk**. During the send `iostat` showed sdb at **109 r/s /
14 MB/s, ~131 KB average request, queue depth ~1.0** — the random-IOPS
ceiling of one HDD walking a fragmented 1.42 TiB dataset. 14 MB/s is
~112 Mbit/s on the wire, which is exactly the observed rate. The network
is idle the whole time.
**Parallelism is the only real lever, and it is a disk-queue effect**, not
a bandwidth one. Running a second instance copy alongside the seed:
| | sequential | + 1 parallel copy |
|---|---|---|
| sdb read | 14 MB/s | **21 MB/s** |
| avg request size | 131 KB | **514 KB** |
| tunnel | 113 Mbit/s | **153 Mbit/s** |
With two senders queued, ZFS issues larger, more sequential reads instead
of seeking one request at a time. Three or four concurrent copies would
plausibly reach 2530 MB/s and roughly halve the seed.
**Decision 2026-09-16: keep it sequential.** The first pass is a one-off,
later refreshes are ZFS-incremental and tiny, and `incus-copy.sh` is
shared by all three legs — parallelising means either reworking the script
or running copies outside its flock, i.e. two jobs contending for the same
dataset. Not worth one overnight. If a full reseed is ever needed and the
wall-clock matters, this is the knob; the disk is the floor either way.
Notes:
- First pass is a full send per instance; later refreshes are