doc: record the real ks4 pull bottleneck — source disk, not the link
c8d759c claimed the ~125 Mbit/s seed rate was ks4's OVH uplink. That was
wrong, and measuring it says so: ks4 uploads at 609 Mbit/s, downloads at
900, and nas pulls 670 from OVH's network. WireGuard is not it either —
zero UdpRcvbufErrors, wg-crypt kworkers at ~2%, both hosts ~85% idle.
The limit is `data` sitting on sdb5, one 7200 rpm HGST 6 TB disk: during
the send it does 109 r/s at 14 MB/s with ~131 KB requests and a queue
depth of ~1.0, which is the random-IOPS ceiling of a single HDD reading a
fragmented 1.42 TiB dataset. 14 MB/s is ~112 Mbit/s on the wire, exactly
what we see, while the network sits idle.
Parallelism is the only lever and it works by disk queue depth, not
bandwidth: a second concurrent copy takes sdb from 14 to 21 MB/s and the
request size from 131 KB to 514 KB, lifting the tunnel from 113 to 153
Mbit/s. Recorded, along with the decision to keep the seed sequential —
one-off pass, incremental refreshes after, and incus-copy.sh is shared by
all three legs.
Also corrects the set size to the actual 1.42 TiB (~29 h, not ~31 h).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5
parent
c8d759ce9e
commit
ceff4ec0d0
+47
-3
@@ -97,8 +97,8 @@ verification command below already assumed. Without it the 18 replicas
|
|||||||
land in `default` alongside nas's live instances.
|
land in `default` alongside nas's live instances.
|
||||||
|
|
||||||
Run it detached rather than in a shell that can drop — the first pass is
|
Run it detached rather than in a shell that can drop — the first pass is
|
||||||
long (measured **~125 Mbit/s**, which is ks4's OVH *uplink*, not the home
|
long: **~1.42 TiB at ~14 MB/s ≈ 29 h** (see the bottleneck section below
|
||||||
downlink: **~31 h** for the full set):
|
for why it is 14 MB/s and not more).
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
systemd-run --unit=ks4-seed --collect \
|
systemd-run --unit=ks4-seed --collect \
|
||||||
@@ -112,11 +112,55 @@ tail -f /var/log/incus-copy-ks4.log
|
|||||||
⚠️ **The seed suppresses the 04:00 `nasbackup` job while it runs.**
|
⚠️ **The seed suppresses the 04:00 `nasbackup` job while it runs.**
|
||||||
`incus-copy.sh` takes a single `/run/lock/incus-copy.lock` for every
|
`incus-copy.sh` takes a single `/run/lock/incus-copy.lock` for every
|
||||||
shape, so any run starting while the seed holds it aborts with
|
shape, so any run starting while the seed holds it aborts with
|
||||||
`another incus-copy run holds …`. Over a ~31 h seed that skips one or two
|
`another incus-copy run holds …`. Over a ~29 h seed that skips one or two
|
||||||
nights of the nas-local copy — accepted; those replicas are small,
|
nights of the nas-local copy — accepted; those replicas are small,
|
||||||
same-host, and rebuildable. The 03:30 nuc push is unaffected (it runs on
|
same-host, and rebuildable. The 03:30 nuc push is unaffected (it runs on
|
||||||
nuc, with nuc's own lock).
|
nuc, with nuc's own lock).
|
||||||
|
|
||||||
|
### Bottleneck: ks4's source disk, not the network
|
||||||
|
|
||||||
|
Measured 2026-09-16, because ~112 Mbit/s looked far too slow for a 2 Gbit/s
|
||||||
|
FTTH line. **It is not the link, and not WireGuard.** Do not go looking for
|
||||||
|
a network fix:
|
||||||
|
|
||||||
|
| Path | Measured |
|
||||||
|
|---|---|
|
||||||
|
| ks4 upload → internet | 609 Mbit/s |
|
||||||
|
| ks4 download ← internet | 900 Mbit/s |
|
||||||
|
| nas download ← OVH network | 670 Mbit/s |
|
||||||
|
| ks4 → nas through `wg-ks4` | **~112 Mbit/s** |
|
||||||
|
|
||||||
|
WireGuard was ruled out too: `UdpRcvbufErrors` 0 on nas (so the default
|
||||||
|
`net.core.rmem_max` of 208 KB is *not* dropping packets), wg-crypt
|
||||||
|
kworkers ~2 %, both hosts ~85 % idle.
|
||||||
|
|
||||||
|
The limit is **`data` living on `sdb5`, a single 7200 rpm HGST 6 TB
|
||||||
|
spinning disk**. During the send `iostat` showed sdb at **109 r/s /
|
||||||
|
14 MB/s, ~131 KB average request, queue depth ~1.0** — the random-IOPS
|
||||||
|
ceiling of one HDD walking a fragmented 1.42 TiB dataset. 14 MB/s is
|
||||||
|
~112 Mbit/s on the wire, which is exactly the observed rate. The network
|
||||||
|
is idle the whole time.
|
||||||
|
|
||||||
|
**Parallelism is the only real lever, and it is a disk-queue effect**, not
|
||||||
|
a bandwidth one. Running a second instance copy alongside the seed:
|
||||||
|
|
||||||
|
| | sequential | + 1 parallel copy |
|
||||||
|
|---|---|---|
|
||||||
|
| sdb read | 14 MB/s | **21 MB/s** |
|
||||||
|
| avg request size | 131 KB | **514 KB** |
|
||||||
|
| tunnel | 113 Mbit/s | **153 Mbit/s** |
|
||||||
|
|
||||||
|
With two senders queued, ZFS issues larger, more sequential reads instead
|
||||||
|
of seeking one request at a time. Three or four concurrent copies would
|
||||||
|
plausibly reach 25–30 MB/s and roughly halve the seed.
|
||||||
|
|
||||||
|
**Decision 2026-09-16: keep it sequential.** The first pass is a one-off,
|
||||||
|
later refreshes are ZFS-incremental and tiny, and `incus-copy.sh` is
|
||||||
|
shared by all three legs — parallelising means either reworking the script
|
||||||
|
or running copies outside its flock, i.e. two jobs contending for the same
|
||||||
|
dataset. Not worth one overnight. If a full reseed is ever needed and the
|
||||||
|
wall-clock matters, this is the knob; the disk is the floor either way.
|
||||||
|
|
||||||
Notes:
|
Notes:
|
||||||
|
|
||||||
- First pass is a full send per instance; later refreshes are
|
- First pass is a full send per instance; later refreshes are
|
||||||
|
|||||||
@@ -715,6 +715,11 @@ Built 2026-09-16 — full runbook, corrections and gotchas:
|
|||||||
missed: nas needs `wireguard-tools` installed and a **managed `incusbr0`
|
missed: nas needs `wireguard-tools` installed and a **managed `incusbr0`
|
||||||
on `192.168.1.254/24`** (never `.1`), and the seed needs **`-p backup`**.
|
on `192.168.1.254/24`** (never `.1`), and the seed needs **`-p backup`**.
|
||||||
|
|
||||||
|
The pull runs at ~14 MB/s (~112 Mbit/s) and that is **ks4's single
|
||||||
|
spinning source disk, not the link or the tunnel** — measured, with the
|
||||||
|
numbers, in [ks2/nas-seed.md](../ks2/nas-seed.md) §Bottleneck. Nothing to
|
||||||
|
fix on the network side.
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
systemd-run --unit=ks4-seed --collect \
|
systemd-run --unit=ks4-seed --collect \
|
||||||
/bin/bash -c '/root/scripts/incus-copy.sh -r ks4 -s ks4backup -p backup \
|
/bin/bash -c '/root/scripts/incus-copy.sh -r ks4 -s ks4backup -p backup \
|
||||||
|
|||||||
Reference in New Issue
Block a user