plakar-s3-data: seed done except seafile-fs (EINTR livelock, excluded)
Seafile's block store wedged the fs walker for 12h in an lstat preemption/EINTR livelock on the live pool; coverage moves to the incus leg reading the sdb replica. Restore test + cron are the remaining steps of the data leg. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Julien Lutran
co-authored by
Claude Fable 5
Claude Fable 5
parent
4b73d77803
commit
ae82d9f682
@@ -0,0 +1,193 @@
|
|||||||
|
# Archer C7: upgrade OpenWrt 19.07.4 → 25.12.5
|
||||||
|
|
||||||
|
> **Outcome (2026-08-20):** upgrade done (flashed clean per this doc).
|
||||||
|
> The USB-tether flapping survived the new kernel → root cause is the
|
||||||
|
> C7's USB port lacking charging current (phone dropped off the bus
|
||||||
|
> whenever it drew charge). **WAN is now the phone's Wi-Fi hotspot**:
|
||||||
|
> 2.4 GHz `radio1` in STA mode → interface `wwan` (`phy1-sta0`, in the
|
||||||
|
> wan zone), phone on a mains charger. `usbwan` stays configured for a
|
||||||
|
> future powered-USB-hub wired option. Post-flash gotcha: fw4 compiled
|
||||||
|
> NAT for `eth0.2` only while `usbwan` was down at reload time — LAN
|
||||||
|
> had no internet until `service firewall restart` with the WAN
|
||||||
|
> interface up.
|
||||||
|
|
||||||
|
Home router **TP-Link Archer C7 v5** (`tplink,archer-c7-v5`, target
|
||||||
|
`ath79/generic`, 16 MB flash / 128 MB RAM) at `192.168.0.2`, running
|
||||||
|
OpenWrt 19.07.4 (2020, kernel 4.14, EOL since 2022). Target release:
|
||||||
|
**OpenWrt 25.12.5** (kernel 6.12.94, firewall4/nftables, `apk` package
|
||||||
|
manager instead of `opkg`).
|
||||||
|
|
||||||
|
Context (2026-08-19): after a reboot the router came up with **dnsmasq,
|
||||||
|
firewall and odhcpd disabled** (missing `/etc/rc.d/` symlinks — no DHCP,
|
||||||
|
no NAT). All three were restarted and re-enabled, but the incident plus
|
||||||
|
the EOL release motivated this upgrade. A fresh flash also rules out
|
||||||
|
any other overlay damage.
|
||||||
|
|
||||||
|
Current setup worth knowing:
|
||||||
|
|
||||||
|
- **WAN = OPPO phone (SM8350) USB-tethered** on `usb0` via RNDIS
|
||||||
|
(`kmod-usb-net-rndis`), interface `usbwan`, DHCP from the phone
|
||||||
|
(10.205.61.0/24). The ethernet WAN port is unused — the ISP box at
|
||||||
|
`.1` is dead, which is also why the router LAN IP is `.2`.
|
||||||
|
- SSH: `ssh -i id_rsa_claude root@192.168.0.2` (key at perso repo root;
|
||||||
|
19.07's dropbear needs
|
||||||
|
`-o HostKeyAlgorithms=+ssh-rsa -o PubkeyAcceptedAlgorithms=+ssh-rsa`
|
||||||
|
from a modern OpenSSH; 25.12 won't)
|
||||||
|
- DNS: dnsmasq forwards everything to **blocky** at `192.168.0.254`
|
||||||
|
(`noresolv` + `server=192.168.0.254`), and DHCP option 6 hands blocky
|
||||||
|
directly to clients
|
||||||
|
- Wi-Fi: 5 GHz only (`radio0`, ch 36, VHT80, SSID `HOME`, WPA2-PSK);
|
||||||
|
2.4 GHz radio disabled
|
||||||
|
- 7 static DHCP leases (see script below)
|
||||||
|
- Only user-installed packages on 19.07: `kmod-usb-net-rndis` + deps
|
||||||
|
|
||||||
|
## Prerequisites (done 2026-08-19)
|
||||||
|
|
||||||
|
- [x] Config backup: `sysupgrade -b` → `~/code/perso/backup-archer-c7-20260819.tar.gz`
|
||||||
|
(laptop; contains Wi-Fi PSK and dropbear host keys — never commit)
|
||||||
|
- [x] Firmware `openwrt-25.12.5-ath79-generic-tplink_archer-c7-v5-squashfs-sysupgrade.bin`
|
||||||
|
downloaded, **sha256 verified** (`f15b16b3…`) against the release
|
||||||
|
`sha256sums`, staged in `~/code/perso/archer-c7-upgrade/` and on
|
||||||
|
the router at `/tmp/` (`/tmp` is RAM — gone after any reboot,
|
||||||
|
re-`scp` if needed)
|
||||||
|
- [x] **Offline driver packages** staged in `~/code/perso/archer-c7-upgrade/`:
|
||||||
|
`kmod-mii`, `kmod-usb-net`, `kmod-usb-net-cdc-ether`,
|
||||||
|
`kmod-usb-net-rndis` (all `6.12.94-r1.apk`). Critical: the phone
|
||||||
|
tether is the only WAN and the RNDIS driver is **not** in the
|
||||||
|
default image, so after flashing the router has no internet to
|
||||||
|
install it from — it must be `scp`'d in.
|
||||||
|
|
||||||
|
## Upgrade
|
||||||
|
|
||||||
|
**Do not keep settings across this jump** (19.07 → 25.12 spans the
|
||||||
|
fw3→fw4 and network-syntax migrations). Flash clean with `-n` and
|
||||||
|
reconfigure with the script below.
|
||||||
|
|
||||||
|
You need the **laptop on ethernet** to a LAN port: the fresh image
|
||||||
|
boots at `192.168.1.1` with Wi-Fi disabled.
|
||||||
|
|
||||||
|
```sh
|
||||||
|
# on the router — image already in /tmp, verify then flash
|
||||||
|
sha256sum /tmp/openwrt-25.12.5-ath79-generic-tplink_archer-c7-v5-squashfs-sysupgrade.bin
|
||||||
|
# expect f15b16b36df7fce8a5cf1302320fdb428c3bb6fbc2949a4858afa257f4f364a2
|
||||||
|
sysupgrade -n -v /tmp/openwrt-25.12.5-ath79-generic-tplink_archer-c7-v5-squashfs-sysupgrade.bin
|
||||||
|
```
|
||||||
|
|
||||||
|
The router reboots (~3 min, don't power-cycle). Then from the laptop
|
||||||
|
(ethernet, DHCP gives a 192.168.1.x address):
|
||||||
|
|
||||||
|
```sh
|
||||||
|
# passwordless root at first boot — set a password and install the key
|
||||||
|
ssh root@192.168.1.1 # no password
|
||||||
|
passwd
|
||||||
|
mkdir -p /etc/dropbear && exit
|
||||||
|
scp ~/code/perso/id_rsa_claude.pub root@192.168.1.1:/etc/dropbear/authorized_keys
|
||||||
|
|
||||||
|
# push the offline RNDIS drivers
|
||||||
|
scp ~/code/perso/archer-c7-upgrade/kmod-*.apk root@192.168.1.1:/tmp/
|
||||||
|
```
|
||||||
|
|
||||||
|
## Reconfigure
|
||||||
|
|
||||||
|
Run on the router. Single source of truth for the whole config;
|
||||||
|
`<wifi-psk>` is in the backup tarball (`etc/config/wireless`).
|
||||||
|
|
||||||
|
```sh
|
||||||
|
#!/bin/sh
|
||||||
|
set -eux
|
||||||
|
|
||||||
|
# RNDIS drivers (offline — phone tether is the only WAN)
|
||||||
|
apk add --allow-untrusted /tmp/kmod-mii-*.apk /tmp/kmod-usb-net-6*.apk \
|
||||||
|
/tmp/kmod-usb-net-cdc-ether-*.apk /tmp/kmod-usb-net-rndis-*.apk
|
||||||
|
|
||||||
|
# --- network: keep the generated switch/bridge config, change only what we need
|
||||||
|
uci set network.lan.ipaddr='192.168.0.2'
|
||||||
|
uci set network.usbwan=interface
|
||||||
|
uci set network.usbwan.proto='dhcp'
|
||||||
|
uci set network.usbwan.device='usb0'
|
||||||
|
uci commit network
|
||||||
|
|
||||||
|
# --- firewall: usbwan joins the wan zone (zone[1] = 'wan' in the default config)
|
||||||
|
uci add_list firewall.@zone[1].network='usbwan'
|
||||||
|
uci commit firewall
|
||||||
|
|
||||||
|
# --- dhcp/dns: forward to blocky, hand blocky to clients, static leases
|
||||||
|
uci set dhcp.@dnsmasq[0].noresolv='1'
|
||||||
|
uci add_list dhcp.@dnsmasq[0].server='192.168.0.254'
|
||||||
|
uci add_list dhcp.lan.dhcp_option='6,192.168.0.254'
|
||||||
|
add_host() {
|
||||||
|
uci add dhcp host >/dev/null
|
||||||
|
uci set dhcp.@host[-1].mac="$1"
|
||||||
|
uci set dhcp.@host[-1].ip="$2"
|
||||||
|
uci set dhcp.@host[-1].name="$3"
|
||||||
|
uci set dhcp.@host[-1].dns='1'
|
||||||
|
}
|
||||||
|
add_host 'F4:5C:89:BE:C1:C3' '192.168.0.20' 'LAPTOP719974'
|
||||||
|
add_host '10:66:6A:87:A1:49' '192.168.0.254' 'blocky'
|
||||||
|
add_host '10:66:6A:C8:E1:0A' '192.168.0.5' 'jellyfin-server'
|
||||||
|
add_host '10:66:6A:FF:59:1C' '192.168.0.6' 'jellyfin-client'
|
||||||
|
add_host '10:66:6A:85:20:B7' '192.168.0.11' 'privoxy'
|
||||||
|
add_host '10:66:6A:55:9F:10' '192.168.0.7' 'transmission-bt'
|
||||||
|
add_host '50:9A:4C:20:D9:2B' '192.168.0.21' 'patate'
|
||||||
|
uci commit dhcp
|
||||||
|
|
||||||
|
# --- wifi: 5 GHz AP as before, upgraded to WPA2/WPA3 mixed; 2.4 GHz stays off
|
||||||
|
uci set wireless.radio0.channel='36'
|
||||||
|
uci set wireless.radio0.htmode='VHT80'
|
||||||
|
uci set wireless.radio0.disabled='0'
|
||||||
|
uci set wireless.default_radio0.ssid='HOME'
|
||||||
|
uci set wireless.default_radio0.encryption='sae-mixed'
|
||||||
|
uci set wireless.default_radio0.key='<wifi-psk>'
|
||||||
|
uci set wireless.default_radio0.network='lan'
|
||||||
|
uci commit wireless
|
||||||
|
|
||||||
|
reload_config
|
||||||
|
/etc/init.d/network restart
|
||||||
|
```
|
||||||
|
|
||||||
|
The LAN IP change kicks you off; reconnect at `192.168.0.2` (plug the
|
||||||
|
phone back in for WAN).
|
||||||
|
|
||||||
|
## Verify
|
||||||
|
|
||||||
|
```sh
|
||||||
|
ssh root@192.168.0.2
|
||||||
|
ifstatus usbwan | grep -E 'up|address' # up:true, 10.205.61.x
|
||||||
|
ping -c2 8.8.8.8 && ping -c2 openwrt.org # WAN + DNS via blocky
|
||||||
|
nft list chain inet fw4 srcnat_wan # masquerade on usbwan
|
||||||
|
ls /etc/rc.d/ | grep -E 'dnsmasq|firewall|odhcpd' # all enabled
|
||||||
|
```
|
||||||
|
|
||||||
|
Then from a LAN client: renew DHCP (expect 192.168.0.100–249, DNS
|
||||||
|
`.254`), check internet, check the static leases land on their IPs.
|
||||||
|
|
||||||
|
## Known issue: 5 GHz `ath10k-ct`
|
||||||
|
|
||||||
|
The C7 v2/v5 has a long-standing 5 GHz issue with the default
|
||||||
|
`ath10k-ct` driver: `SWBA overrun on vdev 0` log spam plus client
|
||||||
|
drops under some loads. **Only if it shows up**, swap to the non-ct
|
||||||
|
driver (needs working WAN, or pre-download the same way as the RNDIS
|
||||||
|
apks):
|
||||||
|
|
||||||
|
```sh
|
||||||
|
apk del kmod-ath10k-ct ath10k-firmware-qca988x-ct
|
||||||
|
apk add kmod-ath10k ath10k-firmware-qca988x
|
||||||
|
reboot
|
||||||
|
```
|
||||||
|
|
||||||
|
## Recovery
|
||||||
|
|
||||||
|
If the flash bricks: C7 v5 has TFTP recovery — hold **Reset** while
|
||||||
|
powering on, router pulls `ArcherC7v5_tp_recovery.bin` (the OpenWrt
|
||||||
|
**factory** image renamed to that) from a TFTP server at
|
||||||
|
`192.168.0.66`. Factory image is in the same release directory as the
|
||||||
|
sysupgrade one.
|
||||||
|
|
||||||
|
## Post-upgrade cleanup
|
||||||
|
|
||||||
|
- Delete `/tmp` copies (automatic on reboot)
|
||||||
|
- Keep `~/code/perso/backup-archer-c7-20260819.tar.gz` until the new
|
||||||
|
setup has survived a reboot + a few days
|
||||||
|
- `~/code/perso/archer-c7-upgrade/` can go once the RNDIS drivers are
|
||||||
|
installed and Wi-Fi is confirmed stable (or keep the apks in case of
|
||||||
|
a re-flash)
|
||||||
@@ -112,6 +112,17 @@ Beyond databases, `outline`/`login` may hold non-DB state (uploaded
|
|||||||
files, docker volumes) — decide per instance whether an fs source is
|
files, docker volumes) — decide per instance whether an fs source is
|
||||||
needed in §4.
|
needed in §4.
|
||||||
|
|
||||||
|
⚠️ **seafile-fs excluded from the data leg (2026-08-24).** The S3
|
||||||
|
seed wedged for 12 h inside a single `lstat()` in seafile's block
|
||||||
|
store (millions of tiny files on the contended live pool). SIGQUIT
|
||||||
|
goroutine dump: main goroutine looping in `os.ignoringEINTR` around
|
||||||
|
`fstatat` — a Go preemption/EINTR **livelock** (100 % CPU, no
|
||||||
|
progress; dmesg clean, so not a kernel hang). `seafile-fs` is
|
||||||
|
commented out of `plakar-sources`; seafile's S3 coverage comes from
|
||||||
|
the incus leg instead, which reads the quiesced replica on the idle
|
||||||
|
`backup` pool. If a live-pool retry is ever wanted:
|
||||||
|
`GODEBUG=asyncpreemptoff=1` is the standard mitigation.
|
||||||
|
|
||||||
Findings from the first dry-run (2026-08-22):
|
Findings from the first dry-run (2026-08-22):
|
||||||
|
|
||||||
- `livetrail`'s `/root/.my.cnf` had a stale `database = spot` default
|
- `livetrail`'s `/root/.my.cnf` had a stale `database = spot` default
|
||||||
|
|||||||
@@ -0,0 +1,109 @@
|
|||||||
|
# seafile — GCID table loss, 6-month CPU burn (fixed 2026-08-22)
|
||||||
|
|
||||||
|
`seaf-server` in the `seafile` container sat at 40–47 % CPU for months
|
||||||
|
while appearing to do nothing: negligible client traffic on `eth0`
|
||||||
|
(344 MB sent over 102 days) but 82 GB and 385 M packets over loopback,
|
||||||
|
and ~11 **days** of accumulated CPU time.
|
||||||
|
|
||||||
|
## Root cause
|
||||||
|
|
||||||
|
The `seafile-db` MariaDB database was missing the `GCID` and
|
||||||
|
`LastGCID` tables. Since Seafile 11 every client sync operation checks
|
||||||
|
a per-repo GC id (`SELECT gc_id FROM GCID WHERE repo_id = ? FOR
|
||||||
|
UPDATE`); with the table gone the query failed, the client retried,
|
||||||
|
and the cycle repeated forever — no sync ever completed:
|
||||||
|
|
||||||
|
- 6 of seaf-server's fileserver worker threads spinning in pure
|
||||||
|
user-space, ~3 new MySQL connections/s (167 loopback `TIME_WAIT`
|
||||||
|
to :3306 at any moment),
|
||||||
|
- `seafile.log` grew to **1.8 GB** of the single warning
|
||||||
|
`Failed to prepare sql SELECT gc_id FROM GCID … Table
|
||||||
|
'seafile-db.GCID' doesn't exist`, several times per second.
|
||||||
|
|
||||||
|
First occurrence **2026-03-01 22:17** — under 11.0.9, i.e. *before*
|
||||||
|
both the 2026-03-16 restart and the 2026-05-11 upgrade to 12.0.14.
|
||||||
|
`GCID` is an old core table (the 6.3.0 upgrade SQL already ALTERs
|
||||||
|
it), so it existed and was lost. **What dropped it around 2026-03-01
|
||||||
|
was never identified** — if tables vanish again, start there.
|
||||||
|
|
||||||
|
## Diagnosis trail (what worked without container root)
|
||||||
|
|
||||||
|
From the host as `julien` (sudo grant covers read-only `incus`
|
||||||
|
subcommands only):
|
||||||
|
|
||||||
|
```sh
|
||||||
|
sudo incus info seafile # PID 2410, 1005 GiB, 2 GiB RAM
|
||||||
|
ps -eo pid,%cpu,args --sort=-%cpu # seaf-server pid 10217 top hog
|
||||||
|
# per-thread CPU from /proc (world-readable, no root needed):
|
||||||
|
for t in /proc/10217/task/*; do awk '{print $1, $14+$15}' $t/stat; done
|
||||||
|
# utime vs stime split → pure user-space compute, ~0 syscalls
|
||||||
|
# container netns TCP table via the process:
|
||||||
|
awk 'NR>1 && $4=="06"' /proc/10217/net/tcp | wc -l # 167 TIME_WAIT → :3306
|
||||||
|
```
|
||||||
|
|
||||||
|
Reading `/opt/seafile/logs/seafile.log` needed a temporary sudoers
|
||||||
|
extension (`/usr/bin/incus exec *`, `/usr/bin/incus file pull *` in a
|
||||||
|
drop-in — **removed again after the fix**). The log line then made the
|
||||||
|
cause obvious.
|
||||||
|
|
||||||
|
## Fix
|
||||||
|
|
||||||
|
The exact DDL ships inside the server binary:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
strings /opt/seafile/seafile-server-12.0.14/seafile/bin/seaf-server \
|
||||||
|
| grep 'CREATE TABLE IF NOT EXISTS.*GCID'
|
||||||
|
```
|
||||||
|
|
||||||
|
Applied (root in the container; additive, no restart needed):
|
||||||
|
|
||||||
|
```sql
|
||||||
|
CREATE TABLE IF NOT EXISTS GCID (
|
||||||
|
id BIGINT NOT NULL PRIMARY KEY AUTO_INCREMENT,
|
||||||
|
repo_id CHAR(36), gc_id CHAR(36),
|
||||||
|
UNIQUE INDEX(repo_id)) ENGINE=INNODB;
|
||||||
|
CREATE TABLE IF NOT EXISTS LastGCID (
|
||||||
|
id BIGINT NOT NULL PRIMARY KEY AUTO_INCREMENT,
|
||||||
|
repo_id CHAR(36), client_id VARCHAR(128), gc_id CHAR(36),
|
||||||
|
UNIQUE INDEX(repo_id, client_id)) ENGINE=INNODB;
|
||||||
|
```
|
||||||
|
|
||||||
|
The warning storm stopped the same second, seaf-server dropped to
|
||||||
|
0 % CPU, host load fell from ~8–12 toward normal. `seafile.log` was
|
||||||
|
truncated (`truncate -s 0`).
|
||||||
|
|
||||||
|
## Log rotation (added 2026-08-22)
|
||||||
|
|
||||||
|
Seafile ships **no rotation** — that's how one warning reached 1.8 GB
|
||||||
|
unnoticed. `/etc/logrotate.d/seafile` in the container:
|
||||||
|
|
||||||
|
```
|
||||||
|
/opt/seafile/logs/*.log {
|
||||||
|
weekly
|
||||||
|
maxsize 100M
|
||||||
|
rotate 8
|
||||||
|
missingok
|
||||||
|
notifempty
|
||||||
|
compress
|
||||||
|
delaycompress
|
||||||
|
copytruncate
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
`copytruncate` avoids pid/SIGUSR1 handling for the various daemons
|
||||||
|
(seaf-server, seahub, seafevents…) at the cost of possibly losing a
|
||||||
|
few lines at rotation — fine here. `maxsize 100M` caps a future spam
|
||||||
|
storm at the next daily logrotate run instead of next week.
|
||||||
|
|
||||||
|
## Related cleanup / leftovers
|
||||||
|
|
||||||
|
- Legacy 4 AM cron on ks4 rsyncing seafile's rootfs to old **ks2**
|
||||||
|
(`164.132.173.57`) removed 2026-08-22 — redundant with the
|
||||||
|
[incus-copy](incus-copy.md) legs and it burned another ~47 % CPU
|
||||||
|
per run.
|
||||||
|
- Still missing from `seafile-db` (harmless, nothing queries them
|
||||||
|
unless per-folder permissions are used): `FolderUserPerm`,
|
||||||
|
`FolderGroupPerm`, `FolderPermTimestamp`. Same `strings`/upgrade-SQL
|
||||||
|
trick provides the DDL if ever needed.
|
||||||
|
- Host swap (1 GB) was 100 % full during the incident; unrelated but
|
||||||
|
worth keeping an eye on.
|
||||||
Reference in New Issue
Block a user