diff --git a/nas/nas-install.md b/nas/nas-install.md index 37d8fc8..bd3665e 100644 --- a/nas/nas-install.md +++ b/nas/nas-install.md @@ -615,43 +615,26 @@ The driver is unchanged — [`incus-copy.sh`](https://git.lutran.fr/julien/scripts/src/branch/main/incus-copy.sh), deployed to `/root/scripts` as everywhere else. -### 9a. nuc's instances -> nas (systemd timer, not cron) +### 9a. nuc's instances -> nas -⚠️ **nuc is an on-demand media box** (see [nuc/README.md](../nuc/README.md)): -since 2026-08-30 it only runs when watching Jellyfin or using the Spotify -kiosk, so it is usually powered off at 03:30. Cron cannot catch up a -missed window; a **systemd timer with `Persistent=true`** runs the job -shortly after the next boot. That is the whole reason this is a timer. +Root crontab on nuc: -`/etc/systemd/system/incus-copy.service` on nuc: - -```ini -[Unit] -Description=Replicate all nuc instances to nas (project backup, pool nucbackup) -After=network-online.target incus.service -Wants=network-online.target -Requires=incus.service - -[Service] -Type=oneshot -ExecStart=/bin/sh -c "/root/scripts/incus-copy.sh -d nas -m push -s nucbackup -p backup >> /var/log/incus-copy.log 2>&1" +```cron +30 3 * * * /root/scripts/incus-copy.sh -d nas -m push -s nucbackup -p backup >> /var/log/incus-copy.log 2>&1 ``` -`/etc/systemd/system/incus-copy.timer`: - -```ini -[Timer] -OnCalendar=03:30 -Persistent=true -RandomizedDelaySec=300 - -[Install] -WantedBy=timers.target -``` - -```sh -systemctl enable --now incus-copy.timer && systemctl list-timers incus-copy -``` +⚠️ **nuc is an on-demand media box** (see +[nuc/README.md](../nuc/README.md)): since 2026-08-30 it only runs when +watching Jellyfin or using the Spotify kiosk, so it is often powered +off at 03:30 and **that night's push is simply skipped** — cron does +not catch up missed windows. Accepted deliberately (2026-08-31): +nuc's instances change rarely and the next time it is up the refresh +is incremental anyway. +This ran briefly as a `systemd` timer with `Persistent=true` (which +*does* catch up after boot); the units are still on disk, disabled, at +`/etc/systemd/system/incus-copy.{service,timer}` if that behaviour is +ever wanted back: +`systemctl enable --now incus-copy.timer` (and remove the cron line). ⚠️ **`-p backup` is not optional.** Without it the replicas land in `default` on nas and collide with nas's *live* instances — both hosts are @@ -674,7 +657,7 @@ incus storage create nasbackup zfs source=tank/backup/nas ``` ```cron -# nas, /etc/cron.d/incus-copy +# nas, root crontab 0 4 * * * root /root/scripts/incus-copy.sh -p backup -s nasbackup >> /var/log/incus-copy.log 2>&1 ``` @@ -742,7 +725,7 @@ Then test-restore one instance before ticking the gate in | When | Host | What | |---|---|---| | 03:00 | nuc, nas | instance snapshots (profile) | -| 03:30 | nuc | push all instances → `nas:nucbackup` (**systemd timer**, `Persistent=true` — catches up after boot) | +| 03:30 | nuc | push all instances → `nas:nucbackup` (root crontab; skipped when nuc is off) | | 04:00 | nas | local copy of nas instances → `nasbackup` (`tank/backup/nas`) | | 05:00 | nas | pull `ks4:*` → `ks4backup` (after FTTH) | | 05:30 | nuc | apt upgrade all containers | @@ -779,7 +762,7 @@ host had no MTA). `ZED_NOTIFY_VERBOSE=1`, **`ZED_NOTIFY_DATA=1`**, `ZED_NOTIFY_INTERVAL_SECS=3600`. - **[`zpool-health.sh`](https://git.lutran.fr/julien/scripts/src/branch/main/zpool-health.sh)** - every 15 min via `/etc/cron.d/zpool-health` — zed does *not* report a + every 15 min from root's crontab — zed does *not* report a suspended pool (the vdev stays `ONLINE`, so `statechange-notify.sh` never fires). That watchdog is the only thing that catches the exact failure this box was built to prevent. diff --git a/nuc/nuc-install.md b/nuc/nuc-install.md index 1c53293..9db0838 100644 --- a/nuc/nuc-install.md +++ b/nuc/nuc-install.md @@ -31,7 +31,7 @@ incus project create backup -c features.images=false -c features.profiles=false /root/scripts/incus-copy.sh -p backup -s nucbackup ``` -Runs nightly via `/etc/cron.d/incus-copy` at **03:30** (30 min after +Runs nightly from root's crontab at **03:30** (30 min after the profile-scheduled 03:00 instance snapshots, so VM refreshes stay incremental), logging to `/var/log/incus-copy.log` (logrotate: `/etc/logrotate.d/incus-copy`). Note the script's `flock` is global: @@ -245,8 +245,8 @@ once `usb4t` is imported.) | When | What | Where | |-------|------|-------| | 03:00 | instance snapshots (`snapshots.schedule` on the default profile, expiry 7d) | incus | -| 03:30 | replicate all instances to the USB pool (`incus-copy.sh -p backup -s nucbackup`) | `/etc/cron.d/incus-copy` → `/var/log/incus-copy.log` | -| 05:00 | apt dist-upgrade all running containers (`incus-container-upgrade.sh`; VMs and non-apt containers skipped; jellyfin pinned to the 10.11 series in-container) | `/etc/cron.d/incus-container-upgrade` → `/var/log/incus-container-upgrade.log` | +| 03:30 | replicate all instances to the USB pool (`incus-copy.sh -p backup -s nucbackup`) | root crontab → `/var/log/incus-copy.log` | +| 05:00 | apt dist-upgrade all running containers (`incus-container-upgrade.sh`; VMs and non-apt containers skipped; jellyfin pinned to the 10.11 series in-container) | root crontab → `/var/log/incus-container-upgrade.log` | The ordering is deliberate: snapshot → backup → upgrade, so a broken upgrade is always one snapshot-restore away and the replicas predate it. diff --git a/nuc/usb4t-dropouts.md b/nuc/usb4t-dropouts.md index d8211bd..1addd13 100644 --- a/nuc/usb4t-dropouts.md +++ b/nuc/usb4t-dropouts.md @@ -10,7 +10,7 @@ > host. Symptom seen first in the nightly backup log -(`/var/log/incus-copy.log`, job in `/etc/cron.d/incus-copy`): every +(`/var/log/incus-copy.log`, job in root's crontab): every instance fails with ``` @@ -140,7 +140,7 @@ before). Two reasons: Fixed 2026-08-30 by adding [`zpool-health.sh`](https://git.lutran.fr/julien/scripts/src/branch/main/zpool-health.sh), -run every 15 min from `/etc/cron.d/zpool-health`. It mails only on +run every 15 min from root's crontab. It mails only on `healthy <-> problem` **transitions**, so it is silent in normal operation and cannot spam; `-t` sends a test. Worth deploying on ks4 too — its `data` pool is single-disk and has the same blind spot.