doc/ks4/restic-backup.md

# restic backups on ks4 — the S3 leg

Status: **live since 2026-08-28.** This is the S3 backup leg of the
[strategy](../backup-strategy.md): database dumps + selected
filesystem trees into the `restic-data` bucket, nightly at 05:00.
Instances themselves are handled by replication (sdb, nuc after FTTH),
not by this leg — an instance-filesystem leg (`restic-incus`, §6) was
written and **shelved**; its script and empty repo stay in place
should that change.

Why restic: its comparison state lives in a **local cache**, so a
night costs a local stat-walk plus the churn — the rsync cost model,
with dedup, compression and encryption on top. (The predecessor and
the measurements that motivated the switch are in
[plakar-s3-data.md](plakar-s3-data.md) — reference only.)

## 1. Repositories — two buckets, two drivers (decided 2026-08-25)

| bucket = repo | contents | driver script |
|---|---|---|
| `restic-data` | SQL dumps + **all** fs sources (8 small + nextcloud + seafile) | `restic-backup.sh` |
| `restic-incus` | *(shelved — empty repo, see §6)* the `backup`-project replicas, per-file via mount | `restic-incus-backup.sh` |

Two repos so each can be pruned and checked independently. Accepted
trade-off: `restic-data` carries the giants' blobs (~3–5 M once
seafile is in), so **every operation on it loads a ~1.5–2 GiB
index** — mitigated by running the whole nightly as a *single*
`restic backup` invocation (one index load, one snapshot covering
all paths), `GOGC=20` in reserve, rustic as the escape hatch.
Buckets (not prefixes) for per-bucket OVH metrics and independent
versioning/lock decisions later.

Passphrase: `/root/.restic-passphrase` (mode 600), one for all repos,
**copy to the password manager immediately**. S3 creds + env in
`/root/.restic-env` (mode 600).

## 2. Option analysis (our numbers, not defaults-worship)

### `--pack-size` (default 16 MiB, max 128)

Packs are the S3 objects; blobs (~1 MiB avg for data) live inside
them. Pack size does **not** change index RAM (that scales with blob
count) — it changes object count, request count, and rewrite
amplification:

| pack-size | S3 objects for ~1.2 T stored | trade-off |
|---|---|---|
| 16 MiB | ~75 000 | slow `check`/listing, more requests; prune rewrites are fine-grained |
| **64 MiB (chosen)** | ~19 000 | 4× fewer objects/requests; assembly RAM ≈ pack×connections ≈ 64 M × 5 ≈ 320 MiB — fine |
| 128 MiB | ~9 500 | fewer still, but prune rewrite amplification doubles and upload buffers grow |

`--pack-size 64` on **every backup/prune invocation** (not a repo
property).

### Memory (the real constraint on this 23/31 GiB-used box)

- Index RAM ≈ 200–300 B/blob → combined `restic-data` index
  ~1 GiB per operation; prune peaks 2–3×. The seafile side of that
  estimate collapsed after its garbage collection (24.9 M → 0.5 M
  objects, [seafile-gc.md](seafile-gc.md)). Measure peak RSS with
  `/usr/bin/time -v` if the box ever feels tight.
- `GOGC=20` trades CPU for a smaller Go heap — enable if the seafile
  prune ever pressures the box.

### Cache location

`~/.cache/restic` would land on `/` (14 G free) and can reach a few
GiB at this scale → `--cache-dir /backup/restic-cache` (sdb pool;
cache reads are ARC-warm after first touch). tmpfs was considered and
rejected: the cache exists to avoid *network* metadata re-fetches;
tmpfs loses it at reboot and eats the RAM the ARC needs.

### Concurrency & compression

- `--read-concurrency 8` (default 2): more outstanding reads lets the
  HDD elevator help on seek-bound trees (our 275-IOPS wall).
- `-o s3.connections=8` (default 5): mild upload parallelism bump.
- Compression: repo-v2 default `auto` is right.

### Excludes — the cheapest optimisation of all

Nextcloud's `appdata_*/preview` (and `dav-photocache`) are
regenerable caches holding a large fraction of the 500k files:

```
# /root/scripts/restic-exclude
/var/lib/incus/storage-pools/data/containers/nextcloud/rootfs/nextcloud/data/appdata_*/preview
/var/lib/incus/storage-pools/data/containers/nextcloud/rootfs/nextcloud/data/appdata_*/dav-photocache
```

Verify before the seed: `find .../data/appdata_* -path '*/preview/*' | wc -l`
for the win size, then `restic backup --dry-run -vv --exclude-file …`
prints every decision without uploading. Per-user `<user>/cache/`
dirs are a further candidate. Seafile: no excludes — the block store
*is* the data.

### Escape hatch: rustic

[rustic](https://github.com/rustic-rs/rustic) speaks the same
repository format — switching later is a binary swap on the same
repos (decided 2026-08-25: restic first, rustic if performance
issues rise; manual upgrades accepted). It would bring lock-free
prune and a lower RAM footprint. Don't run both concurrently on one
repo (rustic ignores restic's locks); a clean switchover is fine.

## 3. Install + init (root on ks4)

Debian 13 ships restic **0.18.0**, which misses fixes we specifically
need (checked 2026-08-25 against upstream changelogs): the 0.18.1
crash fix for directories deleted mid-walk (#5421 — our live trees do
that nightly), 0.19.0's much faster index loading (#5713 — the merged
restic-data repo loads its index every op) and lower check memory
(#5610), index-repair robustness (#21827/28), clean exit codes
(#4467/#5363 — our drivers key off rc), and 0.19.1's
skip-inaccessible-source fix (#5667 — an unmounted source must not
commit an empty "successful" snapshot). → install upstream **0.19.1**;
future upgrades via `restic self-update` (signature-verified).

```sh
apt install sshfs bzip2       # sshfs: needed by `incus file mount` (§6)
curl -LO https://github.com/restic/restic/releases/download/v0.19.1/restic_0.19.1_linux_amd64.bz2
bunzip2 restic_0.19.1_linux_amd64.bz2
install -m755 restic_0.19.1_linux_amd64 /usr/local/bin/restic
restic version
head -c 32 /dev/urandom | base64 > /root/.restic-passphrase && chmod 600 /root/.restic-passphrase
# ⚠️ password manager, NOW.

cat > /root/.restic-env <<'EOF'
export AWS_ACCESS_KEY_ID=<AK>
export AWS_SECRET_ACCESS_KEY=<SK>
export RESTIC_PASSWORD_FILE=/root/.restic-passphrase
export RESTIC_CACHE_DIR=/backup/restic-cache
EOF
chmod 600 /root/.restic-env && . /root/.restic-env && mkdir -p /backup/restic-cache

for r in restic-data restic-incus; do
    restic -r s3:s3.sbg.io.cloud.ovh.net/$r init
done
```

## 4. Seed plan (executed 2026-08-25 → 08-28)

1. **ARC floor first** (helps every walk on the box):
   ```sh
   echo 8589934592 > /sys/module/zfs/parameters/zfs_arc_min
   echo "options zfs zfs_arc_min=8589934592" > /etc/modprobe.d/zfs-arc.conf
   update-initramfs -u
   awk '/^c_min/ {printf "%.1f GiB\n", $3/1073741824}' /proc/spl/kstat/zfs/arcstats
   ```
   ⚠️ the floor is defended even under pressure — real headroom is
   ~31 G − services − 8 G; revisit before adding a fat service.
2. `restic-data`, staged (all into the same repo; separate
   invocations so each stage is independently restartable —
   `restic-backup.sh -s` runs the same thing):
   - dumps + 8 small sources (~50 G — an hour)
   - nextcloud path with the exclude file (proven ~22 MiB/s from the
     live pool → ~8 h, screen)
   - seafile — the one-time debt: 1–2 days, restartable at any point
     (committed packs dedup on retry)
3. `restic-incus` (16 small replicas via §6 — a few hours).

## 5. Nightly driver — `scripts/restic-backup.sh`

Shape: flock, env file, loud logging. Phases:

1. **dumps** — incus DB
   dumps + auto-discovered MariaDB/PostgreSQL (native and docker)
   dumps into the dump dir (during the parallel week restic dumps
   into its own `/backup/dumps`; carry the archived `geo.sql` over
   once with `cp -a`).
2. **one backup invocation** into `restic-data`: dump dir + all fs
   paths from `/root/scripts/restic-paths`
   (`--files-from-verbatim`), nextcloud excludes applied globally
   (patterns only match nextcloud paths). One index load, one
   snapshot per night.
3. **retention** — `restic forget --group-by host --keep-daily 14
   --keep-weekly 8 --keep-monthly 6` (instant without `--prune`;
   `--group-by host` because seed-era snapshots have different path
   sets than nightly ones and must age in one group).

The instances leg (`restic-incus-backup.sh`, §6) is shelved — nothing
chains onto the 01:00 line today.

```cron
0 5 * * * /root/scripts/restic-backup.sh >> /var/log/restic-backup.log 2>&1
```

⚠️ cron gotcha (bit us 2026-08-28): cron's PATH lacks `/usr/local/bin`,
so the scripts call `/usr/local/bin/restic` by absolute path.

```cron
```

Weekly maintenance (Sunday; prune takes an **exclusive lock** — never
overlap the 05:00 run):

```cron
0 14 * * 0 /root/scripts/restic-maintenance.sh >> /var/log/restic-maintenance.log 2>&1
```

Per repo: `restic prune --max-unused 10% --max-repack-size 4G
--pack-size 64` — our dead-data rate is ~0.1–0.4 GiB/day against
~700 GiB repos (seafile append-mostly ≈ nothing dies), so 10 %
tolerated slack lets prune skip repacking for months, and the 4 G cap
bounds any single Sunday to ~10–15 min even after a mass deletion.
Then `restic check` (structure, cheap) + `check
--read-data-subset=1/52` rotating — full data verification of every
byte once a year, ~25 GiB read per week.

## 6. Instances without a plugin — `incus file mount` (SHELVED)

> Decided 2026-08-28: **not deployed.** Instances are replicated (sdb,
> nuc after FTTH, ks2 in the meantime) and everything needed to
> rebuild one is in `restic-data`, so a third copy of their
> filesystems was judged not worth the nightly moving parts. The
> design and the script remain, ready if that changes.
> If the `restic-incus` bucket is ever deleted, drop it from `REPOS`
> in `restic-maintenance.sh` too.

Neither restic nor rustic accepts source plugins, and forking the
backup tool is the wrong place to carry a patch. Instead, incus
exposes any instance (including **stopped replicas**) as a FUSE mount
over the per-instance sftp API:

```sh
for inst in $(incus list --project backup -c n -f csv); do
    case ",$EXCLUDE_INSTANCES," in *",$inst,"*) continue;; esac   # nextcloud,seafile
    mnt=/run/restic-incus/$inst          # per-instance mountpoint: restic
    mkdir -p "$mnt"                      # picks parents by host+path, so
    incus config show "$inst" --project backup --expanded > "$mnt.yaml"
    incus file mount "$inst/" "$mnt" --project backup &   # foreground proc
    MPID=$!; sleep 2
    restic -r s3:…/restic-incus backup --pack-size 64 --tag "$inst" "$mnt" "$mnt.yaml"
    kill $MPID; wait $MPID 2>/dev/null; umount "$mnt" 2>/dev/null
done
```

⚠️ **per-instance mountpoints are load-bearing**: a shared mountpoint
would make every snapshot's parent the previous *other* instance's
snapshot (restic matches parents by host+path) → nightly full
re-reads. Distinct stable paths give correct parents and stat-skip.
Mount lifecycle hardened in `restic-incus-backup.sh` (`incus file
mount` runs foreground; needs `sshfs`). All instances by default,
opt-out via a variable — the no-manifest-drift rule survives the tool
change. Chained after the 01:00 incus-copy once validated, same
freshness reasoning as before.

One shared repo would dedup the Ubuntu base across all replicas. Restore granularity: per-file; instance definitions ride
the dump tree (`incus/<inst>.yaml` + `incus-global-db.sql`).

## 7. Restore drill (gate, then quarterly)

Scripted — [`restic-restore-test.sh`](https://git.lutran.fr/julien/scripts/src/branch/main/restic-restore-test.sh):

```sh
/root/scripts/restic-restore-test.sh        # tree + dump, ~2 min
/root/scripts/restic-restore-test.sh -d     # also loads the dump into a
                                            # throwaway container (strongest)
```

It restores a filesystem tree from the latest snapshot and diffs it
against the live one, restores a database dump and checks it is
well-formed (`CREATE TABLE` count + `Dump completed` marker), then
cleans up (`-k` keeps the files, `-w` changes the work dir). Never
restore into `/tmp` — it is a 16 GiB tmpfs on ks4; the script writes
under `/backup`.

A backup leg that has never been restored from is a hypothesis, not a
backup: run this at the cutover gate and once a quarter after.

## 8. Cutover (done) + what is left

Legacy on the box: plakar is still installed and its `plakar-data`
bucket still exists, kept only to reproduce
[#2338](https://github.com/PlakarKorp/plakar/issues/2338) if upstream
asks; no cron references it. The empty `plakar-incus` bucket can be
deleted.

- **Cutover 2026-08-28**: the previous S3 leg's crons are off and
  restic's 05:00 run is the S3 backup. The formal parallel-week
  comparison was cut short — the evidence had already settled it
  (predecessor: ~800 MiB metadata re-read and 3–4 h per night on the
  nextcloud tree, churn-independent; restic: seed at 60–115 MiB/s,
  local-cache incrementals). Details in
  [plakar-s3-data.md](plakar-s3-data.md).
- **Done 2026-08-28**: initial sync (1,114,212 files / 1.239 TiB in
  4h10m, only 2.4 GiB added thanks to dedup + the seafile GC), restore
  drill passed (§7), interim ks2 push re-enabled.
- **Steady state, measured 2026-08-29** (first full nightly run):
  **1,114,343 files / 1.239 TiB walked in 1 min 43 s**, 201 MiB added
  (63 MiB stored), `rc=0`. For comparison, the previous S3 tool needed
  3–4 h per night for *one* of those trees. This is the local-cache
  cost model working as intended: the walk is stat-only, and only the
  churn is read, chunked and uploaded.
- **First maintenance run 2026-08-30**: `prune` 0 blobs / 0 B removed,
  `check` and the rotating `--read-data-subset` slice both clean, 7 min,
  `rc=0`. Nothing to reclaim because the killed seed's packs were
  *adopted* by the successful seed's dedup rather than orphaned — worth
  knowing before assuming an interrupted backup wastes storage.
- **Left**: the nuc leg after FTTH, then the ks2 decommission
  ([../ks2/plan.md](../ks2/plan.md)). The instance leg (§6) is
  shelved, not pending.
- **Old tool**: see the legacy note above; the `integration-incus`
  repo retires from production (optionally still published to
  PlakarKorp/hub as a community contribution).
- **Interim ks2 push** (until the nuc leg seeds after FTTH, expected
  before end of September): re-enable the replica push so instances
  keep an off-site copy meanwhile:
  ```cron
  0 2 * * * /root/scripts/incus-copy.sh -d ks2 -m push >> /var/log/incus-copy.log 2>&1
  ```