doc: add backup-strategy.md and new-container.md for non-specialist readers
backup-strategy.md: the two-tool architecture (incus copy + restic), ASCII map of the legs (ks4 sdb / nuc pull / two S3 buckets), the schedule table, a 30-second health check, and restore recipes for a file, a database and a whole instance. new-container.md: conventions for adding a service — one container per service, no Docker (incus runs OCI natively, incus-compose for stacks), gateway/proxy ingress, leave the snapshot schedule alone, what the backups pick up automatically vs the one line to add to restic-paths, DB-discovery requirements (incl. the .my.cnf trap), a verification run, and the pitfalls (tmpfs /tmp, tiny-file swarms, cron PATH). README: start-here links, ks4 durability bullet and flow chart updated to the restic era. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Julien Lutran
co-authored by
Claude Fable 5
Claude Fable 5
parent
33b2a87d8f
commit
c18c8edc5b
@@ -0,0 +1,144 @@
|
||||
# Creating a new service on ks4 — conventions
|
||||
|
||||
Audience: you have root on ks4 and want to add a service. Follow this
|
||||
and the service is automatically replicated, backed up and
|
||||
restorable, without touching the backup machinery.
|
||||
Companion page: [backup-strategy.md](backup-strategy.md).
|
||||
|
||||
## 1. One service = one incus container
|
||||
|
||||
```sh
|
||||
incus launch images:debian/13 myservice
|
||||
incus config device add myservice eth0 nic nictype=bridged parent=incusbr0 \
|
||||
ipv4.address=192.168.1.30 # pick a free IP, note it in the doc
|
||||
incus exec myservice -- bash # install and configure from here
|
||||
```
|
||||
|
||||
Rules of thumb:
|
||||
|
||||
- **Debian 13 or Ubuntu 24.04 images**, unprivileged (the default).
|
||||
Privileged only with a written reason (`nuc/jellyfin-client.md` is
|
||||
the one example: it needs raw device access).
|
||||
- **Configure through `incus exec` and write the commands down** in
|
||||
`doc/ks4/<service>.md` as you go. The doc *is* the rebuild
|
||||
procedure — that is the whole convention: re-running it recreates
|
||||
the service.
|
||||
- The container is on the NAT bridge `incusbr0` (192.168.1.0/24). It
|
||||
is **not** reachable from the internet by itself.
|
||||
- **Web services**: do not open ports. Add an nginx vhost in the
|
||||
`gateway` container that proxies to `http://192.168.1.30:PORT`;
|
||||
gateway is the single HTTP/S entry point and handles TLS.
|
||||
Non-HTTP services (mail, wireguard) use an incus `proxy` device —
|
||||
copy an existing one as a model.
|
||||
|
||||
## 2. No Docker inside the container
|
||||
|
||||
Docker-in-incus works (`login`/`outline` predate this rule) but costs
|
||||
a second layer of networking, storage and updates, and hides the
|
||||
service's data behind docker volumes.
|
||||
|
||||
Since incus can run **OCI images natively**, an upstream
|
||||
`docker-compose.yml` becomes incus instances:
|
||||
|
||||
```sh
|
||||
incus remote add docker https://docker.io --protocol=oci # once per host
|
||||
incus launch docker:library/redis myservice-redis
|
||||
```
|
||||
|
||||
and multi-container stacks are described with
|
||||
[`incus-compose`](https://github.com/lxc/incus-compose) (upstream,
|
||||
compose-file syntax → incus instances). Each piece is then a normal
|
||||
instance: same network, same snapshots, same backups, `incus exec` to
|
||||
debug, no nested runtime.
|
||||
|
||||
If you truly need Docker (upstream ships only a compose stack you
|
||||
don't want to translate), say so in the service doc and note where
|
||||
the volumes live — you will need them in step 4.
|
||||
|
||||
## 3. Snapshots (do nothing, but know why)
|
||||
|
||||
The `default` profile gives every instance `snapshots.schedule =
|
||||
0 3 * * *`, `snapshots.expiry = 7d`. Leave it. Those snapshots are
|
||||
what make the nightly replication incremental — an instance without
|
||||
them forces a full re-send of its whole disk every night.
|
||||
|
||||
## 4. Wire it into the backups
|
||||
|
||||
**Automatic, nothing to do:**
|
||||
|
||||
- the 01:00 replica leg copies *every* instance to the `backup` pool
|
||||
and to nuc — new instances included;
|
||||
- the S3 instance leg backs up every replica the same way;
|
||||
- **databases are auto-discovered**: any running container with
|
||||
MariaDB/MySQL or PostgreSQL gets every non-system database dumped
|
||||
nightly, plus users/grants.
|
||||
|
||||
**Manual, one line:** data that lives in the filesystem (uploads,
|
||||
repositories, mail spools…) must be listed in
|
||||
`/root/scripts/restic-paths`, one absolute host path per line:
|
||||
|
||||
```
|
||||
/var/lib/incus/storage-pools/data/containers/myservice/rootfs/var/lib/myservice
|
||||
```
|
||||
|
||||
(The container's `/x` is `…/containers/<name>/rootfs/x` on the host.)
|
||||
No comments in that file — every line is read as a path. Regenerable
|
||||
caches can be skipped by adding a pattern to
|
||||
`/root/scripts/restic-exclude`. Both files live in the `scripts` git
|
||||
repo: edit there, commit, `git pull` on ks4.
|
||||
|
||||
For the database auto-discovery to work, keep the defaults:
|
||||
|
||||
- MariaDB/MySQL: client **and** dump binaries installed, root access
|
||||
over the local unix socket. **Do not leave a `database = …` line in
|
||||
`/root/.my.cnf`** — it breaks `SHOW DATABASES` and the container is
|
||||
then skipped (this happened once and went unnoticed for months).
|
||||
- PostgreSQL: reachable as the `postgres` system user (peer auth).
|
||||
- PostgreSQL in Docker: the container's image name must contain
|
||||
"postgres" (official images do).
|
||||
|
||||
A database you do *not* want dumped (huge, static, re-importable)
|
||||
goes into `/root/scripts/plakar-db-exclude` as `<instance>/<db>` —
|
||||
every skip is logged, so the list cannot rot silently.
|
||||
|
||||
## 5. Verify, once
|
||||
|
||||
```sh
|
||||
cd /root/scripts && git pull
|
||||
/root/scripts/restic-backup.sh -s dumps # ~2 min: see your DB in the log
|
||||
ls /backup/dumps/mariadb/myservice/ # dump present?
|
||||
/root/scripts/restic-backup.sh # full run, then:
|
||||
. /root/.restic-env
|
||||
restic -r s3:s3.sbg.io.cloud.ovh.net/restic-data ls latest | grep myservice | head
|
||||
```
|
||||
|
||||
Seeing your paths and dumps in that listing means the service is
|
||||
protected. Do this the day you create the service, not the day you
|
||||
need it.
|
||||
|
||||
## 6. Pitfalls (learned the hard way)
|
||||
|
||||
- **`/tmp` on ks4 is a 16 GiB tmpfs — it is RAM.** Never restore, dump
|
||||
or stage data there; use `/backup/…`.
|
||||
- **Avoid millions of tiny files.** A service that keeps every version
|
||||
of everything (seafile did, 24.9 M objects) makes every backup and
|
||||
every filesystem walk take days. If the upstream software has a
|
||||
garbage-collection or retention setting, turn it on *and schedule
|
||||
it* the day you deploy — see [ks4/seafile-gc.md](ks4/seafile-gc.md).
|
||||
- **Keep data under one predictable path** per service; scattered data
|
||||
means several `restic-paths` lines and things get forgotten.
|
||||
- **Static IP + a note in the service doc**; two containers fighting
|
||||
over one IP is a confusing outage.
|
||||
- Cron on ks4 does not have `/usr/local/bin` in its `PATH` — use
|
||||
absolute paths in anything you schedule.
|
||||
|
||||
## 7. Checklist
|
||||
|
||||
- [ ] container created from a Debian/Ubuntu image, static IP noted
|
||||
- [ ] no Docker (or a written reason + volume paths)
|
||||
- [ ] install/config steps written in `doc/ks4/<service>.md`
|
||||
- [ ] exposed through gateway (HTTP) or a proxy device (other)
|
||||
- [ ] snapshots left at the profile default
|
||||
- [ ] data paths added to `restic-paths` (committed to git)
|
||||
- [ ] `restic-backup.sh` run once; dumps and paths verified in the repo
|
||||
- [ ] service added to the table in [ks4/install.md](ks4/install.md)
|
||||
Reference in New Issue
Block a user