# Creating a new service on ks4 — conventions

Audience: you have root on ks4 and want to add a service. Follow this
and the service is automatically replicated, backed up and
restorable, without touching the backup machinery.
Companion page: [backup-strategy.md](backup-strategy.md).

## 1. One service = one incus container

```sh
incus launch images:debian/13 myservice
incus config device add myservice eth0 nic nictype=bridged parent=incusbr0 \
    ipv4.address=192.168.1.30            # pick a free IP, note it in the doc
incus exec myservice -- bash             # install and configure from here
```

Rules of thumb:

- **Debian 13 or Ubuntu 24.04 images**, unprivileged (the default).
  Privileged only with a written reason (`nuc/jellyfin-client.md` is
  the one example: it needs raw device access).
- **Configure through `incus exec` and write the commands down** in
  `doc/ks4/<service>.md` as you go. The doc *is* the rebuild
  procedure — that is the whole convention: re-running it recreates
  the service.
- The container is on the NAT bridge `incusbr0` (192.168.1.0/24). It
  is **not** reachable from the internet by itself.
- **Web services**: do not open ports. Add an nginx vhost in the
  `gateway` container that proxies to `http://192.168.1.30:PORT`;
  gateway is the single HTTP/S entry point and handles TLS.
  Non-HTTP services (mail, wireguard) use an incus `proxy` device —
  copy an existing one as a model.

## 2. No Docker inside the container

Docker-in-incus works (`login`/`outline` predate this rule) but costs
a second layer of networking, storage and updates, and hides the
service's data behind docker volumes.

Since incus can run **OCI images natively**, an upstream
`docker-compose.yml` becomes incus instances:

```sh
incus remote add docker https://docker.io --protocol=oci    # once per host
incus launch docker:library/redis myservice-redis
```

and multi-container stacks are described with
[`incus-compose`](https://github.com/lxc/incus-compose) (upstream,
compose-file syntax → incus instances). Each piece is then a normal
instance: same network, same snapshots, same backups, `incus exec` to
debug, no nested runtime.

If you truly need Docker (upstream ships only a compose stack you
don't want to translate), say so in the service doc and note where
the volumes live — you will need them in step 4.

## 3. Snapshots (do nothing, but know why)

The `default` profile gives every instance `snapshots.schedule =
0 3 * * *`, `snapshots.expiry = 7d`. Leave it. Those snapshots are
what make the nightly replication incremental — an instance without
them forces a full re-send of its whole disk every night.

## 4. Wire it into the backups

**Automatic, nothing to do:**

- the 01:00 replica leg copies *every* instance to the `backup` pool
  and to nuc — new instances included;
- the S3 instance leg backs up every replica the same way;
- **databases are auto-discovered**: any running container with
  MariaDB/MySQL or PostgreSQL gets every non-system database dumped
  nightly, plus users/grants.

**Manual, one line:** data that lives in the filesystem (uploads,
repositories, mail spools…) must be listed in
`/root/scripts/restic-paths`, one absolute host path per line:

```
/var/lib/incus/storage-pools/data/containers/myservice/rootfs/var/lib/myservice
```

(The container's `/x` is `…/containers/<name>/rootfs/x` on the host.)
No comments in that file — every line is read as a path. Regenerable
caches can be skipped by adding a pattern to
`/root/scripts/restic-exclude`. Both files live in the `scripts` git
repo: edit there, commit, `git pull` on ks4.

For the database auto-discovery to work, keep the defaults:

- MariaDB/MySQL: client **and** dump binaries installed, root access
  over the local unix socket. **Do not leave a `database = …` line in
  `/root/.my.cnf`** — it breaks `SHOW DATABASES` and the container is
  then skipped (this happened once and went unnoticed for months).
- PostgreSQL: reachable as the `postgres` system user (peer auth).
- PostgreSQL in Docker: the container's image name must contain
  "postgres" (official images do).

A database you do *not* want dumped (huge, static, re-importable)
goes into `/root/scripts/plakar-db-exclude` as `<instance>/<db>` —
every skip is logged, so the list cannot rot silently.

## 5. Verify, once

```sh
cd /root/scripts && git pull
/root/scripts/restic-backup.sh -s dumps        # ~2 min: see your DB in the log
ls /backup/dumps/mariadb/myservice/            # dump present?
/root/scripts/restic-backup.sh                 # full run, then:
. /root/.restic-env
restic -r s3:s3.sbg.io.cloud.ovh.net/restic-data ls latest | grep myservice | head
```

Seeing your paths and dumps in that listing means the service is
protected. Do this the day you create the service, not the day you
need it.

## 6. Pitfalls (learned the hard way)

- **`/tmp` on ks4 is a 16 GiB tmpfs — it is RAM.** Never restore, dump
  or stage data there; use `/backup/…`.
- **Avoid millions of tiny files.** A service that keeps every version
  of everything (seafile did, 24.9 M objects) makes every backup and
  every filesystem walk take days. If the upstream software has a
  garbage-collection or retention setting, turn it on *and schedule
  it* the day you deploy — see [ks4/seafile-gc.md](ks4/seafile-gc.md).
- **Keep data under one predictable path** per service; scattered data
  means several `restic-paths` lines and things get forgotten.
- **Static IP + a note in the service doc**; two containers fighting
  over one IP is a confusing outage.
- Cron on ks4 does not have `/usr/local/bin` in its `PATH` — use
  absolute paths in anything you schedule.

## 7. Checklist

- [ ] container created from a Debian/Ubuntu image, static IP noted
- [ ] no Docker (or a written reason + volume paths)
- [ ] install/config steps written in `doc/ks4/<service>.md`
- [ ] exposed through gateway (HTTP) or a proxy device (other)
- [ ] snapshots left at the profile default
- [ ] data paths added to `restic-paths` (committed to git)
- [ ] `restic-backup.sh` run once; dumps and paths verified in the repo
- [ ] service added to the table in [ks4/install.md](ks4/install.md)