doc/new-container.md

Creating a new service on ks4 — conventions

Audience: you have root on ks4 and want to add a service. Follow this and the service is automatically replicated, backed up and restorable, without touching the backup machinery. Companion page: backup-strategy.md.

1. One service = one incus container

incus launch images:debian/13 myservice
incus config device add myservice eth0 nic nictype=bridged parent=incusbr0 \
    ipv4.address=192.168.1.30            # pick a free IP, note it in the doc
incus exec myservice -- bash             # install and configure from here

Rules of thumb:

• Debian 13 or Ubuntu 24.04 images, unprivileged (the default). Privileged only with a written reason (nuc/jellyfin-client.md is the one example: it needs raw device access).
• Configure through incus exec and write the commands down in doc/ks4/<service>.md as you go. The doc is the rebuild procedure — that is the whole convention: re-running it recreates the service.
• The container is on the NAT bridge incusbr0 (192.168.1.0/24). It is not reachable from the internet by itself.
• Web services: do not open ports. Add an nginx vhost in the gateway container that proxies to http://192.168.1.30:PORT; gateway is the single HTTP/S entry point and handles TLS. Non-HTTP services (mail, wireguard) use an incus proxy device — copy an existing one as a model.

2. No Docker inside the container

Docker-in-incus works (login/outline predate this rule) but costs a second layer of networking, storage and updates, and hides the service's data behind docker volumes.

Since incus can run OCI images natively, an upstream docker-compose.yml becomes incus instances:

incus remote add docker https://docker.io --protocol=oci    # once per host
incus launch docker:library/redis myservice-redis

and multi-container stacks are described with incus-compose (upstream, compose-file syntax → incus instances). Each piece is then a normal instance: same network, same snapshots, same backups, incus exec to debug, no nested runtime.

If you truly need Docker (upstream ships only a compose stack you don't want to translate), say so in the service doc and note where the volumes live — you will need them in step 4.

3. Snapshots (do nothing, but know why)

The default profile gives every instance snapshots.schedule = 0 3 * * *, snapshots.expiry = 7d. Leave it. Those snapshots are what make the nightly replication incremental — an instance without them forces a full re-send of its whole disk every night.

4. Wire it into the backups

Automatic, nothing to do:

• the 01:00 replica leg copies every instance to the backup pool and off-site — new instances included;
• databases are auto-discovered: any running container with MariaDB/MySQL or PostgreSQL gets every non-system database dumped nightly, plus users/grants.

Manual, one line: data that lives in the filesystem (uploads, repositories, mail spools…) must be listed in restic-paths, one absolute host path per line:

/var/lib/incus/storage-pools/data/containers/myservice/rootfs/var/lib/myservice

(The container's /x is …/containers/<name>/rootfs/x on the host.) No comments in that file — every line is read as a path. Regenerable caches can be skipped by adding a pattern to restic-exclude. Both files live in the scripts git repo: edit there, commit, git pull on ks4.

For the database auto-discovery to work, keep the defaults:

• MariaDB/MySQL: client and dump binaries installed, root access over the local unix socket. Do not leave a database = … line in /root/.my.cnf — it breaks SHOW DATABASES and the container is then skipped (this happened once and went unnoticed for months).
• PostgreSQL: reachable as the postgres system user (peer auth).
• PostgreSQL in Docker: the container's image name must contain "postgres" (official images do).

A database you do not want dumped (huge, static, re-importable) goes into /root/scripts/db-exclude as <instance>/<db> — every skip is logged, so the list cannot rot silently.

5. Verify, once

cd /root/scripts && git pull
/root/scripts/restic-backup.sh -s dumps        # ~2 min: see your DB in the log
ls /backup/dumps/mariadb/myservice/            # dump present?
/root/scripts/restic-backup.sh                 # full run, then:
. /root/.restic-env
restic -r s3:s3.sbg.io.cloud.ovh.net/restic-data ls latest | grep myservice | head

Seeing your paths and dumps in that listing means the service is protected. Do this the day you create the service, not the day you need it.

6. Pitfalls (learned the hard way)

• /tmp on ks4 is a 16 GiB tmpfs — it is RAM. Never restore, dump or stage data there; use /backup/….
• Avoid millions of tiny files. A service that keeps every version of everything (seafile did, 24.9 M objects) makes every backup and every filesystem walk take days. If the upstream software has a garbage-collection or retention setting, turn it on and schedule it the day you deploy — see ks4/seafile-gc.md.
• Keep data under one predictable path per service; scattered data means several restic-paths lines and things get forgotten.
• Static IP + a note in the service doc; two containers fighting over one IP is a confusing outage.
• Cron on ks4 does not have /usr/local/bin in its PATH — use absolute paths in anything you schedule.

7. Checklist

• container created from a Debian/Ubuntu image, static IP noted
• no Docker (or a written reason + volume paths)
• install/config steps written in doc/ks4/<service>.md
• exposed through gateway (HTTP) or a proxy device (other)
• snapshots left at the profile default
• data paths added to restic-paths (committed to git)
• restic-backup.sh run once; dumps and paths verified in the repo
• service added to the table in ks4/install.md