solar-relay/README.md

# solar-relay

Local Fronius GEN24 recorder and HTTPS relay for the sibling `solar` website.
Python 3.11+ and a project-local virtual environment are required. License:
AGPL-3.0-or-later, see [LICENSE](LICENSE).

The collector takes one observation every **five minutes**, aligned to the clock
(`:00`, `:05`, …). Startup and restarts wait for the next five-minute boundary.
`poll_interval_seconds` must be 300. Each observation
contains production, consumption, grid power, cumulative energy counters, boiler
switch state, individual source timestamps, errors, and the original JSON responses.
Requests to the inverter remain at least four seconds apart, so these readings are
not simultaneous. The default inverter is `http://fronius.local`, meter `0`, boiler `pin1`.

An HTTP/network request failure retries the affected endpoint up to three times
(four attempts total), keeping the same request spacing. Successful endpoints are
not repeated. If retries are exhausted, that source is recorded as unavailable.
Retries belong to the same observation and do not reset the five-minute clock.
Scheduled collection remains anchored to `:00`, `:05`, `:10`, … boundaries.

Observations are saved durably to **Parquet** before upload. Separate workers handle
delivery and optional history recovery; an internet outage does not block recording.
Acknowledged observations are deleted **30 days after acknowledgement**. Pending
observations never expire, even if their measurement dates are much older.

## Installation

```bash
python3 -m venv .venv
source .venv/bin/activate
python3 -m pip install --only-binary=pyarrow -r requirements.txt
cp config.example.toml config.toml
python3 -m solar_relay --config config.toml status
python3 -m solar_relay --config config.toml run
```

Use `.venv/bin/python` directly if you prefer not to activate the environment.
All packages stay inside the venv. Parquet storage uses the pinned **PyArrow**
dependency in `requirements.txt`. Use a 64-bit Raspberry Pi OS
with an available PyArrow wheel; `--only-binary=pyarrow` avoids a large source build.
The deployment workflow installs dependencies in both its test and runtime venvs.

Configuration, credentials, `.venv` and Parquet data are ignored by Git.
Relative paths resolve against the configuration file, rather than the working directory.
Uploads default to disabled until `[upload].url` is configured.

## Sources and measurement limits

| Source | Endpoint |
| --- | --- |
| Solar API: PV, AC output, household load, grid power | `/solar_api/v1/GetPowerFlowRealtimeData.fcgi` |
| Solar API: grid meter counters | `/solar_api/v1/GetMeterRealtimeData.cgi?Scope=System` |
| Internal GEN24 API: boiler switch | `/api/status/emrs` |

Solar API must be enabled in the inverter's Communication settings. The collector
only reads these endpoints. The boiler endpoint is an internal firmware interface;
if it stops working, boiler state is recorded as unknown rather than off.

| Reading | Meaning |
| --- | --- |
| `production_w` | `Site.P_PV`, PV generator DC power |
| `production_ac_w` | Sum of inverter `P`, AC output |
| `consumption_w` | `-Site.P_Load`, household load on this GEN24 |
| `grid_w` | Positive import, negative export at the feed-in meter |
| `grid_import_w`, `grid_export_w` | Nonnegative directional grid power |
| `production_ac_total_wh` | Lifetime AC production counter |
| `grid_import_total_wh`, `grid_export_total_wh` | Grid meter lifetime import/export counters |
| `boiler_enabled`, `boiler_reason` | Output state (`true`, `false`, or `null`) and reason code |

The meter must be installed at the grid feed-in point. Missing/stale readings and
source errors never become zero energy or an off boiler. DC panel power differs
from AC output by conversion losses. The website uses AC output for its energy balance.

The boiler switch enables heating but does not measure electricity use: the thermostat
can stop heating while the output remains enabled. Five-minute polling can miss short
transitions; enabled durations are estimates. A branch meter is needed for boiler kWh.
The 2000 W threshold is power (2 kW), not energy (kWh).

## Parquet archive, delivery and retention

`collector.archive` is a directory, containing one partition per **UTC date**:

```text
archive/
  .lock
  date=2026-11-01/samples.parquet
  date=2026-11-02/samples.parquet
```

The files contain typed numeric/boolean readings, IDs, UTC observation timestamps,
coverage flags, `uploaded_at`, and the complete original sample as JSON. They use
Zstandard compression. Each update writes a temporary file, flushes/fsyncs it,
atomically replaces the daily file, and fsyncs the directory. File permissions are
0600 and new archive directories are 0700. Thread/process locks prevent a collector,
uploader, status request or CLI replay from overwriting another operation. HTTP
requests happen outside the archive lock.

The upload request is a JSON object with `schema_version: 1` and a `samples` array,
authenticated with `upload.token` from private `config.toml`, or the environment
variable named by `upload.token_env` when that setting is empty. The website must
commit its data before responding:

```json
{"accepted_ids": ["1599c148-612c-46ab-aa9a-cf9a13271a72"]}
```

Only explicitly acknowledged IDs receive an `uploaded_at` timestamp. A plain HTTP
200, malformed response, timeout or unknown ID never clears the queue. Partial
acknowledgements mark only the accepted records; remaining records retry with
exponential backoff. Delivery is **at least once**: the website deduplicates stable
UUIDs when a response is lost, so retries and replays do not double-count energy.

`collector.retention_days` defaults to 30. Automatic cleanup runs at service startup
and hourly, deleting records acknowledged more than that many days ago. Old pending
records remain. Replay clears acknowledgement and therefore prevents expiry until
another successful transfer. The main website retains its own imported history.

```bash
.venv/bin/python -m solar_relay --config config.toml prune
.venv/bin/python -m solar_relay --config config.toml export \
  --from 2026-10-29 --until 2026-10-31 > readings.jsonl
.venv/bin/python -m solar_relay --config config.toml replay \
  --from 2026-10-29 --until 2026-10-31
.venv/bin/python -m solar_relay --config config.toml upload
```

Date bounds use `collector.timezone`, including DST transitions. `--from` is inclusive;
`--until` is exclusive. ISO timestamps may include an explicit offset. Unbounded
export/replay covers all retained observations. Export provides a consistent snapshot;
stop the service and other CLI writers before copying the archive directory as a backup.

## Recovering history after Wi-Fi loss

The default mode always collects live data from the inverter's local Fronius API.
The optional backup (`[history].enabled = true`) fills historical energy gaps from
Solar.web in a separate worker. It does not replace or change live collection;
when there are no eligible gaps, it makes no Solar.web requests.

GEN24 records historical data for transfer to Solar.web after reconnecting. This is
separate from its local Solar API: Fronius explicitly lists archive access for GEN24
as unavailable, and this inverter's `GetArchiveData.cgi` returns 404. Inspecting its
current web interface also found no energy-history retrieval endpoint. Its SDP support
download is an encrypted `.cry` diagnostic file, not a documented energy-history API.

See the official [Solar API manual, section 5.1.1](https://www.fronius.com/~/downloads/Solar%20Energy/Operating%20Instructions/42%2C0410%2C2012.pdf)
for the local API's archive limitations.

Optional recovery signs in to Solar.web with a real Fronius account and reads its
private daily charts after the inverter uploads its backlog, using the same login
flow and chart series as the sibling `solar` website. No cloud calls occur while
disabled. A public guest link and Query API subscription/keys are not needed.

Create a dedicated Solar.web / Fronius account and grant it permission to view this
installation's metrics. Keep public sharing disabled. The account can have guest
permissions on the installation, but it signs in privately through the normal account
login; the deactivated public guest interface is never used.

Set the backup account credentials directly in your private `config.toml`:

```toml
[history]
enabled = true
pv_system_id = "your-pv-system-uuid"
username = "your-account-email"
password = "your-account-password"
```

No extra environment file is needed for this login. Set both `username` and `password`;
an incomplete pair is rejected. Direct credentials take precedence over the environment.
Use TOML literal strings (`'...'`) if a password contains backslashes or double quotes.

Alternatively, leave both credentials empty and set the environment variables named
by `username_env` and `password_env`. For the systemd service, these can go in its
root-readable `/etc/solar-relay/solar-relay.env`:

```text
SOLARWEB_USERNAME=your-account-email
SOLARWEB_PASSWORD=your-account-password
```

While live collection runs, the separate recovery worker checks at startup and hourly
by default. It checks every five-minute energy interval back to
`collector.retention_days` (30 days by default), including entirely empty days and
dates before the first local observation. A two-hour delay allows the inverter to
upload its recent history. Only local dates containing gaps are fetched, and existing
live or recovered energy is preserved. When all eligible intervals are present, no
Solar.web requests are made. Unavailable chart days remain missing for retry and do
not prevent other dates from being recovered. Login and network failures stop that
attempt and retry on the next scheduled check.

Set `collector.timezone` to the installation's Solar.web timezone; local chart dates
include 23/25-hour DST days. The old `history.lookback_days` setting is accepted but
ignored; `collector.retention_days` controls the automatic recovery window.

Recovery pairs the `FromGen` Production series from the consumption view with the
`ToConsumer` Consumption series from the production view, regardless of the display
language. As in `solar`, each five-minute W value is treated as constant over that
interval and divided by 12 to obtain Wh. Self-consumption is the smaller of production
and consumption; purchase/export are their nonnegative differences. These are chart
estimates, rather than the former Query API's measured Wh channels. The original
paired power values and conversion method are retained in each recovered record.
Null or unpaired readings stay missing and are tried again later.

Login uses HTTPS and in-memory cookies. Credentials are submitted only to the Fronius
password endpoint; SSO callbacks and redirects are restricted to the Solar.web and
Fronius login hosts. Credentials are excluded from configuration reprs and never
logged. Session cookies and authentication responses are never logged or written to
disk. An expired session is renewed once. Rejected credentials,
MFA, CAPTCHA, consent, or a changed sign-in form stop that attempt with a redacted
error. Complete any required interactive account setup in a browser.

When migrating from Query API recovery, replace `access_key_id_env` and
`access_key_value_env` with `username` and `password` in `[history]`, or use the
environment fallback above. Obsolete configuration keys are accepted and ignored
for migration. Restart the service after updating its configuration or environment.
Recovery requires the account credentials even if API keys remain set.

```bash
.venv/bin/python -m solar_relay --config config.toml recover \
  --from 2026-10-29 --until 2026-10-31
```

Manual recovery with explicit date bounds accepts at most 30 days per invocation and
can query dates older than the automatic retention window. Unbounded recovery checks
the whole retention window, even when longer than 30 days. Only aligned five-minute
chart samples in W are accepted;
longer aggregates and unexpected units are rejected.
Recovery stores stable UUIDs and `history_interval` records, which the main website
uses directly without integrating them as live power. Install the corresponding
`solar` receiver update before enabling cloud recovery.

**Boiler switch history and cumulative meter readings are not recovered.** Historical
energy intervals leave those sources unknown, and the status page shows partial
coverage. Counter differences can describe a total across an outage, but cannot
recover individual five-minute values. Fully local operation leaves Wi-Fi gaps visible.

## Raspberry Pi service

Deploy the checkout to `/var/www/html/solar-relay` and create/install its venv as above.
The service reads **`/var/www/html/solar-relay/config.toml`**. Keep your existing local
file there; deployment excludes it from both replacement and deletion. If this is a
fresh installation, create it from the example only when it does not exist:

```bash
cd /var/www/html/solar-relay
test -e config.toml || cp config.example.toml config.toml
sudo chgrp www-data config.toml
chmod 0640 config.toml
```

The `www-data` service user needs read access to this private configuration, which
may contain the upload token and Solar.web login credentials. Keep real credentials
out of `config.example.toml` and Git. Ensure the proxy serves the status endpoint
rather than repository files.

Keep the archive under `/var/www/html/solar-relay/data/archive`. The relative setting
`data/archive` resolves to that location because paths are relative to `config.toml`.
In the existing `[collector]` section, retain your other settings:

```toml
[collector]
source_id = "home-gen24"
archive = "data/archive"
timezone = "Europe/Zurich"
poll_interval_seconds = 300
retention_days = 30
```

The service uses the existing `www-data` account. Deployment creates `data/archive`
as the `www-data` runner before restarting the service. For a manual installation,
create the directory as that account before starting the service:

```bash
sudo -u www-data install -d -m 0700 \
  /var/www/html/solar-relay/data /var/www/html/solar-relay/data/archive
```

The unit keeps the checkout read-only with `ProtectSystem=strict` and allows writes
to `/var/www/html/solar-relay/data` through `ReadWritePaths`. The data directory must
exist before starting the service; otherwise systemd exits with `226/NAMESPACE`
before Python can create it. Existing data must be writable by `www-data`. Use
`sudo -u www-data` for archive-writing CLI commands to preserve ownership.

Set the upload URL and shared bearer token in private `config.toml`:

```toml
[upload]
url = "https://your-solar-host/index.php?a=relay"
token = "your-shared-token"
```

Configure the same token and source ID on the main website as described in its README.
No extra environment file is needed. Alternatively, leave `token` empty and provide
the variable named by `token_env` (default `SOLAR_RELAY_TOKEN`). For systemd, put
`SOLAR_RELAY_TOKEN=your-shared-token` in `/etc/solar-relay/solar-relay.env` with
permissions 0600, readable by root. Create `/etc/solar-relay` with
`sudo install -d -m 0755 /etc/solar-relay` when adding that optional environment file.

```bash
sudo cp deploy/solar-relay.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable solar-relay
```

Recording/uploading remains separate from serving status. The first Gitea deployment
installs the code/venv and starts the service through restart. After a manual installation,
use `systemctl start solar-relay` when the code and venv are ready. Running
tests or `sample` does not install/start the service. Initial unit installation is manual.
Inspect service logs with `sudo journalctl -u solar-relay -f` after startup.

## HTTPS reverse proxy and status page

Python listens at **`http://127.0.0.1:8765`**. Choose one proxy configuration and include
it inside the existing HTTPS virtual host for `rpioffice.nest.lutran.ch`:

- Nginx: [`deploy/nginx-solar-relay.conf`](deploy/nginx-solar-relay.conf).
- Apache: [`deploy/apache-solar-relay.conf`](deploy/apache-solar-relay.conf).
  Enable `proxy`, `proxy_http` and `headers` (`sudo a2enmod proxy proxy_http headers`).

Keep your existing TLS certificates on the proxy. Proxy the entire `/solar-relay/`
prefix; do not serve the repository/archive as static files. The collector accepts
both a stripped proxy path and a preserved `/solar-relay/` prefix. Validate the web
server configuration before reloading it (`nginx -t` or `apachectl configtest`).

The status page is **https://rpioffice.nest.lutran.ch/solar-relay/**. It shows last
extraction/last success, upload backlog, retained observations, and daily five-minute
completeness with a selectable interval grid and gap ranges. Dates use the configured
timezone. DST days have 276 or 300 intervals; ordinary days have 288.

Complete means at least one fresh, successful extraction for energy, meter and boiler
in that five-minute slot. It does not mean continuously observed power or switch state.
The unfinished current slot is excluded from missing counts. Historical recovery
contributes energy only. After retention cleanup, pruned records no longer contribute
to this local page; the website keeps the transferred history. Refresh is every 15 seconds.

Endpoints under `/solar-relay/`:

- `/`: status page.
- `/api/status?days=14`: status and extraction completeness JSON.
- `/healthz`: process liveness, also used by deployment checks.

All routes are read-only and expose no raw readings, credentials or filesystem paths.
Disable `[http].enabled` and remove the workflow's health probe if HTTP is not needed.

## Gitea deployment

`.gitea/workflows/deploy.yml` deploys pushes to `main` or manual workflow runs to
`/var/www/html/solar-relay` and restarts `solar-relay.service`. Register the Pi runner
with **`solar-relay:host`**; its checkout must be separate from production.

The runner needs Python 3.11+, `python3-venv`/pip, a supported 64-bit PyArrow platform,
`rsync`, `flock`, `curl`, `sudo`, systemd and Node.js 20+ for `actions/checkout@v4`.
The configured runner account is `www-data`. Give it write access to the deployment
directory and keep the local config there. Install the unit before the first deployment,
then add a restricted sudoers rule using the actual runner
account and the path from `command -v systemctl`:

```sh
sudo visudo -f /etc/sudoers.d/solar-relay
```

```sudoers
www-data ALL=(root) NOPASSWD: /usr/bin/systemctl restart solar-relay
```

The workflow installs dependencies and runs tests in `.venv-ci`, checks the installed
unit, serializes deployment with `flock`, syncs source, installs runtime dependencies
inside `.venv`, creates the archive directory if needed, restarts and checks the
service, then probes the public HTTPS health
endpoint with certificate verification. Runtime venv/configuration, `data/` and
Parquet files are preserved. The archive under `data/archive` is protected by the
`/data/` rsync exclusion, and the credential environment file under `/etc` is outside
the sync. The local `config.toml` is protected by the explicit rsync exclusion even
when `--delete` is used.
Deployment never copies `config.example.toml` over your configuration.

If the deployment directory is initially empty, enable the unit without `--now`; the
first deployment creates the venv and starts it. Set up the proxy before deployment.
For unit changes, install the updated file and run `systemctl daemon-reload`; the
workflow detects a mismatch and does not grant itself broader sudo access.

## Checks

```bash
.venv/bin/python -m unittest discover -s tests -v
node --check solar_relay/web/status.js
```

Tests cover normalization, real local HTTP delivery/retries, Parquet round-trips,
concurrent writers, interrupted writes, retention, DST completeness,
mocked authenticated history recovery, status routes and the deployment script.
Live Solar.web recovery requires real account credentials and is not exercised by tests.