solar-relay/README.md

# solar-relay

Local Fronius GEN24 recorder and HTTPS relay for the sibling `solar` website.
Python 3.11+ and a project-local virtual environment are required. License:
AGPL-3.0-or-later, see [LICENSE](LICENSE).

The collector takes one observation every **five minutes**, aligned to the clock
(`:00`, `:05`, …). Startup and restarts wait for the next five-minute boundary.
`poll_interval_seconds` must be 300. Each observation
contains production, consumption, grid power, cumulative energy counters, boiler
switch state, individual source timestamps, errors, and the original JSON responses.
Requests to the inverter remain at least four seconds apart, so these readings are
not simultaneous. The default inverter is `http://fronius.local`, meter `0`, boiler `pin1`.

An HTTP/network request failure retries the affected endpoint up to three times
(four attempts total), keeping the same request spacing. Successful endpoints are
not repeated. If retries are exhausted, that source is recorded as unavailable.
Retries belong to the same observation and do not reset the five-minute clock.
Scheduled collection remains anchored to `:00`, `:05`, `:10`, … boundaries.

Observations are saved durably to **Parquet** before upload. Separate workers handle
delivery and optional history recovery; an internet outage does not block recording.
Acknowledged observations are retained for **30 full local calendar days after
acknowledgement**. Cleanup removes whole observation days. A pending or recently
acknowledged record keeps its whole local day, even when its measurements are much older.

## Installation

```bash
python3 -m venv .venv
source .venv/bin/activate
python3 -m pip install --only-binary=pyarrow -r requirements.txt
cp config.example.toml config.toml
python3 -m solar_relay --config config.toml status
python3 -m solar_relay --config config.toml run
```

Use `.venv/bin/python` directly if you prefer not to activate the environment.
All packages stay inside the venv. Parquet storage uses the pinned **PyArrow**
dependency in `requirements.txt`. Use a 64-bit Raspberry Pi OS
with an available PyArrow wheel; `--only-binary=pyarrow` avoids a large source build.
The deployment workflow installs dependencies in both its test and runtime venvs.

Configuration, credentials, `.venv` and Parquet data are ignored by Git.
Relative paths resolve against the configuration file, rather than the working directory.
Uploads default to disabled until `[upload].url` is configured.

## Sources and measurement limits

| Source | Endpoint |
| --- | --- |
| Solar API: PV, AC output, household load, grid power | `/solar_api/v1/GetPowerFlowRealtimeData.fcgi` |
| Solar API: grid meter counters | `/solar_api/v1/GetMeterRealtimeData.cgi?Scope=System` |
| Internal GEN24 API: boiler switch | `/api/status/emrs` |
| Internal GEN24 API: boiler switch-on threshold | `/api/config/emrs` |

Solar API must be enabled in the inverter's Communication settings. The collector
only reads these endpoints. The boiler endpoint is an internal firmware interface;
if it stops working, boiler state is recorded as unknown rather than off.

| Reading | Meaning |
| --- | --- |
| `production_w` | `Site.P_PV`, PV generator DC power |
| `production_ac_w` | Sum of inverter `P`, AC output |
| `consumption_w` | `-Site.P_Load`, household load on this GEN24 |
| `grid_w` | Positive import, negative export at the feed-in meter |
| `grid_import_w`, `grid_export_w` | Nonnegative directional grid power |
| `production_ac_total_wh` | Lifetime AC production counter |
| `grid_import_total_wh`, `grid_export_total_wh` | Grid meter lifetime import/export counters |
| `boiler_enabled`, `boiler_reason` | Output state (`true`, `false`, or `null`) and reason code |
| `boiler_threshold_w`, `boiler_threshold_source` | Switch-on threshold in watts and `api` or `config` provenance |

The meter must be installed at the grid feed-in point. Missing/stale readings and
source errors never become zero energy or an off boiler. DC panel power differs
from AC output by conversion losses. The website uses AC output for its energy balance.

The boiler switch enables heating but does not measure electricity use: the thermostat
can stop heating while the output remains enabled. Five-minute polling can miss short
transitions; enabled durations are estimates. A branch meter is needed for boiler kWh.
The threshold is power (W), not energy (Wh).

The collector tries `/api/config/emrs` once per observation and reads
`rules.<boiler_pin>.thresholdOn`, matching the GEN24's own web interface. Negative
`pgrid` thresholds denote exported surplus; their absolute magnitude is used.
This endpoint can require authentication even while the switch state is public.
If it is protected, unavailable or has no valid threshold, the collector uses
`[inverter].boiler_threshold_w` (default `2000`) from its private configuration.
A settings failure does not invalidate a successfully read switch state. The
threshold and its source are saved with each new observation and sent to Solar.
Requests still respect the minimum four-second spacing. Existing archives remain
readable; legacy switch-only samples receive the configured fallback when uploaded,
without rewriting their original payload or replacing a threshold already recorded.

## Parquet archive, delivery and retention

`collector.archive` is a directory, containing one partition per **UTC date**:

```text
archive/
  .lock
  date=2026-11-01/samples.parquet
  date=2026-11-02/samples.parquet
```

The files contain typed numeric/boolean readings, IDs, UTC observation timestamps,
coverage flags, `uploaded_at`, and the complete original sample as JSON. They use
Zstandard compression. Each update writes a temporary file, flushes/fsyncs it,
atomically replaces the daily file, and fsyncs the directory. File permissions are
0600 and new archive directories are 0700. Thread/process locks prevent a collector,
uploader, status request or CLI replay from overwriting another operation. HTTP
requests happen outside the archive lock.

Live and recovered records share the same Parquet table and watt columns:
`production_ac_w`, `consumption_w`, `grid_w`, `grid_import_w`, and `grid_export_w`.
Recovered chart values are stored directly in those fields; `production_w` (PV DC),
lifetime counters, and boiler state remain unknown. `record_kind` and the original
chart data distinguish recovered values from live snapshots. Both sources use the
same validation and completeness checks. The relay stores power directly; energy
totals are calculated by the receiving application.

These common power readings match the sibling `solar` application's existing
model: its `stats` table stores timestamped watts for Production and Consumption,
then applies `5/60` only when calculating Wh totals. `production_ac_w` maps to
Production and `consumption_w` maps to Consumption. Solar currently imports only
the boiler switch from these uploads; it continues to obtain production and
consumption directly from Solar.web.

The upload request is a JSON object with `schema_version: 1` and a `samples` array,
authenticated with `upload.token` from private `config.toml`, or the environment
variable named by `upload.token_env` when that setting is empty. The website must
commit its data before responding:

```json
{"result":"success","data":{"accepted_ids":["1599c148-612c-46ab-aa9a-cf9a13271a72"]},"desc_lang_id":"","desc_lang_params":[]}
```

Only explicitly acknowledged IDs receive an `uploaded_at` timestamp. A plain HTTP
200, malformed response, timeout or unknown ID never clears the queue. Partial
acknowledgements mark only the accepted records; remaining records retry with
exponential backoff. Delivery is **at least once**: a receiver must deduplicate stable
UUIDs when a response is lost, so retries and replays do not double-count energy.
Solar uses its standard JSON result envelope: `result` must be `success`, and
acknowledgements are read from `data.accepted_ids`. Plain `accepted_ids` responses
from other compatible receivers are also supported.
The sibling Solar app provides `index.php?a=relay` as this receiver. Set `[upload].url`
to its HTTPS URL and `[upload].token` to the same private value as Solar's
`Settings::SOLAR_RELAY_TOKEN`. The collector's `source_id` must match Solar's
`Settings::SOLAR_RELAY_SOURCE_ID` (default `home-gen24`). Restart the relay after
changing its private configuration. Pending samples then upload automatically;
use `replay` to resend retained samples already acknowledged elsewhere.

Solar stores five-minute `boiler` stats in watts: the recorded threshold when enabled,
zero when off. It calculates daily Wh by summing watts × 5/60 and uses those totals
for monthly and yearly charts. It acknowledges other readings without importing them,
including historical recovery records whose boiler state is unknown. Solar's
existing Solar.web import continues to supply production and consumption.

`collector.retention_days` defaults to 30. All retention cutoffs are midnight in
`collector.timezone`. For example, on November 2 with 30 days retained, the cutoff
is October 3 at 00:00 local time, regardless of the execution time. The entire
October 3 day is kept and eligible for history recovery.

Automatic cleanup runs at service startup and hourly. It removes an older local
observation day only when every record in that day was acknowledged before the
cutoff. Pending or recently acknowledged records protect the entire day, including
its rows in adjacent UTC Parquet partitions. Replay clears acknowledgement and
therefore prevents expiry until another successful transfer. The main website
retains its own imported history.

```bash
.venv/bin/python -m solar_relay --config config.toml prune
.venv/bin/python -m solar_relay --config config.toml export \
  --from 2026-10-29 --until 2026-10-31 > readings.jsonl
.venv/bin/python -m solar_relay --config config.toml replay \
  --from 2026-10-29 --until 2026-10-31
.venv/bin/python -m solar_relay --config config.toml upload
```

Date bounds use `collector.timezone`, including DST transitions. `--from` is inclusive;
`--until` is exclusive. ISO timestamps may include an explicit offset. Unbounded
export/replay covers all retained observations. Export provides a consistent snapshot;
stop the service and other CLI writers before copying the archive directory as a backup.

## Recovering history after Wi-Fi loss

The default mode always collects live data from the inverter's local Fronius API.
The optional backup (`[history].enabled = true`) fills historical energy gaps from
Solar.web in a separate worker. It does not replace or change live collection;
when there are no eligible gaps, it makes no Solar.web requests.

GEN24 records historical data for transfer to Solar.web after reconnecting. This is
separate from its local Solar API: Fronius explicitly lists archive access for GEN24
as unavailable, and this inverter's `GetArchiveData.cgi` returns 404. Inspecting its
current web interface also found no energy-history retrieval endpoint. Its SDP support
download is an encrypted `.cry` diagnostic file, not a documented energy-history API.

See the official [Solar API manual, section 5.1.1](https://www.fronius.com/~/downloads/Solar%20Energy/Operating%20Instructions/42%2C0410%2C2012.pdf)
for the local API's archive limitations.

Optional recovery signs in to Solar.web with a real Fronius account and reads its
private daily charts after the inverter uploads its backlog, using the same login
flow and chart series as the sibling `solar` website. No cloud calls occur while
disabled. A public guest link is not needed.

Create a dedicated Solar.web / Fronius account and grant it permission to view this
installation's metrics. Keep public sharing disabled. The account can have guest
permissions on the installation, but it signs in privately through the normal account
login; the deactivated public guest interface is never used.

Set the backup account credentials directly in your private `config.toml`:

```toml
[history]
enabled = true
pv_system_id = "your-pv-system-uuid"
username = "your-account-email"
password = "your-account-password"
```

No extra environment file is needed for this login. Set both `username` and `password`;
an incomplete pair is rejected. Direct credentials take precedence over the environment.
Use TOML literal strings (`'...'`) if a password contains backslashes or double quotes.

Alternatively, leave both credentials empty and set the environment variables named
by `username_env` and `password_env`. For the systemd service, these can go in its
root-readable `/etc/solar-relay/solar-relay.env`:

```text
SOLARWEB_USERNAME=your-account-email
SOLARWEB_PASSWORD=your-account-password
```

While live collection runs, the separate recovery worker checks at startup and hourly
by default. It checks every five-minute energy interval from midnight at the start
of the date `collector.retention_days` ago (30 days by default), including empty
days and dates before the first local observation. A two-hour delay allows the inverter to
upload its recent history. Only local dates containing gaps are fetched, and existing
live or recovered energy is preserved. When all eligible intervals are present, no
Solar.web requests are made. Unavailable chart days remain missing for retry and do
not prevent other dates from being recovered. Login and network failures stop that
attempt and retry on the next scheduled check.

Set `collector.timezone` to the installation's Solar.web timezone; local chart dates
include whole 23/25-hour DST days. `collector.retention_days` controls the automatic
recovery window.

Recovery pairs the `FromGen` Production series from the consumption view with the
`ToConsumer` Consumption series from the production view, regardless of the display
language. Paired power values are stored directly in the common watt readings.
Grid import/export are their nonnegative differences. The relay keeps the original
chart watts without converting them to interval energy.
Null or unpaired readings stay missing and are tried again later.

Login uses HTTPS and in-memory cookies. Credentials are submitted only to the Fronius
password endpoint; SSO callbacks and redirects are restricted to the Solar.web and
Fronius login hosts. Credentials are excluded from configuration reprs and never
logged. Session cookies and authentication responses are never logged or written to
disk. An expired session is renewed once. Rejected credentials,
MFA, CAPTCHA, consent, or a changed sign-in form stop that attempt with a redacted
error. Complete any required interactive account setup in a browser.

```bash
.venv/bin/python -m solar_relay --config config.toml recover \
  --from 2026-10-29 --until 2026-10-31
```

Manual recovery with explicit date bounds accepts at most 30 local calendar days
per invocation and can query dates older than the automatic retention window. Unbounded recovery checks
the whole retention window, even when longer than 30 days. Only aligned five-minute
chart samples in W are accepted;
longer aggregates and unexpected units are rejected.
Recovery stores stable UUIDs and common power readings with their chart provenance.
A future solar-side importer can put both
live and recovered production/consumption in the existing `stats` watt table and
use its normal five-minute Wh conversion exactly once.

**Boiler switch history and cumulative meter readings are not recovered.** Historical
energy intervals leave those sources unknown, and the status page shows partial
coverage. Counter differences can describe a total across an outage, but cannot
recover individual five-minute values. Fully local operation leaves Wi-Fi gaps visible.

## Raspberry Pi service

Deploy the checkout to `/var/www/html/solar-relay` and create/install its venv as above.
The service reads **`/var/www/html/solar-relay/config.toml`**. Keep your existing local
file there; deployment excludes it from both replacement and deletion. If this is a
fresh installation, create it from the example only when it does not exist:

```bash
cd /var/www/html/solar-relay
test -e config.toml || cp config.example.toml config.toml
sudo chgrp www-data config.toml
chmod 0640 config.toml
```

The `www-data` service user needs read access to this private configuration, which
may contain the upload token and Solar.web login credentials. Keep real credentials
out of `config.example.toml` and Git. Ensure the proxy serves the status endpoint
rather than repository files.

Keep the archive under `/var/www/html/solar-relay/data/archive`. The relative setting
`data/archive` resolves to that location because paths are relative to `config.toml`.
In the existing `[collector]` section, retain your other settings:

```toml
[collector]
source_id = "home-gen24"
archive = "data/archive"
timezone = "Europe/Zurich"
poll_interval_seconds = 300
retention_days = 30
```

The service uses the existing `www-data` account. Deployment creates `data/archive`
as the `www-data` runner before restarting the service. For a manual installation,
create the directory as that account before starting the service:

```bash
sudo -u www-data install -d -m 0700 \
  /var/www/html/solar-relay/data /var/www/html/solar-relay/data/archive
```

The unit keeps the checkout read-only with `ProtectSystem=strict` and allows writes
to `/var/www/html/solar-relay/data` through `ReadWritePaths`. The data directory must
exist before starting the service; otherwise systemd exits with `226/NAMESPACE`
before Python can create it. Existing data must be writable by `www-data`. Use
`sudo -u www-data` for archive-writing CLI commands to preserve ownership.

Set the upload URL and shared bearer token in private `config.toml`:

```toml
[upload]
url = "https://your-solar-host/index.php?a=relay"
token = "your-shared-token"
```

Configure the same token and source ID on the main website as described in its README.
No extra environment file is needed. Alternatively, leave `token` empty and provide
the variable named by `token_env` (default `SOLAR_RELAY_TOKEN`). For systemd, put
`SOLAR_RELAY_TOKEN=your-shared-token` in `/etc/solar-relay/solar-relay.env` with
permissions 0600, readable by root. Create `/etc/solar-relay` with
`sudo install -d -m 0755 /etc/solar-relay` when adding that optional environment file.

```bash
sudo cp deploy/solar-relay.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable solar-relay
```

Recording/uploading remains separate from serving status. The first Gitea deployment
installs the code/venv and starts the service through restart. After a manual installation,
use `systemctl start solar-relay` when the code and venv are ready. Running
tests or `sample` does not install/start the service. Initial unit installation is manual.
Inspect service logs with `sudo journalctl -u solar-relay -f` after startup.

## HTTPS reverse proxy and status page

Python listens at **`http://127.0.0.1:8765`**. Choose one proxy configuration and include
it inside the existing HTTPS virtual host for `rpioffice.nest.lutran.ch`:

- Nginx: [`deploy/nginx-solar-relay.conf`](deploy/nginx-solar-relay.conf).
- Apache: [`deploy/apache-solar-relay.conf`](deploy/apache-solar-relay.conf).
  Enable `proxy`, `proxy_http` and `headers` (`sudo a2enmod proxy proxy_http headers`).

Keep your existing TLS certificates on the proxy. Proxy the entire `/solar-relay/`
prefix; do not serve the repository/archive as static files. The collector accepts
both a stripped proxy path and a preserved `/solar-relay/` prefix. Validate the web
server configuration before reloading it (`nginx -t` or `apachectl configtest`).

The status page is **https://rpioffice.nest.lutran.ch/solar-relay/**. It shows last
extraction/last success, upload backlog, retained observations, and daily five-minute
completeness with a selectable interval grid and gap ranges. Dates use the configured
timezone. DST days have 276 or 300 intervals; ordinary days have 288.

The five-minute interval card appears above daily coverage. Status shows Energy and
Boiler; meter counters remain archived but do not affect status, completeness, or
interval details. Complete means valid energy and boiler data in the same five-minute
slot. It does not mean continuously observed power or switch state.
The current slot shows its saved extraction immediately, including partial or failed
results. Until a record arrives, it is pending and excluded from missing counts.
Future slots remain upcoming. Historical recovery
contributes energy only. After retention cleanup, pruned records no longer contribute
to this local page; the website keeps the transferred history. Refresh is every 15 seconds.

Endpoints under `/solar-relay/`:

- `/`: status page.
- `/api/status?days=14`: status and extraction completeness JSON.
- `/healthz`: process liveness, also used by deployment checks.

All routes are read-only and expose no raw readings, credentials or filesystem paths.
Disable `[http].enabled` and remove the workflow's health probe if HTTP is not needed.

## Gitea deployment

`.gitea/workflows/deploy.yml` deploys pushes to `main` or manual workflow runs to
`/var/www/html/solar-relay` and restarts `solar-relay.service`. Register the Pi runner
with **`solar-relay:host`**; its checkout must be separate from production.

The runner needs Python 3.11+, `python3-venv`/pip, a supported 64-bit PyArrow platform,
`rsync`, `flock`, `curl`, `sudo`, systemd and Node.js 20+ for `actions/checkout@v4`.
The configured runner account is `www-data`. Give it write access to the deployment
directory and keep the local config there. Install the unit before the first deployment,
then add a restricted sudoers rule using the actual runner
account and the path from `command -v systemctl`:

```sh
sudo visudo -f /etc/sudoers.d/solar-relay
```

```sudoers
www-data ALL=(root) NOPASSWD: /usr/bin/systemctl restart solar-relay
```

The workflow installs dependencies and runs tests in `.venv-ci`, checks the installed
unit, serializes deployment with `flock`, syncs source, installs runtime dependencies
inside `.venv`, creates the archive directory if needed, restarts and checks the
service, then probes the public HTTPS health
endpoint with certificate verification. Runtime venv/configuration, `data/` and
Parquet files are preserved. The archive under `data/archive` is protected by the
`/data/` rsync exclusion, and the credential environment file under `/etc` is outside
the sync. The local `config.toml` is protected by the explicit rsync exclusion even
when `--delete` is used.
Deployment never copies `config.example.toml` over your configuration.

If the deployment directory is initially empty, enable the unit without `--now`; the
first deployment creates the venv and starts it. Set up the proxy before deployment.
For unit changes, install the updated file and run `systemctl daemon-reload`; the
workflow detects a mismatch and does not grant itself broader sudo access.

## Checks

```bash
.venv/bin/python -m unittest discover -s tests -v
node --check solar_relay/web/status.js
```

Tests cover normalization, real local HTTP delivery/retries, Parquet round-trips,
concurrent writers, interrupted writes, retention, DST completeness,
mocked authenticated history recovery, status routes and the deployment script.
Live Solar.web recovery requires real account credentials and is not exercised by tests.