solar-relay/README.md

# solar-relay

Local Fronius GEN24 recorder and HTTPS relay for the sibling `solar` website.
Python 3.11+ and a project-local virtual environment are required. License:
AGPL-3.0-or-later, see [LICENSE](LICENSE).

The collector takes one observation every **five minutes**, aligned to the clock
(`:00`, `:05`, …). Startup and restarts wait for the next five-minute boundary.
`poll_interval_seconds` must be 300. Each observation
contains production, consumption, grid power, cumulative energy counters, boiler
switch state, individual source timestamps, errors, and the original JSON responses.
Requests to the inverter remain at least four seconds apart, so these readings are
not simultaneous. The default inverter is `http://fronius.local`, meter `0`, boiler `pin1`.

An HTTP/network request failure retries the affected endpoint up to three times
(four attempts total), keeping the same request spacing. Successful endpoints are
not repeated. If retries are exhausted, that source is recorded as unavailable.
Retries belong to the same observation and do not reset the five-minute clock.
Scheduled collection remains anchored to `:00`, `:05`, `:10`, … boundaries.

Observations are saved durably to **Parquet** before upload. Separate workers handle
delivery and optional history recovery; an internet outage does not block recording.
Acknowledged observations are deleted **30 days after acknowledgement**. Pending
observations never expire, even if their measurement dates are much older.

## Installation

```bash
python3 -m venv .venv
source .venv/bin/activate
python -m pip install --only-binary=pyarrow -r requirements.txt
cp config.example.toml config.toml
python -m solar_relay --config config.toml status
python -m solar_relay --config config.toml run
```

Use `.venv/bin/python` directly if you prefer not to activate the environment.
All packages stay inside the venv. Parquet storage uses the pinned **PyArrow**
dependency in `requirements.txt`. Use a 64-bit Raspberry Pi OS
with an available PyArrow wheel; `--only-binary=pyarrow` avoids a large source build.
The deployment workflow installs dependencies in both its test and runtime venvs.

Configuration, credentials, `.venv` and Parquet data are ignored by Git.
Relative paths resolve against the configuration file, rather than the working directory.
Uploads default to disabled until `[upload].url` is configured.

## Sources and measurement limits

| Source | Endpoint |
| --- | --- |
| Solar API: PV, AC output, household load, grid power | `/solar_api/v1/GetPowerFlowRealtimeData.fcgi` |
| Solar API: grid meter counters | `/solar_api/v1/GetMeterRealtimeData.cgi?Scope=System` |
| Internal GEN24 API: boiler switch | `/api/status/emrs` |

Solar API must be enabled in the inverter's Communication settings. The collector
only reads these endpoints. The boiler endpoint is an internal firmware interface;
if it stops working, boiler state is recorded as unknown rather than off.

| Reading | Meaning |
| --- | --- |
| `production_w` | `Site.P_PV`, PV generator DC power |
| `production_ac_w` | Sum of inverter `P`, AC output |
| `consumption_w` | `-Site.P_Load`, household load on this GEN24 |
| `grid_w` | Positive import, negative export at the feed-in meter |
| `grid_import_w`, `grid_export_w` | Nonnegative directional grid power |
| `production_ac_total_wh` | Lifetime AC production counter |
| `grid_import_total_wh`, `grid_export_total_wh` | Grid meter lifetime import/export counters |
| `boiler_enabled`, `boiler_reason` | Output state (`true`, `false`, or `null`) and reason code |

The meter must be installed at the grid feed-in point. Missing/stale readings and
source errors never become zero energy or an off boiler. DC panel power differs
from AC output by conversion losses. The website uses AC output for its energy balance.

The boiler switch enables heating but does not measure electricity use: the thermostat
can stop heating while the output remains enabled. Five-minute polling can miss short
transitions; enabled durations are estimates. A branch meter is needed for boiler kWh.
The 2000 W threshold is power (2 kW), not energy (kWh).

## Parquet archive, delivery and retention

`collector.archive` is a directory, containing one partition per **UTC date**:

```text
archive/
  .lock
  date=2026-11-01/samples.parquet
  date=2026-11-02/samples.parquet
```

The files contain typed numeric/boolean readings, IDs, UTC observation timestamps,
coverage flags, `uploaded_at`, and the complete original sample as JSON. They use
Zstandard compression. Each update writes a temporary file, flushes/fsyncs it,
atomically replaces the daily file, and fsyncs the directory. File permissions are
0600 and new archive directories are 0700. Thread/process locks prevent a collector,
uploader, status request or CLI replay from overwriting another operation. HTTP
requests happen outside the archive lock.

The upload request is a JSON object with `schema_version: 1` and a `samples` array,
authenticated with the bearer token named by `upload.token_env`. The website must
commit its data before responding:

```json
{"accepted_ids": ["1599c148-612c-46ab-aa9a-cf9a13271a72"]}
```

Only explicitly acknowledged IDs receive an `uploaded_at` timestamp. A plain HTTP
200, malformed response, timeout or unknown ID never clears the queue. Partial
acknowledgements mark only the accepted records; remaining records retry with
exponential backoff. Delivery is **at least once**: the website deduplicates stable
UUIDs when a response is lost, so retries and replays do not double-count energy.

`collector.retention_days` defaults to 30. Automatic cleanup runs at service startup
and hourly, deleting records acknowledged more than that many days ago. Old pending
records remain. Replay clears acknowledgement and therefore prevents expiry until
another successful transfer. The main website retains its own imported history.

```bash
.venv/bin/python -m solar_relay --config config.toml prune
.venv/bin/python -m solar_relay --config config.toml export \
  --from 2026-10-29 --until 2026-10-31 > readings.jsonl
.venv/bin/python -m solar_relay --config config.toml replay \
  --from 2026-10-29 --until 2026-10-31
.venv/bin/python -m solar_relay --config config.toml upload
```

Date bounds use `collector.timezone`, including DST transitions. `--from` is inclusive;
`--until` is exclusive. ISO timestamps may include an explicit offset. Unbounded
export/replay covers all retained observations. Export provides a consistent snapshot;
stop the service and other CLI writers before copying the archive directory as a backup.

## Recovering history after Wi-Fi loss

GEN24 records historical data for transfer to Solar.web after reconnecting. This is
separate from its local Solar API: Fronius explicitly lists archive access for GEN24
as unavailable, and this inverter's `GetArchiveData.cgi` returns 404. Inspecting its
current web interface also found no energy-history retrieval endpoint. Its SDP support
download is an encrypted `.cry` diagnostic file, not a documented energy-history API.

See the official [Solar API manual, section 5.1.1](https://www.fronius.com/~/downloads/Solar%20Energy/Operating%20Instructions/42%2C0410%2C2012.pdf)
and [Solar.web Query API manual, section 6.5](https://www.fronius.com/~/downloads/Solar%20Energy/User%20Information/SE_UI_API_InterfaceDocumentation_EN.pdf).

Optional recovery reads missing five-minute **measured energy** through the separate,
authenticated Solar.web Query API, after the inverter uploads its backlog. This uses
no public guest page. Obtain Query API access and keys from Fronius; its commercial
terms are separate from normal Solar.web access. No cloud calls occur while disabled.

Set `[history].enabled = true` and `pv_system_id` in the configuration, and put these
in the service's root-readable `/etc/solar-relay/solar-relay.env`:

```text
SOLARWEB_ACCESS_KEY_ID=your-access-key-id
SOLARWEB_ACCESS_KEY_VALUE=your-access-key-value
```

The recovery worker checks hourly, looking back seven days by default and allowing
a two-hour delay for the inverter's hourly uploads. It fetches only UTC days with
missing energy extractions, requests five energy channels, limits each request to
24 hours, and skips intervals already present. Requests use HTTPS, headers for
credentials, timeouts and no redirects. Missing/partial cloud data stays missing and
is tried again later. The configured lookback cannot exceed the retention period.

```bash
.venv/bin/python -m solar_relay --config config.toml recover \
  --from 2026-10-29 --until 2026-10-31
```

Manual recovery accepts at most 30 days per invocation and can query dates older than
the automatic lookback. Only aligned 300-second energy records with all required Wh
channels are accepted; longer aggregates are not split into invented five-minute values.
Recovery stores stable UUIDs and `history_interval` records, which the main website
uses directly without integrating them as live power. Install the corresponding
`solar` receiver update before enabling cloud recovery.

**Boiler switch history and cumulative meter readings are not recovered.** Historical
energy intervals leave those sources unknown, and the status page shows partial
coverage. Counter differences can describe a total across an outage, but cannot
recover individual five-minute values. Fully local operation leaves Wi-Fi gaps visible.

## Raspberry Pi service

Deploy the checkout to `/var/www/html/solar-relay` and create/install its venv as above.
The service reads **`/var/www/html/solar-relay/config.toml`**. Keep your existing local
file there; deployment excludes it from both replacement and deletion. If this is a
fresh installation, create it from the example only when it does not exist:

```bash
cd /var/www/html/solar-relay
test -e config.toml || cp config.example.toml config.toml
chmod 0644 config.toml
```

The `www-data` service user needs read access to this configuration. It contains
setting values and environment variable names; keep credentials in the root-readable
environment file below. Ensure the proxy serves the status endpoint rather than
repository files.

Keep the archive under `/var/www/html/solar-relay/data/archive`. The relative setting
`data/archive` resolves to that location because paths are relative to `config.toml`.
In the existing `[collector]` section, retain your other settings:

```toml
[collector]
source_id = "home-gen24"
archive = "data/archive"
timezone = "Europe/Zurich"
poll_interval_seconds = 300
retention_days = 30
```

The service uses the existing `www-data` account. Deployment creates `data/archive`
as the `www-data` runner before restarting the service. For a manual installation,
create the directory as that account before starting the service:

```bash
sudo -u www-data install -d -m 0700 \
  /var/www/html/solar-relay/data /var/www/html/solar-relay/data/archive
```

The unit keeps the checkout read-only with `ProtectSystem=strict` and allows writes
to `/var/www/html/solar-relay/data` through `ReadWritePaths`. The data directory must
exist before starting the service; otherwise systemd exits with `226/NAMESPACE`
before Python can create it. Existing data must be writable by `www-data`. Use
`sudo -u www-data` for archive-writing CLI commands to preserve ownership.

Set `[upload].url` to the main website's `https://your-solar-host/index.php?a=relay`.
Put `SOLAR_RELAY_TOKEN=your-shared-token` in `/etc/solar-relay/solar-relay.env` with
permissions 0600, readable by root. Configure the same token and source ID on the
main website as described in its README. Tokens and API keys stay out of TOML/Git.
Create `/etc/solar-relay` with `sudo install -d -m 0755 /etc/solar-relay` when adding
that optional environment file.

```bash
sudo cp deploy/solar-relay.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable solar-relay
```

Recording/uploading remains separate from serving status. The first Gitea deployment
installs the code/venv and starts the service through restart. After a manual installation,
use `systemctl start solar-relay` when the code and venv are ready. Running
tests or `sample` does not install/start the service. Initial unit installation is manual.
Inspect service logs with `sudo journalctl -u solar-relay -f` after startup.

## HTTPS reverse proxy and status page

Python listens at **`http://127.0.0.1:8765`**. Choose one proxy configuration and include
it inside the existing HTTPS virtual host for `rpioffice.nest.lutran.ch`:

- Nginx: [`deploy/nginx-solar-relay.conf`](deploy/nginx-solar-relay.conf).
- Apache: [`deploy/apache-solar-relay.conf`](deploy/apache-solar-relay.conf).
  Enable `proxy`, `proxy_http` and `headers` (`sudo a2enmod proxy proxy_http headers`).

Keep your existing TLS certificates on the proxy. Proxy the entire `/solar-relay/`
prefix; do not serve the repository/archive as static files. The collector accepts
both a stripped proxy path and a preserved `/solar-relay/` prefix. Validate the web
server configuration before reloading it (`nginx -t` or `apachectl configtest`).

The status page is **https://rpioffice.nest.lutran.ch/solar-relay/**. It shows last
extraction/last success, upload backlog, retained observations, and daily five-minute
completeness with a selectable interval grid and gap ranges. Dates use the configured
timezone. DST days have 276 or 300 intervals; ordinary days have 288.

Complete means at least one fresh, successful extraction for energy, meter and boiler
in that five-minute slot. It does not mean continuously observed power or switch state.
The unfinished current slot is excluded from missing counts. Historical recovery
contributes energy only. After retention cleanup, pruned records no longer contribute
to this local page; the website keeps the transferred history. Refresh is every 15 seconds.

Endpoints under `/solar-relay/`:

- `/`: status page.
- `/api/status?days=14`: status and extraction completeness JSON.
- `/healthz`: process liveness, also used by deployment checks.

All routes are read-only and expose no raw readings, credentials or filesystem paths.
Disable `[http].enabled` and remove the workflow's health probe if HTTP is not needed.

## Gitea deployment

`.gitea/workflows/deploy.yml` deploys pushes to `main` or manual workflow runs to
`/var/www/html/solar-relay` and restarts `solar-relay.service`. Register the Pi runner
with **`solar-relay:host`**; its checkout must be separate from production.

The runner needs Python 3.11+, `python3-venv`/pip, a supported 64-bit PyArrow platform,
`rsync`, `flock`, `curl`, `sudo`, systemd and Node.js 20+ for `actions/checkout@v4`.
The configured runner account is `www-data`. Give it write access to the deployment
directory and keep the local config there. Install the unit before the first deployment,
then add a restricted sudoers rule using the actual runner
account and the path from `command -v systemctl`:

```sh
sudo visudo -f /etc/sudoers.d/solar-relay
```

```sudoers
www-data ALL=(root) NOPASSWD: /usr/bin/systemctl restart solar-relay
```

The workflow installs dependencies and runs tests in `.venv-ci`, checks the installed
unit, serializes deployment with `flock`, syncs source, installs runtime dependencies
inside `.venv`, creates the archive directory if needed, restarts and checks the
service, then probes the public HTTPS health
endpoint with certificate verification. Runtime venv/configuration, `data/` and
Parquet files are preserved. The archive under `data/archive` is protected by the
`/data/` rsync exclusion, and the credential environment file under `/etc` is outside
the sync. The local `config.toml` is protected by the explicit rsync exclusion even
when `--delete` is used.
Deployment never copies `config.example.toml` over your configuration.

If the deployment directory is initially empty, enable the unit without `--now`; the
first deployment creates the venv and starts it. Set up the proxy before deployment.
For unit changes, install the updated file and run `systemctl daemon-reload`; the
workflow detects a mismatch and does not grant itself broader sudo access.

## Checks

```bash
.venv/bin/python -m unittest discover -s tests -v
node --check solar_relay/web/status.js
```

Tests cover normalization, real local HTTP delivery/retries, Parquet round-trips,
concurrent writers, interrupted writes, retention, DST completeness,
mocked authenticated history recovery, status routes and the deployment script.
Live Solar.web recovery requires real Query API credentials and is not exercised by tests.