solar-relay/README.md

solar-relay

Local Fronius GEN24 recorder and HTTPS relay for the sibling solar website. Python 3.11+ and a project-local virtual environment are required. License: AGPL-3.0-or-later, see LICENSE.

The collector takes one observation every five minutes, aligned to the clock (:00, :05, …). Startup and restarts wait for the next five-minute boundary. poll_interval_seconds must be 300. Each observation contains production, consumption, grid power, cumulative energy counters, boiler switch state, individual source timestamps, errors, and the original JSON responses. Requests to the inverter remain at least four seconds apart, so these readings are not simultaneous. The default inverter is http://fronius.local, meter 0, boiler pin1.

An HTTP/network request failure retries the affected endpoint up to three times (four attempts total), keeping the same request spacing. Successful endpoints are not repeated. If retries are exhausted, that source is recorded as unavailable. Retries belong to the same observation and do not reset the five-minute clock. Scheduled collection remains anchored to :00, :05, :10, … boundaries.

Observations are saved durably to Parquet before upload. Separate workers handle delivery and optional history recovery; an internet outage does not block recording. Acknowledged observations are retained for 30 full local calendar days after acknowledgement. Cleanup removes whole observation days. A pending or recently acknowledged record keeps its whole local day, even when its measurements are much older.

Installation

python3 -m venv .venv
source .venv/bin/activate
python3 -m pip install --only-binary=pyarrow -r requirements.txt
cp config.example.toml config.toml
python3 -m solar_relay --config config.toml status
python3 -m solar_relay --config config.toml run

Use .venv/bin/python directly if you prefer not to activate the environment. All packages stay inside the venv. Parquet storage uses the pinned PyArrow dependency in requirements.txt. Use a 64-bit Raspberry Pi OS with an available PyArrow wheel; --only-binary=pyarrow avoids a large source build. The deployment workflow installs dependencies in both its test and runtime venvs.

Configuration, credentials, .venv and Parquet data are ignored by Git. Relative paths resolve against the configuration file, rather than the working directory. Uploads default to disabled until [upload].url is configured.

Sources and measurement limits

Source	Endpoint
Solar API: PV, AC output, household load, grid power	/solar_api/v1/GetPowerFlowRealtimeData.fcgi
Solar API: grid meter counters	/solar_api/v1/GetMeterRealtimeData.cgi?Scope=System
Internal GEN24 API: boiler switch	/api/status/emrs
Internal GEN24 API: boiler switch-on threshold	/api/config/emrs

Solar API must be enabled in the inverter's Communication settings. The collector only reads these endpoints. The boiler endpoint is an internal firmware interface; if it stops working, boiler state is recorded as unknown rather than off.

Reading	Meaning
production_w	Site.P_PV, PV generator DC power
production_ac_w	Sum of inverter P, AC output
consumption_w	-Site.P_Load, household load on this GEN24
grid_w	Positive import, negative export at the feed-in meter
grid_import_w, grid_export_w	Nonnegative directional grid power
production_ac_total_wh	Lifetime AC production counter
grid_import_total_wh, grid_export_total_wh	Grid meter lifetime import/export counters
boiler_enabled, boiler_reason	Output state (true, false, or null) and reason code
boiler_threshold_w, boiler_threshold_source	Switch-on threshold in watts and api or config provenance

The meter must be installed at the grid feed-in point. Missing/stale readings and source errors never become zero energy or an off boiler. DC panel power differs from AC output by conversion losses. The website uses AC output for its energy balance.

The boiler switch enables heating but does not measure electricity use: the thermostat can stop heating while the output remains enabled. Five-minute polling can miss short transitions; enabled durations are estimates. A branch meter is needed for boiler kWh. The threshold is power (W), not energy (Wh).

The collector tries /api/config/emrs once per observation and reads rules.<boiler_pin>.thresholdOn, matching the GEN24's own web interface. Negative pgrid thresholds denote exported surplus; their absolute magnitude is used. This endpoint can require authentication even while the switch state is public. If it is protected, unavailable or has no valid threshold, the collector uses [inverter].boiler_threshold_w (default 2000) from its private configuration. A settings failure does not invalidate a successfully read switch state. The threshold and its source are saved with each new observation and sent to Solar. Requests still respect the minimum four-second spacing. Existing archives remain readable; legacy switch-only samples receive the configured fallback when uploaded, without rewriting their original payload or replacing a threshold already recorded.

Parquet archive, delivery and retention

collector.archive is a directory, containing one partition per UTC date:

archive/
  .lock
  date=2026-11-01/samples.parquet
  date=2026-11-02/samples.parquet

The files contain typed numeric/boolean readings, IDs, UTC observation timestamps, coverage flags, uploaded_at, and the complete original sample as JSON. They use Zstandard compression. Each update writes a temporary file, flushes/fsyncs it, atomically replaces the daily file, and fsyncs the directory. File permissions are 0600 and new archive directories are 0700. Thread/process locks prevent a collector, uploader, status request or CLI replay from overwriting another operation. HTTP requests happen outside the archive lock.

Live and recovered records share the same Parquet table and watt columns: production_ac_w, consumption_w, grid_w, grid_import_w, and grid_export_w. Recovered chart values are stored directly in those fields; production_w (PV DC), lifetime counters, and boiler state remain unknown. record_kind and the original chart data distinguish recovered values from live snapshots. Both sources use the same validation and completeness checks. The relay stores power directly; energy totals are calculated by the receiving application.

These common power readings match the sibling solar application's existing model: its stats table stores timestamped watts for Production and Consumption, then applies 5/60 only when calculating Wh totals. production_ac_w maps to Production and consumption_w maps to Consumption. Solar currently imports only the boiler switch from these uploads; it continues to obtain production and consumption directly from Solar.web.

The upload request is a JSON object with schema_version: 1 and a samples array, authenticated with upload.token from private config.toml, or the environment variable named by upload.token_env when that setting is empty. The website must commit its data before responding:

{"result":"success","data":{"accepted_ids":["1599c148-612c-46ab-aa9a-cf9a13271a72"]},"desc_lang_id":"","desc_lang_params":[]}

Only explicitly acknowledged IDs receive an uploaded_at timestamp. A plain HTTP 200, malformed response, timeout or unknown ID never clears the queue. Partial acknowledgements mark only the accepted records; remaining records retry with exponential backoff. Delivery is at least once: a receiver must deduplicate stable UUIDs when a response is lost, so retries and replays do not double-count energy. Solar uses its standard JSON result envelope: result must be success, and acknowledgements are read from data.accepted_ids. Plain accepted_ids responses from other compatible receivers are also supported. The sibling Solar app provides index.php?a=relay as this receiver. Set [upload].url to its HTTPS URL and [upload].token to the same private value as Solar's Settings::SOLAR_RELAY_TOKEN. The collector's source_id must match Solar's Settings::SOLAR_RELAY_SOURCE_ID (default home-gen24). Restart the relay after changing its private configuration. Pending samples then upload automatically; use replay to resend retained samples already acknowledged elsewhere.

Solar stores five-minute boiler stats in watts: the recorded threshold when enabled, zero when off. It calculates daily Wh by summing watts × 5/60 and uses those totals for monthly and yearly charts. It acknowledges other readings without importing them, including historical recovery records whose boiler state is unknown. Solar's existing Solar.web import continues to supply production and consumption.

collector.retention_days defaults to 30. All retention cutoffs are midnight in collector.timezone. For example, on November 2 with 30 days retained, the cutoff is October 3 at 00:00 local time, regardless of the execution time. The entire October 3 day is kept and eligible for history recovery.

Automatic cleanup runs at service startup and hourly. It removes an older local observation day only when every record in that day was acknowledged before the cutoff. Pending or recently acknowledged records protect the entire day, including its rows in adjacent UTC Parquet partitions. Replay clears acknowledgement and therefore prevents expiry until another successful transfer. The main website retains its own imported history.

.venv/bin/python -m solar_relay --config config.toml prune
.venv/bin/python -m solar_relay --config config.toml export \
  --from 2026-10-29 --until 2026-10-31 > readings.jsonl
.venv/bin/python -m solar_relay --config config.toml replay \
  --from 2026-10-29 --until 2026-10-31
.venv/bin/python -m solar_relay --config config.toml upload

Date bounds use collector.timezone, including DST transitions. --from is inclusive; --until is exclusive. ISO timestamps may include an explicit offset. Unbounded export/replay covers all retained observations. Export provides a consistent snapshot; stop the service and other CLI writers before copying the archive directory as a backup.

Recovering history after Wi-Fi loss

The default mode always collects live data from the inverter's local Fronius API. The optional backup ([history].enabled = true) fills historical energy gaps from Solar.web in a separate worker. It does not replace or change live collection; when there are no eligible gaps, it makes no Solar.web requests.

GEN24 records historical data for transfer to Solar.web after reconnecting. This is separate from its local Solar API: Fronius explicitly lists archive access for GEN24 as unavailable, and this inverter's GetArchiveData.cgi returns 404. Inspecting its current web interface also found no energy-history retrieval endpoint. Its SDP support download is an encrypted .cry diagnostic file, not a documented energy-history API.

See the official Solar API manual, section 5.1.1 for the local API's archive limitations.

Optional recovery signs in to Solar.web with a real Fronius account and reads its private daily charts after the inverter uploads its backlog, using the same login flow and chart series as the sibling solar website. No cloud calls occur while disabled. A public guest link is not needed.

Create a dedicated Solar.web / Fronius account and grant it permission to view this installation's metrics. Keep public sharing disabled. The account can have guest permissions on the installation, but it signs in privately through the normal account login; the deactivated public guest interface is never used.

Set the backup account credentials directly in your private config.toml:

[history]
enabled = true
pv_system_id = "your-pv-system-uuid"
username = "your-account-email"
password = "your-account-password"

No extra environment file is needed for this login. Set both username and password; an incomplete pair is rejected. Direct credentials take precedence over the environment. Use TOML literal strings ('...') if a password contains backslashes or double quotes.

Alternatively, leave both credentials empty and set the environment variables named by username_env and password_env. For the systemd service, these can go in its root-readable /etc/solar-relay/solar-relay.env:

SOLARWEB_USERNAME=your-account-email
SOLARWEB_PASSWORD=your-account-password

While live collection runs, the separate recovery worker checks at startup and hourly by default. It checks every five-minute energy interval from midnight at the start of the date collector.retention_days ago (30 days by default), including empty days and dates before the first local observation. A two-hour delay allows the inverter to upload its recent history. Only local dates containing gaps are fetched, and existing live or recovered energy is preserved. When all eligible intervals are present, no Solar.web requests are made. Unavailable chart days remain missing for retry and do not prevent other dates from being recovered. Login and network failures stop that attempt and retry on the next scheduled check.

Set collector.timezone to the installation's Solar.web timezone; local chart dates include whole 23/25-hour DST days. collector.retention_days controls the automatic recovery window.

Recovery pairs the FromGen Production series from the consumption view with the ToConsumer Consumption series from the production view, regardless of the display language. Paired power values are stored directly in the common watt readings. Grid import/export are their nonnegative differences. The relay keeps the original chart watts without converting them to interval energy. Null or unpaired readings stay missing and are tried again later.

Login uses HTTPS and in-memory cookies. Credentials are submitted only to the Fronius password endpoint; SSO callbacks and redirects are restricted to the Solar.web and Fronius login hosts. Credentials are excluded from configuration reprs and never logged. Session cookies and authentication responses are never logged or written to disk. An expired session is renewed once. Rejected credentials, MFA, CAPTCHA, consent, or a changed sign-in form stop that attempt with a redacted error. Complete any required interactive account setup in a browser.

.venv/bin/python -m solar_relay --config config.toml recover \
  --from 2026-10-29 --until 2026-10-31

Manual recovery with explicit date bounds accepts at most 30 local calendar days per invocation and can query dates older than the automatic retention window. Unbounded recovery checks the whole retention window, even when longer than 30 days. Only aligned five-minute chart samples in W are accepted; longer aggregates and unexpected units are rejected. Recovery stores stable UUIDs and common power readings with their chart provenance. A future solar-side importer can put both live and recovered production/consumption in the existing stats watt table and use its normal five-minute Wh conversion exactly once.

Boiler switch history and cumulative meter readings are not recovered. Historical energy intervals leave those sources unknown, and the status page shows partial coverage. Counter differences can describe a total across an outage, but cannot recover individual five-minute values. Fully local operation leaves Wi-Fi gaps visible.

Raspberry Pi service

Deploy the checkout to /var/www/html/solar-relay and create/install its venv as above. The service reads /var/www/html/solar-relay/config.toml. Keep your existing local file there; deployment excludes it from both replacement and deletion. If this is a fresh installation, create it from the example only when it does not exist:

cd /var/www/html/solar-relay
test -e config.toml || cp config.example.toml config.toml
sudo chgrp www-data config.toml
chmod 0640 config.toml

The www-data service user needs read access to this private configuration, which may contain the upload token and Solar.web login credentials. Keep real credentials out of config.example.toml and Git. Ensure the proxy serves the status endpoint rather than repository files.

Keep the archive under /var/www/html/solar-relay/data/archive. The relative setting data/archive resolves to that location because paths are relative to config.toml. In the existing [collector] section, retain your other settings:

[collector]
source_id = "home-gen24"
archive = "data/archive"
timezone = "Europe/Zurich"
poll_interval_seconds = 300
retention_days = 30

The service uses the existing www-data account. Deployment creates data/archive as the www-data runner before restarting the service. For a manual installation, create the directory as that account before starting the service:

sudo -u www-data install -d -m 0700 \
  /var/www/html/solar-relay/data /var/www/html/solar-relay/data/archive

The unit keeps the checkout read-only with ProtectSystem=strict and allows writes to /var/www/html/solar-relay/data through ReadWritePaths. The data directory must exist before starting the service; otherwise systemd exits with 226/NAMESPACE before Python can create it. Existing data must be writable by www-data. Use sudo -u www-data for archive-writing CLI commands to preserve ownership.

Set the upload URL and shared bearer token in private config.toml:

[upload]
url = "https://your-solar-host/index.php?a=relay"
token = "your-shared-token"

Configure the same token and source ID on the main website as described in its README. No extra environment file is needed. Alternatively, leave token empty and provide the variable named by token_env (default SOLAR_RELAY_TOKEN). For systemd, put SOLAR_RELAY_TOKEN=your-shared-token in /etc/solar-relay/solar-relay.env with permissions 0600, readable by root. Create /etc/solar-relay with sudo install -d -m 0755 /etc/solar-relay when adding that optional environment file.

sudo cp deploy/solar-relay.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable solar-relay

Recording/uploading remains separate from serving status. The first Gitea deployment installs the code/venv and starts the service through restart. After a manual installation, use systemctl start solar-relay when the code and venv are ready. Running tests or sample does not install/start the service. Initial unit installation is manual. Inspect service logs with sudo journalctl -u solar-relay -f after startup.

HTTPS reverse proxy and status page

Python listens at http://127.0.0.1:8765. Choose one proxy configuration and include it inside the existing HTTPS virtual host for rpioffice.nest.lutran.ch:

• Nginx: deploy/nginx-solar-relay.conf.
• Apache: deploy/apache-solar-relay.conf. Enable proxy, proxy_http and headers (sudo a2enmod proxy proxy_http headers).

Keep your existing TLS certificates on the proxy. Proxy the entire /solar-relay/ prefix; do not serve the repository/archive as static files. The collector accepts both a stripped proxy path and a preserved /solar-relay/ prefix. Validate the web server configuration before reloading it (nginx -t or apachectl configtest).

The status page is https://rpioffice.nest.lutran.ch/solar-relay/. It shows last extraction/last success, upload backlog, retained observations, and daily five-minute completeness with a selectable interval grid and gap ranges. Dates use the configured timezone. DST days have 276 or 300 intervals; ordinary days have 288.

The five-minute interval card appears above daily coverage. Status shows Energy and Boiler; meter counters remain archived but do not affect status, completeness, or interval details. Complete means valid energy and boiler data in the same five-minute slot. It does not mean continuously observed power or switch state. The current slot shows its saved extraction immediately, including partial or failed results. Until a record arrives, it is pending and excluded from missing counts. Future slots remain upcoming. Historical recovery contributes energy only. After retention cleanup, pruned records no longer contribute to this local page; the website keeps the transferred history. Refresh is every 15 seconds.

Endpoints under /solar-relay/:

• /: status page.
• /api/status?days=14: status and extraction completeness JSON.
• /healthz: process liveness, also used by deployment checks.

All routes are read-only and expose no raw readings, credentials or filesystem paths. Disable [http].enabled and remove the workflow's health probe if HTTP is not needed.

Gitea deployment

.gitea/workflows/deploy.yml deploys pushes to main or manual workflow runs to /var/www/html/solar-relay and restarts solar-relay.service. Register the Pi runner with solar-relay:host; its checkout must be separate from production.

The runner needs Python 3.11+, python3-venv/pip, a supported 64-bit PyArrow platform, rsync, flock, curl, sudo, systemd and Node.js 20+ for actions/checkout@v4. The configured runner account is www-data. Give it write access to the deployment directory and keep the local config there. Install the unit before the first deployment, then add a restricted sudoers rule using the actual runner account and the path from command -v systemctl:

sudo visudo -f /etc/sudoers.d/solar-relay

www-data ALL=(root) NOPASSWD: /usr/bin/systemctl restart solar-relay

The workflow installs dependencies and runs tests in .venv-ci, checks the installed unit, serializes deployment with flock, syncs source, installs runtime dependencies inside .venv, creates the archive directory if needed, restarts and checks the service, then probes the public HTTPS health endpoint with certificate verification. Runtime venv/configuration, data/ and Parquet files are preserved. The archive under data/archive is protected by the /data/ rsync exclusion, and the credential environment file under /etc is outside the sync. The local config.toml is protected by the explicit rsync exclusion even when --delete is used. Deployment never copies config.example.toml over your configuration.

If the deployment directory is initially empty, enable the unit without --now; the first deployment creates the venv and starts it. Set up the proxy before deployment. For unit changes, install the updated file and run systemctl daemon-reload; the workflow detects a mismatch and does not grant itself broader sudo access.

Checks

.venv/bin/python -m unittest discover -s tests -v
node --check solar_relay/web/status.js

Tests cover normalization, real local HTTP delivery/retries, Parquet round-trips, concurrent writers, interrupted writes, retention, DST completeness, mocked authenticated history recovery, status routes and the deployment script. Live Solar.web recovery requires real account credentials and is not exercised by tests.