solar-relay/README.md

solar-relay

Local Fronius GEN24 recorder and HTTPS relay for the sibling solar website. Python 3.11+ and a project-local virtual environment are required. License: AGPL-3.0-or-later, see LICENSE.

The collector takes one observation every five minutes, aligned to the clock (:00, :05, …). Startup and restarts wait for the next five-minute boundary. poll_interval_seconds must be 300. Each observation contains production, consumption, grid power, cumulative energy counters, boiler switch state, individual source timestamps, errors, and the original JSON responses. Requests to the inverter remain at least four seconds apart, so these readings are not simultaneous. The default inverter is http://fronius.local, meter 0, boiler pin1.

An HTTP/network request failure retries the affected endpoint up to three times (four attempts total), keeping the same request spacing. Successful endpoints are not repeated. If retries are exhausted, that source is recorded as unavailable. Retries belong to the same observation and do not reset the five-minute clock. Scheduled collection remains anchored to :00, :05, :10, … boundaries.

Observations are saved durably to Parquet before upload. Separate workers handle delivery and optional history recovery; an internet outage does not block recording. Acknowledged observations are deleted 30 days after acknowledgement. Pending observations never expire, even if their measurement dates are much older.

Installation

python3 -m venv .venv
source .venv/bin/activate
python -m pip install --only-binary=pyarrow -r requirements.txt
cp config.example.toml config.toml
python -m solar_relay --config config.toml status
python -m solar_relay --config config.toml run

Use .venv/bin/python directly if you prefer not to activate the environment. All packages stay inside the venv. Parquet storage uses the pinned PyArrow dependency in requirements.txt. Use a 64-bit Raspberry Pi OS with an available PyArrow wheel; --only-binary=pyarrow avoids a large source build. The deployment workflow installs dependencies in both its test and runtime venvs.

Configuration, credentials, .venv and Parquet data are ignored by Git. Relative paths resolve against the configuration file, rather than the working directory. Uploads default to disabled until [upload].url is configured.

Sources and measurement limits

Source	Endpoint
Solar API: PV, AC output, household load, grid power	/solar_api/v1/GetPowerFlowRealtimeData.fcgi
Solar API: grid meter counters	/solar_api/v1/GetMeterRealtimeData.cgi?Scope=System
Internal GEN24 API: boiler switch	/api/status/emrs

Solar API must be enabled in the inverter's Communication settings. The collector only reads these endpoints. The boiler endpoint is an internal firmware interface; if it stops working, boiler state is recorded as unknown rather than off.

Reading	Meaning
production_w	Site.P_PV, PV generator DC power
production_ac_w	Sum of inverter P, AC output
consumption_w	-Site.P_Load, household load on this GEN24
grid_w	Positive import, negative export at the feed-in meter
grid_import_w, grid_export_w	Nonnegative directional grid power
production_ac_total_wh	Lifetime AC production counter
grid_import_total_wh, grid_export_total_wh	Grid meter lifetime import/export counters
boiler_enabled, boiler_reason	Output state (true, false, or null) and reason code

The meter must be installed at the grid feed-in point. Missing/stale readings and source errors never become zero energy or an off boiler. DC panel power differs from AC output by conversion losses. The website uses AC output for its energy balance.

The boiler switch enables heating but does not measure electricity use: the thermostat can stop heating while the output remains enabled. Five-minute polling can miss short transitions; enabled durations are estimates. A branch meter is needed for boiler kWh. The 2000 W threshold is power (2 kW), not energy (kWh).

Parquet archive, delivery and retention

collector.archive is a directory, containing one partition per UTC date:

archive/
  .lock
  date=2026-11-01/samples.parquet
  date=2026-11-02/samples.parquet

The files contain typed numeric/boolean readings, IDs, UTC observation timestamps, coverage flags, uploaded_at, and the complete original sample as JSON. They use Zstandard compression. Each update writes a temporary file, flushes/fsyncs it, atomically replaces the daily file, and fsyncs the directory. File permissions are 0600 and new archive directories are 0700. Thread/process locks prevent a collector, uploader, status request or CLI replay from overwriting another operation. HTTP requests happen outside the archive lock.

The upload request is a JSON object with schema_version: 1 and a samples array, authenticated with the bearer token named by upload.token_env. The website must commit its data before responding:

{"accepted_ids": ["1599c148-612c-46ab-aa9a-cf9a13271a72"]}

Only explicitly acknowledged IDs receive an uploaded_at timestamp. A plain HTTP 200, malformed response, timeout or unknown ID never clears the queue. Partial acknowledgements mark only the accepted records; remaining records retry with exponential backoff. Delivery is at least once: the website deduplicates stable UUIDs when a response is lost, so retries and replays do not double-count energy.

collector.retention_days defaults to 30. Automatic cleanup runs at service startup and hourly, deleting records acknowledged more than that many days ago. Old pending records remain. Replay clears acknowledgement and therefore prevents expiry until another successful transfer. The main website retains its own imported history.

.venv/bin/python -m solar_relay --config config.toml prune
.venv/bin/python -m solar_relay --config config.toml export \
  --from 2026-10-29 --until 2026-10-31 > readings.jsonl
.venv/bin/python -m solar_relay --config config.toml replay \
  --from 2026-10-29 --until 2026-10-31
.venv/bin/python -m solar_relay --config config.toml upload

Date bounds use collector.timezone, including DST transitions. --from is inclusive; --until is exclusive. ISO timestamps may include an explicit offset. Unbounded export/replay covers all retained observations. Export provides a consistent snapshot; stop the service and other CLI writers before copying the archive directory as a backup.

Recovering history after Wi-Fi loss

GEN24 records historical data for transfer to Solar.web after reconnecting. This is separate from its local Solar API: Fronius explicitly lists archive access for GEN24 as unavailable, and this inverter's GetArchiveData.cgi returns 404. Inspecting its current web interface also found no energy-history retrieval endpoint. Its SDP support download is an encrypted .cry diagnostic file, not a documented energy-history API.

See the official Solar API manual, section 5.1.1 and Solar.web Query API manual, section 6.5.

Optional recovery reads missing five-minute measured energy through the separate, authenticated Solar.web Query API, after the inverter uploads its backlog. This uses no public guest page. Obtain Query API access and keys from Fronius; its commercial terms are separate from normal Solar.web access. No cloud calls occur while disabled.

Set [history].enabled = true and pv_system_id in the configuration, and put these in the service's root-readable /etc/solar-relay/solar-relay.env:

SOLARWEB_ACCESS_KEY_ID=your-access-key-id
SOLARWEB_ACCESS_KEY_VALUE=your-access-key-value

The recovery worker checks hourly, looking back seven days by default and allowing a two-hour delay for the inverter's hourly uploads. It fetches only UTC days with missing energy extractions, requests five energy channels, limits each request to 24 hours, and skips intervals already present. Requests use HTTPS, headers for credentials, timeouts and no redirects. Missing/partial cloud data stays missing and is tried again later. The configured lookback cannot exceed the retention period.

.venv/bin/python -m solar_relay --config config.toml recover \
  --from 2026-10-29 --until 2026-10-31

Manual recovery accepts at most 30 days per invocation and can query dates older than the automatic lookback. Only aligned 300-second energy records with all required Wh channels are accepted; longer aggregates are not split into invented five-minute values. Recovery stores stable UUIDs and history_interval records, which the main website uses directly without integrating them as live power. Install the corresponding solar receiver update before enabling cloud recovery.

Boiler switch history and cumulative meter readings are not recovered. Historical energy intervals leave those sources unknown, and the status page shows partial coverage. Counter differences can describe a total across an outage, but cannot recover individual five-minute values. Fully local operation leaves Wi-Fi gaps visible.

Raspberry Pi service

Deploy the checkout to /var/www/html/solar-relay and create/install its venv as above. The service reads /var/www/html/solar-relay/config.toml. Keep your existing local file there; deployment excludes it from both replacement and deletion. If this is a fresh installation, create it from the example only when it does not exist:

cd /var/www/html/solar-relay
test -e config.toml || cp config.example.toml config.toml
chmod 0644 config.toml

The dynamic service user needs read access to this configuration. It contains setting values and environment variable names; keep credentials in the root-readable environment file below. Ensure the proxy serves the status endpoint rather than repository files.

For systemd, set the archive to its writable state directory in this local config, retaining your other settings. The relative development setting data/archive points inside the checkout, which the hardened service cannot write:

[collector]
source_id = "home-gen24"
archive = "/var/lib/solar-relay/archive"
timezone = "Europe/Zurich"
poll_interval_seconds = 300
retention_days = 30

Set [upload].url to the main website's https://your-solar-host/index.php?a=relay. Put SOLAR_RELAY_TOKEN=your-shared-token in /etc/solar-relay/solar-relay.env with permissions 0600, readable by root. Configure the same token and source ID on the main website as described in its README. Tokens and API keys stay out of TOML/Git. Create /etc/solar-relay with sudo install -d -m 0755 /etc/solar-relay when adding that optional environment file.

sudo cp deploy/solar-relay.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable solar-relay

systemd provides /var/lib/solar-relay to a dynamic, unprivileged service user. Recording/uploading remains separate from serving status. The first Gitea deployment installs the code/venv and starts the service through restart. After a manual installation, use systemctl start solar-relay when the code and venv are ready. Running tests or sample does not install/start the service. Initial unit installation is manual. Inspect service logs with sudo journalctl -u solar-relay -f after startup.

HTTPS reverse proxy and status page

Python listens at http://127.0.0.1:8765. Choose one proxy configuration and include it inside the existing HTTPS virtual host for rpioffice.nest.lutran.ch:

• Nginx: deploy/nginx-solar-relay.conf.
• Apache: deploy/apache-solar-relay.conf. Enable proxy, proxy_http and headers (sudo a2enmod proxy proxy_http headers).

Keep your existing TLS certificates on the proxy. Proxy the entire /solar-relay/ prefix; do not serve the repository/archive as static files. The collector accepts both a stripped proxy path and a preserved /solar-relay/ prefix. Validate the web server configuration before reloading it (nginx -t or apachectl configtest).

The status page is https://rpioffice.nest.lutran.ch/solar-relay/. It shows last extraction/last success, upload backlog, retained observations, and daily five-minute completeness with a selectable interval grid and gap ranges. Dates use the configured timezone. DST days have 276 or 300 intervals; ordinary days have 288.

Complete means at least one fresh, successful extraction for energy, meter and boiler in that five-minute slot. It does not mean continuously observed power or switch state. The unfinished current slot is excluded from missing counts. Historical recovery contributes energy only. After retention cleanup, pruned records no longer contribute to this local page; the website keeps the transferred history. Refresh is every 15 seconds.

Endpoints under /solar-relay/:

• /: status page.
• /api/status?days=14: status and extraction completeness JSON.
• /healthz: process liveness, also used by deployment checks.

All routes are read-only and expose no raw readings, credentials or filesystem paths. Disable [http].enabled and remove the workflow's health probe if HTTP is not needed.

Gitea deployment

.gitea/workflows/deploy.yml deploys pushes to main or manual workflow runs to /var/www/html/solar-relay and restarts solar-relay.service. Register the Pi runner with solar-relay:host; its checkout must be separate from production.

The runner needs Python 3.11+, python3-venv/pip, a supported 64-bit PyArrow platform, rsync, flock, curl, sudo, systemd and Node.js 20+ for actions/checkout@v4. The configured runner account is www-data. Give it write access to the deployment directory and keep the local config there. Install the unit before the first deployment, then add a restricted sudoers rule using the actual runner account and the path from command -v systemctl:

www-data ALL=(root) NOPASSWD: /usr/bin/systemctl restart solar-relay.service

The workflow installs dependencies and runs tests in .venv-ci, checks the installed unit, serializes deployment with flock, syncs source, installs runtime dependencies inside .venv, restarts and checks the service, then probes the public HTTPS health endpoint with certificate verification. Runtime venv/configuration, data/ and Parquet files are preserved. Recommended state under /var/lib and the credential environment file under /etc are outside the sync. The local config.toml is protected by the explicit rsync exclusion even when --delete is used. Deployment never copies config.example.toml over your configuration.

If the deployment directory is initially empty, enable the unit without --now; the first deployment creates the venv and starts it. Set up the proxy before deployment. For unit changes, install the updated file and run systemctl daemon-reload; the workflow detects a mismatch and does not grant itself broader sudo access.

Checks

.venv/bin/python -m unittest discover -s tests -v
node --check solar_relay/web/status.js

Tests cover normalization, real local HTTP delivery/retries, Parquet round-trips, concurrent writers, interrupted writes, retention, DST completeness, mocked authenticated history recovery, status routes and the deployment script. Live Solar.web recovery requires real Query API credentials and is not exercised by tests.