Initial commit

README.md

@@ -1,3 +1,289 @@
 # solar-relay
 
-Fronius data extractor and relay for the solar app
+Local Fronius GEN24 recorder and HTTPS relay for the sibling `solar` website.
+Python 3.11+ and a project-local virtual environment are required. License:
+AGPL-3.0-or-later, see [LICENSE](LICENSE).
+
+The collector takes one observation every **five minutes**, aligned to the clock
+(`:00`, `:05`, …). Startup and restarts wait for the next five-minute boundary.
+`poll_interval_seconds` must be 300. Each observation
+contains production, consumption, grid power, cumulative energy counters, boiler
+switch state, individual source timestamps, errors, and the original JSON responses.
+Requests to the inverter remain at least four seconds apart, so these readings are
+not simultaneous. The default inverter is `http://fronius.local`, meter `0`, boiler `pin1`.
+
+An HTTP/network request failure retries the affected endpoint up to three times
+(four attempts total), keeping the same request spacing. Successful endpoints are
+not repeated. If retries are exhausted, that source is recorded as unavailable.
+Retries belong to the same observation and do not reset the five-minute clock.
+Scheduled collection remains anchored to `:00`, `:05`, `:10`, … boundaries.
+
+Observations are saved durably to **Parquet** before upload. Separate workers handle
+delivery and optional history recovery; an internet outage does not block recording.
+Acknowledged observations are deleted **30 days after acknowledgement**. Pending
+observations never expire, even if their measurement dates are much older.
+
+## Installation
+
+```bash
+python3 -m venv .venv
+source .venv/bin/activate
+python -m pip install --only-binary=pyarrow -r requirements.txt
+cp config.example.toml config.toml
+python -m solar_relay --config config.toml status
+python -m solar_relay --config config.toml run
+```
+
+Use `.venv/bin/python` directly if you prefer not to activate the environment.
+All packages stay inside the venv. Parquet storage uses the pinned **PyArrow**
+dependency in `requirements.txt`. Use a 64-bit Raspberry Pi OS
+with an available PyArrow wheel; `--only-binary=pyarrow` avoids a large source build.
+The deployment workflow installs dependencies in both its test and runtime venvs.
+
+Configuration, credentials, `.venv` and Parquet data are ignored by Git.
+Relative paths resolve against the configuration file, rather than the working directory.
+Uploads default to disabled until `[upload].url` is configured.
+
+## Sources and measurement limits
+
+| Source | Endpoint |
+| --- | --- |
+| Solar API: PV, AC output, household load, grid power | `/solar_api/v1/GetPowerFlowRealtimeData.fcgi` |
+| Solar API: grid meter counters | `/solar_api/v1/GetMeterRealtimeData.cgi?Scope=System` |
+| Internal GEN24 API: boiler switch | `/api/status/emrs` |
+
+Solar API must be enabled in the inverter's Communication settings. The collector
+only reads these endpoints. The boiler endpoint is an internal firmware interface;
+if it stops working, boiler state is recorded as unknown rather than off.
+
+| Reading | Meaning |
+| --- | --- |
+| `production_w` | `Site.P_PV`, PV generator DC power |
+| `production_ac_w` | Sum of inverter `P`, AC output |
+| `consumption_w` | `-Site.P_Load`, household load on this GEN24 |
+| `grid_w` | Positive import, negative export at the feed-in meter |
+| `grid_import_w`, `grid_export_w` | Nonnegative directional grid power |
+| `production_ac_total_wh` | Lifetime AC production counter |
+| `grid_import_total_wh`, `grid_export_total_wh` | Grid meter lifetime import/export counters |
+| `boiler_enabled`, `boiler_reason` | Output state (`true`, `false`, or `null`) and reason code |
+
+The meter must be installed at the grid feed-in point. Missing/stale readings and
+source errors never become zero energy or an off boiler. DC panel power differs
+from AC output by conversion losses. The website uses AC output for its energy balance.
+
+The boiler switch enables heating but does not measure electricity use: the thermostat
+can stop heating while the output remains enabled. Five-minute polling can miss short
+transitions; enabled durations are estimates. A branch meter is needed for boiler kWh.
+The 2000 W threshold is power (2 kW), not energy (kWh).
+
+## Parquet archive, delivery and retention
+
+`collector.archive` is a directory, containing one partition per **UTC date**:
+
+```text
+archive/
+  .lock
+  date=2026-11-01/samples.parquet
+  date=2026-11-02/samples.parquet
+```
+
+The files contain typed numeric/boolean readings, IDs, UTC observation timestamps,
+coverage flags, `uploaded_at`, and the complete original sample as JSON. They use
+Zstandard compression. Each update writes a temporary file, flushes/fsyncs it,
+atomically replaces the daily file, and fsyncs the directory. File permissions are
+0600 and new archive directories are 0700. Thread/process locks prevent a collector,
+uploader, status request or CLI replay from overwriting another operation. HTTP
+requests happen outside the archive lock.
+
+The upload request is a JSON object with `schema_version: 1` and a `samples` array,
+authenticated with the bearer token named by `upload.token_env`. The website must
+commit its data before responding:
+
+```json
+{"accepted_ids": ["1599c148-612c-46ab-aa9a-cf9a13271a72"]}
+```
+
+Only explicitly acknowledged IDs receive an `uploaded_at` timestamp. A plain HTTP
+200, malformed response, timeout or unknown ID never clears the queue. Partial
+acknowledgements mark only the accepted records; remaining records retry with
+exponential backoff. Delivery is **at least once**: the website deduplicates stable
+UUIDs when a response is lost, so retries and replays do not double-count energy.
+
+`collector.retention_days` defaults to 30. Automatic cleanup runs at service startup
+and hourly, deleting records acknowledged more than that many days ago. Old pending
+records remain. Replay clears acknowledgement and therefore prevents expiry until
+another successful transfer. The main website retains its own imported history.
+
+```bash
+.venv/bin/python -m solar_relay --config config.toml prune
+.venv/bin/python -m solar_relay --config config.toml export \
+  --from 2026-10-29 --until 2026-10-31 > readings.jsonl
+.venv/bin/python -m solar_relay --config config.toml replay \
+  --from 2026-10-29 --until 2026-10-31
+.venv/bin/python -m solar_relay --config config.toml upload
+```
+
+Date bounds use `collector.timezone`, including DST transitions. `--from` is inclusive;
+`--until` is exclusive. ISO timestamps may include an explicit offset. Unbounded
+export/replay covers all retained observations. Export provides a consistent snapshot;
+stop the service and other CLI writers before copying the archive directory as a backup.
+
+## Recovering history after Wi-Fi loss
+
+GEN24 records historical data for transfer to Solar.web after reconnecting. This is
+separate from its local Solar API: Fronius explicitly lists archive access for GEN24
+as unavailable, and this inverter's `GetArchiveData.cgi` returns 404. Inspecting its
+current web interface also found no energy-history retrieval endpoint. Its SDP support
+download is an encrypted `.cry` diagnostic file, not a documented energy-history API.
+
+See the official [Solar API manual, section 5.1.1](https://www.fronius.com/~/downloads/Solar%20Energy/Operating%20Instructions/42%2C0410%2C2012.pdf)
+and [Solar.web Query API manual, section 6.5](https://www.fronius.com/~/downloads/Solar%20Energy/User%20Information/SE_UI_API_InterfaceDocumentation_EN.pdf).
+
+Optional recovery reads missing five-minute **measured energy** through the separate,
+authenticated Solar.web Query API, after the inverter uploads its backlog. This uses
+no public guest page. Obtain Query API access and keys from Fronius; its commercial
+terms are separate from normal Solar.web access. No cloud calls occur while disabled.
+
+Set `[history].enabled = true` and `pv_system_id` in the configuration, and put these
+in the service's root-readable `/etc/solar-relay/solar-relay.env`:
+
+```text
+SOLARWEB_ACCESS_KEY_ID=your-access-key-id
+SOLARWEB_ACCESS_KEY_VALUE=your-access-key-value
+```
+
+The recovery worker checks hourly, looking back seven days by default and allowing
+a two-hour delay for the inverter's hourly uploads. It fetches only UTC days with
+missing energy extractions, requests five energy channels, limits each request to
+24 hours, and skips intervals already present. Requests use HTTPS, headers for
+credentials, timeouts and no redirects. Missing/partial cloud data stays missing and
+is tried again later. The configured lookback cannot exceed the retention period.
+
+```bash
+.venv/bin/python -m solar_relay --config config.toml recover \
+  --from 2026-10-29 --until 2026-10-31
+```
+
+Manual recovery accepts at most 30 days per invocation and can query dates older than
+the automatic lookback. Only aligned 300-second energy records with all required Wh
+channels are accepted; longer aggregates are not split into invented five-minute values.
+Recovery stores stable UUIDs and `history_interval` records, which the main website
+uses directly without integrating them as live power. Install the corresponding
+`solar` receiver update before enabling cloud recovery.
+
+**Boiler switch history and cumulative meter readings are not recovered.** Historical
+energy intervals leave those sources unknown, and the status page shows partial
+coverage. Counter differences can describe a total across an outage, but cannot
+recover individual five-minute values. Fully local operation leaves Wi-Fi gaps visible.
+
+## Raspberry Pi service
+
+Deploy the checkout to `/var/www/html/solar-relay` and create/install its venv as above.
+Place configuration outside the web directory:
+
+```bash
+sudo install -d -m 0755 /etc/solar-relay
+sudo install -m 0644 config.example.toml /etc/solar-relay/config.toml
+sudoedit /etc/solar-relay/config.toml
+```
+
+Use these collector settings, retaining the example's other sections:
+
+```toml
+[collector]
+source_id = "home-gen24"
+archive = "/var/lib/solar-relay/archive"
+timezone = "Europe/Zurich"
+poll_interval_seconds = 300
+retention_days = 30
+```
+
+Set `[upload].url` to the main website's `https://your-solar-host/index.php?a=relay`.
+Put `SOLAR_RELAY_TOKEN=your-shared-token` in `/etc/solar-relay/solar-relay.env` with
+permissions 0600, readable by root. Configure the same token and source ID on the
+main website as described in its README. Tokens and API keys stay out of TOML/Git.
+
+```bash
+sudo cp deploy/solar-relay.service /etc/systemd/system/
+sudo systemctl daemon-reload
+sudo systemctl enable --now solar-relay
+sudo journalctl -u solar-relay -f
+```
+
+systemd provides `/var/lib/solar-relay` to a dynamic, unprivileged service user.
+Recording/uploading remains separate from serving status. Running tests or `sample`
+does not install/start the service. Initial systemd/configuration installation is manual.
+
+## HTTPS reverse proxy and status page
+
+Python listens at **`http://127.0.0.1:8765`**. Choose one proxy configuration and include
+it inside the existing HTTPS virtual host for `rpioffice.nest.lutran.ch`:
+
+- Nginx: [`deploy/nginx-solar-relay.conf`](deploy/nginx-solar-relay.conf).
+- Apache: [`deploy/apache-solar-relay.conf`](deploy/apache-solar-relay.conf).
+  Enable `proxy`, `proxy_http` and `headers` (`sudo a2enmod proxy proxy_http headers`).
+
+Keep your existing TLS certificates on the proxy. Proxy the entire `/solar-relay/`
+prefix; do not serve the repository/archive as static files. The collector accepts
+both a stripped proxy path and a preserved `/solar-relay/` prefix. Validate the web
+server configuration before reloading it (`nginx -t` or `apachectl configtest`).
+
+The status page is **https://rpioffice.nest.lutran.ch/solar-relay/**. It shows last
+extraction/last success, upload backlog, retained observations, and daily five-minute
+completeness with a selectable interval grid and gap ranges. Dates use the configured
+timezone. DST days have 276 or 300 intervals; ordinary days have 288.
+
+Complete means at least one fresh, successful extraction for energy, meter and boiler
+in that five-minute slot. It does not mean continuously observed power or switch state.
+The unfinished current slot is excluded from missing counts. Historical recovery
+contributes energy only. After retention cleanup, pruned records no longer contribute
+to this local page; the website keeps the transferred history. Refresh is every 15 seconds.
+
+Endpoints under `/solar-relay/`:
+
+- `/`: status page.
+- `/api/status?days=14`: status and extraction completeness JSON.
+- `/healthz`: process liveness, also used by deployment checks.
+
+All routes are read-only and expose no raw readings, credentials or filesystem paths.
+Disable `[http].enabled` and remove the workflow's health probe if HTTP is not needed.
+
+## Gitea deployment
+
+`.gitea/workflows/deploy.yml` deploys pushes to `main` or manual workflow runs to
+`/var/www/html/solar-relay` and restarts `solar-relay.service`. Register the Pi runner
+with **`solar-relay:host`**; its checkout must be separate from production.
+
+The runner needs Python 3.11+, `python3-venv`/pip, a supported 64-bit PyArrow platform,
+`rsync`, `flock`, `curl`, `sudo`, systemd and Node.js 20+ for `actions/checkout@v4`.
+Give its account write access to the deployment directory. Install the config and unit
+before the first deployment, then add a restricted sudoers rule using the actual runner
+account and the path from `command -v systemctl`:
+
+```sudoers
+gitea-runner ALL=(root) NOPASSWD: /usr/bin/systemctl restart solar-relay.service
+```
+
+The workflow installs dependencies and runs tests in `.venv-ci`, checks the installed
+unit, serializes deployment with `flock`, syncs source, installs runtime dependencies
+inside `.venv`, restarts and checks the service, then probes the public HTTPS health
+endpoint with certificate verification. Runtime venv/configuration, `data/` and
+Parquet files are preserved. Recommended state under `/var/lib` and
+configuration under `/etc` are entirely outside the sync and web directory.
+
+If the deployment directory is initially empty, enable the unit without `--now`; the
+first deployment creates the venv and starts it. Set up the proxy before deployment.
+For unit changes, install the updated file and run `systemctl daemon-reload`; the
+workflow detects a mismatch and does not grant itself broader sudo access.
+
+## Checks
+
+```bash
+.venv/bin/python -m unittest discover -s tests -v
+node --check solar_relay/web/status.js
+```
+
+Tests cover normalization, real local HTTP delivery/retries, Parquet round-trips,
+concurrent writers, interrupted writes, retention, DST completeness,
+mocked authenticated history recovery, status routes and the deployment script.
+Live Solar.web recovery requires real Query API credentials and is not exercised by tests.