diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..2a60855 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,9 @@ +# Private configuration + +`config.toml` is private and must remain secret from coding agents. + +- Never open, read, search, copy, print, diff, or otherwise inspect that file's contents with agent tools or helper scripts. +- Exclude it explicitly from repository searches, even when searching ignored files. +- The application may load its private configuration internally during user-authorized runs; this does not authorize displaying or inspecting the configuration or credentials. Use isolated tests with synthetic configuration and credentials for unit verification. +- Make configuration changes only in `config.example.toml`; the user maintains the private file. +- Do not inspect or log credentials, runtime session cookies, or authentication tokens. The user permits inspecting API/chart response data for debugging. diff --git a/README.md b/README.md index 77847e8..f81aa85 100644 --- a/README.md +++ b/README.md @@ -28,10 +28,10 @@ observations never expire, even if their measurement dates are much older. ```bash python3 -m venv .venv source .venv/bin/activate -python -m pip install --only-binary=pyarrow -r requirements.txt +python3 -m pip install --only-binary=pyarrow -r requirements.txt cp config.example.toml config.toml -python -m solar_relay --config config.toml status -python -m solar_relay --config config.toml run +python3 -m solar_relay --config config.toml status +python3 -m solar_relay --config config.toml run ``` Use `.venv/bin/python` directly if you prefer not to activate the environment. @@ -96,7 +96,8 @@ uploader, status request or CLI replay from overwriting another operation. HTTP requests happen outside the archive lock. The upload request is a JSON object with `schema_version: 1` and a `samples` array, -authenticated with the bearer token named by `upload.token_env`. The website must +authenticated with `upload.token` from private `config.toml`, or the environment +variable named by `upload.token_env` when that setting is empty. The website must commit its data before responding: ```json @@ -130,6 +131,11 @@ stop the service and other CLI writers before copying the archive directory as a ## Recovering history after Wi-Fi loss +The default mode always collects live data from the inverter's local Fronius API. +The optional backup (`[history].enabled = true`) fills historical energy gaps from +Solar.web in a separate worker. It does not replace or change live collection; +when there are no eligible gaps, it makes no Solar.web requests. + GEN24 records historical data for transfer to Solar.web after reconnecting. This is separate from its local Solar API: Fronius explicitly lists archive access for GEN24 as unavailable, and this inverter's `GetArchiveData.cgi` returns 404. Inspecting its @@ -137,36 +143,88 @@ current web interface also found no energy-history retrieval endpoint. Its SDP s download is an encrypted `.cry` diagnostic file, not a documented energy-history API. See the official [Solar API manual, section 5.1.1](https://www.fronius.com/~/downloads/Solar%20Energy/Operating%20Instructions/42%2C0410%2C2012.pdf) -and [Solar.web Query API manual, section 6.5](https://www.fronius.com/~/downloads/Solar%20Energy/User%20Information/SE_UI_API_InterfaceDocumentation_EN.pdf). +for the local API's archive limitations. -Optional recovery reads missing five-minute **measured energy** through the separate, -authenticated Solar.web Query API, after the inverter uploads its backlog. This uses -no public guest page. Obtain Query API access and keys from Fronius; its commercial -terms are separate from normal Solar.web access. No cloud calls occur while disabled. +Optional recovery signs in to Solar.web with a real Fronius account and reads its +private daily charts after the inverter uploads its backlog, using the same login +flow and chart series as the sibling `solar` website. No cloud calls occur while +disabled. A public guest link and Query API subscription/keys are not needed. -Set `[history].enabled = true` and `pv_system_id` in the configuration, and put these -in the service's root-readable `/etc/solar-relay/solar-relay.env`: +Create a dedicated Solar.web / Fronius account and grant it permission to view this +installation's metrics. Keep public sharing disabled. The account can have guest +permissions on the installation, but it signs in privately through the normal account +login; the deactivated public guest interface is never used. -```text -SOLARWEB_ACCESS_KEY_ID=your-access-key-id -SOLARWEB_ACCESS_KEY_VALUE=your-access-key-value +Set the backup account credentials directly in your private `config.toml`: + +```toml +[history] +enabled = true +pv_system_id = "your-pv-system-uuid" +username = "your-account-email" +password = "your-account-password" ``` -The recovery worker checks hourly, looking back seven days by default and allowing -a two-hour delay for the inverter's hourly uploads. It fetches only UTC days with -missing energy extractions, requests five energy channels, limits each request to -24 hours, and skips intervals already present. Requests use HTTPS, headers for -credentials, timeouts and no redirects. Missing/partial cloud data stays missing and -is tried again later. The configured lookback cannot exceed the retention period. +No extra environment file is needed for this login. Set both `username` and `password`; +an incomplete pair is rejected. Direct credentials take precedence over the environment. +Use TOML literal strings (`'...'`) if a password contains backslashes or double quotes. + +Alternatively, leave both credentials empty and set the environment variables named +by `username_env` and `password_env`. For the systemd service, these can go in its +root-readable `/etc/solar-relay/solar-relay.env`: + +```text +SOLARWEB_USERNAME=your-account-email +SOLARWEB_PASSWORD=your-account-password +``` + +While live collection runs, the separate recovery worker checks at startup and hourly +by default. It checks every five-minute energy interval back to +`collector.retention_days` (30 days by default), including entirely empty days and +dates before the first local observation. A two-hour delay allows the inverter to +upload its recent history. Only local dates containing gaps are fetched, and existing +live or recovered energy is preserved. When all eligible intervals are present, no +Solar.web requests are made. Unavailable chart days remain missing for retry and do +not prevent other dates from being recovered. Login and network failures stop that +attempt and retry on the next scheduled check. + +Set `collector.timezone` to the installation's Solar.web timezone; local chart dates +include 23/25-hour DST days. The old `history.lookback_days` setting is accepted but +ignored; `collector.retention_days` controls the automatic recovery window. + +Recovery pairs the `FromGen` Production series from the consumption view with the +`ToConsumer` Consumption series from the production view, regardless of the display +language. As in `solar`, each five-minute W value is treated as constant over that +interval and divided by 12 to obtain Wh. Self-consumption is the smaller of production +and consumption; purchase/export are their nonnegative differences. These are chart +estimates, rather than the former Query API's measured Wh channels. The original +paired power values and conversion method are retained in each recovered record. +Null or unpaired readings stay missing and are tried again later. + +Login uses HTTPS and in-memory cookies. Credentials are submitted only to the Fronius +password endpoint; SSO callbacks and redirects are restricted to the Solar.web and +Fronius login hosts. Credentials are excluded from configuration reprs and never +logged. Session cookies and authentication responses are never logged or written to +disk. An expired session is renewed once. Rejected credentials, +MFA, CAPTCHA, consent, or a changed sign-in form stop that attempt with a redacted +error. Complete any required interactive account setup in a browser. + +When migrating from Query API recovery, replace `access_key_id_env` and +`access_key_value_env` with `username` and `password` in `[history]`, or use the +environment fallback above. Obsolete configuration keys are accepted and ignored +for migration. Restart the service after updating its configuration or environment. +Recovery requires the account credentials even if API keys remain set. ```bash .venv/bin/python -m solar_relay --config config.toml recover \ --from 2026-10-29 --until 2026-10-31 ``` -Manual recovery accepts at most 30 days per invocation and can query dates older than -the automatic lookback. Only aligned 300-second energy records with all required Wh -channels are accepted; longer aggregates are not split into invented five-minute values. +Manual recovery with explicit date bounds accepts at most 30 days per invocation and +can query dates older than the automatic retention window. Unbounded recovery checks +the whole retention window, even when longer than 30 days. Only aligned five-minute +chart samples in W are accepted; +longer aggregates and unexpected units are rejected. Recovery stores stable UUIDs and `history_interval` records, which the main website uses directly without integrating them as live power. Install the corresponding `solar` receiver update before enabling cloud recovery. @@ -186,13 +244,14 @@ fresh installation, create it from the example only when it does not exist: ```bash cd /var/www/html/solar-relay test -e config.toml || cp config.example.toml config.toml -chmod 0644 config.toml +sudo chgrp www-data config.toml +chmod 0640 config.toml ``` -The `www-data` service user needs read access to this configuration. It contains -setting values and environment variable names; keep credentials in the root-readable -environment file below. Ensure the proxy serves the status endpoint rather than -repository files. +The `www-data` service user needs read access to this private configuration, which +may contain the upload token and Solar.web login credentials. Keep real credentials +out of `config.example.toml` and Git. Ensure the proxy serves the status endpoint +rather than repository files. Keep the archive under `/var/www/html/solar-relay/data/archive`. The relative setting `data/archive` resolves to that location because paths are relative to `config.toml`. @@ -222,12 +281,20 @@ exist before starting the service; otherwise systemd exits with `226/NAMESPACE` before Python can create it. Existing data must be writable by `www-data`. Use `sudo -u www-data` for archive-writing CLI commands to preserve ownership. -Set `[upload].url` to the main website's `https://your-solar-host/index.php?a=relay`. -Put `SOLAR_RELAY_TOKEN=your-shared-token` in `/etc/solar-relay/solar-relay.env` with -permissions 0600, readable by root. Configure the same token and source ID on the -main website as described in its README. Tokens and API keys stay out of TOML/Git. -Create `/etc/solar-relay` with `sudo install -d -m 0755 /etc/solar-relay` when adding -that optional environment file. +Set the upload URL and shared bearer token in private `config.toml`: + +```toml +[upload] +url = "https://your-solar-host/index.php?a=relay" +token = "your-shared-token" +``` + +Configure the same token and source ID on the main website as described in its README. +No extra environment file is needed. Alternatively, leave `token` empty and provide +the variable named by `token_env` (default `SOLAR_RELAY_TOKEN`). For systemd, put +`SOLAR_RELAY_TOKEN=your-shared-token` in `/etc/solar-relay/solar-relay.env` with +permissions 0600, readable by root. Create `/etc/solar-relay` with +`sudo install -d -m 0755 /etc/solar-relay` when adding that optional environment file. ```bash sudo cp deploy/solar-relay.service /etc/systemd/system/ @@ -322,4 +389,4 @@ node --check solar_relay/web/status.js Tests cover normalization, real local HTTP delivery/retries, Parquet round-trips, concurrent writers, interrupted writes, retention, DST completeness, mocked authenticated history recovery, status routes and the deployment script. -Live Solar.web recovery requires real Query API credentials and is not exercised by tests. +Live Solar.web recovery requires real account credentials and is not exercised by tests. diff --git a/config.example.toml b/config.example.toml index 08cb51e..96c5955 100644 --- a/config.example.toml +++ b/config.example.toml @@ -14,7 +14,7 @@ min_request_interval_seconds = 4 [upload] url = "" # Set to https://your-solar-host/index.php?a=relay to enable uploads. -token_env = "SOLAR_RELAY_TOKEN" +token = "" # Shared bearer token; enter the real value only in private config.toml. batch_size = 100 request_timeout_seconds = 10 retry_initial_seconds = 5 @@ -28,14 +28,10 @@ base_path = "/solar-relay" history_days = 14 [history] -# Optional, authenticated Solar.web Query API. No public guest link is used. -# GEN24's local Solar API has no historical-data endpoint. enabled = false pv_system_id = "" # Your Solar.web PV system UUID; required when enabled. -access_key_id_env = "SOLARWEB_ACCESS_KEY_ID" -access_key_value_env = "SOLARWEB_ACCESS_KEY_VALUE" -lookback_days = 7 +username = "" # Account email; enter real credentials only in private config.toml. +password = "" lag_seconds = 7200 # Allow the inverter to upload its hourly historical data. check_interval_seconds = 3600 -request_timeout_seconds = 10 - +request_timeout_seconds = 45 diff --git a/solar_relay/cli.py b/solar_relay/cli.py index 42ba05c..2c9c4c3 100644 --- a/solar_relay/cli.py +++ b/solar_relay/cli.py @@ -43,7 +43,8 @@ def _run(config, store): stop = threading.Event() previous = {} uploader = Uploader(config.upload, store) if config.upload.url else None - recovery = HistoryRecovery(config.history, config.collector.source_id, store) if config.history.enabled else None + recovery = HistoryRecovery(config.history, config.collector.source_id, store, + timezone_name=config.collector.timezone) if config.history.enabled else None worker = None history_worker = None status_server = None @@ -137,8 +138,9 @@ def main(argv=None): print(json.dumps({"deleted": store.prune()})) elif args.command == "recover": if not config.history.enabled: - raise ValueError("Enable history and configure Solar.web API credentials before recovery") - recovery = HistoryRecovery(config.history, config.collector.source_id, store) + raise ValueError("Enable history and configure Solar.web account credentials before recovery") + recovery = HistoryRecovery(config.history, config.collector.source_id, store, + timezone_name=config.collector.timezone) print(json.dumps({"recovered": recovery.recover(start, until)})) elif args.command == "export": for sample in store.records(start, until): diff --git a/solar_relay/config.py b/solar_relay/config.py index 7fa2228..a88de11 100644 --- a/solar_relay/config.py +++ b/solar_relay/config.py @@ -1,6 +1,6 @@ -"""Configuration contains environment variable names, never upload credentials.""" +"""Private runtime configuration; credential fields are excluded from reprs.""" -from dataclasses import dataclass +from dataclasses import dataclass, field from datetime import datetime, timezone import math from pathlib import Path @@ -9,6 +9,8 @@ import tomllib from urllib.parse import urlsplit from zoneinfo import ZoneInfo +LEGACY_HISTORY_SETTINGS = {"access_key_id_env", "access_key_value_env", "lookback_days"} + def utc_text(value: datetime) -> str: if value.tzinfo is None: @@ -52,6 +54,7 @@ class UploadConfig: request_timeout_seconds: float = 10 retry_initial_seconds: float = 5 retry_max_seconds: float = 300 + token: str = field(default="", repr=False) @dataclass(frozen=True) @@ -67,12 +70,13 @@ class HttpConfig: class HistoryConfig: enabled: bool = False pv_system_id: str = "" - access_key_id_env: str = "SOLARWEB_ACCESS_KEY_ID" - access_key_value_env: str = "SOLARWEB_ACCESS_KEY_VALUE" - lookback_days: int = 7 + username_env: str = "SOLARWEB_USERNAME" + password_env: str = "SOLARWEB_PASSWORD" lag_seconds: int = 7200 check_interval_seconds: int = 3600 - request_timeout_seconds: float = 10 + request_timeout_seconds: float = 45 + username: str = field(default="", repr=False) + password: str = field(default="", repr=False) @dataclass(frozen=True) @@ -118,7 +122,8 @@ def load_config(path: Path) -> Config: "inverter": set(InverterConfig.__dataclass_fields__), "upload": set(UploadConfig.__dataclass_fields__), "http": set(HttpConfig.__dataclass_fields__), - "history": set(HistoryConfig.__dataclass_fields__), + # Keep existing private configurations valid while using the full retention window. + "history": set(HistoryConfig.__dataclass_fields__) | LEGACY_HISTORY_SETTINGS, }.items(): if not isinstance(data.get(section, {}), dict): raise ValueError(f"{section} must be a table") @@ -157,6 +162,10 @@ def load_config(path: Path) -> Config: raise ValueError("upload.url must be a string") if upload.url: _url(upload.url, "upload.url", upload=True) + if not isinstance(upload.token, str): + raise ValueError("upload.token must be a string") + if upload.token and any(ord(c) < 33 or ord(c) > 126 for c in upload.token): + raise ValueError("upload.token must be a bearer token without whitespace") if not isinstance(upload.token_env, str) or not re.fullmatch(r"[A-Za-z_][A-Za-z0-9_]*", upload.token_env): raise ValueError("upload.token_env must be an environment variable name") if isinstance(upload.batch_size, bool) or not isinstance(upload.batch_size, int) or not 1 <= upload.batch_size <= 500: @@ -175,21 +184,25 @@ def load_config(path: Path) -> Config: raise ValueError("http.base_path must be empty or a URL path without a trailing slash") if isinstance(http.history_days, bool) or not isinstance(http.history_days, int) or not 1 <= http.history_days <= 90: raise ValueError("http.history_days must be an integer between 1 and 90") - history = HistoryConfig(**data.get("history", {})) + history = HistoryConfig(**{key: value for key, value in data.get("history", {}).items() + if key not in LEGACY_HISTORY_SETTINGS}) if not isinstance(history.enabled, bool): raise ValueError("history.enabled must be true or false") if not isinstance(history.pv_system_id, str) or (history.pv_system_id and not re.fullmatch(r"[0-9a-fA-F]{8}(?:-[0-9a-fA-F]{4}){3}-[0-9a-fA-F]{12}", history.pv_system_id)): raise ValueError("history.pv_system_id must be a UUID") if history.enabled and not history.pv_system_id: raise ValueError("Set history.pv_system_id before enabling Solar.web recovery") - for name in ("access_key_id_env", "access_key_value_env"): + for name in ("username", "password"): + if not isinstance(getattr(history, name), str): + raise ValueError(f"history.{name} must be a string") + if (history.username or history.password) and (not history.username.strip() or not history.password): + raise ValueError("Set both history.username and history.password, or leave both empty to use environment variables") + for name in ("username_env", "password_env"): if not isinstance(getattr(history, name), str) or not re.fullmatch(r"[A-Za-z_][A-Za-z0-9_]*", getattr(history, name)): raise ValueError(f"history.{name} must name an environment variable") - for name, minimum in [("lookback_days", 1), ("lag_seconds", 3600), ("check_interval_seconds", 3600)]: + for name, minimum in [("lag_seconds", 3600), ("check_interval_seconds", 3600)]: value = getattr(history, name) if isinstance(value, bool) or not isinstance(value, int) or value < minimum: raise ValueError(f"history.{name} must be an integer of at least {minimum}") - if history.lookback_days > 30 or (history.enabled and history.lookback_days > retention): - raise ValueError("history.lookback_days must be at most 30 and no longer than retention_days when enabled") _number(history.request_timeout_seconds, "history.request_timeout_seconds", 0.1) return Config(collector, inverter, upload, http, history) diff --git a/solar_relay/coverage.py b/solar_relay/coverage.py index abdef41..9f6562d 100644 --- a/solar_relay/coverage.py +++ b/solar_relay/coverage.py @@ -25,7 +25,7 @@ def _finite(value): def sample_coverage(sample): observed = milliseconds(sample["observed_at"]) if "history_interval" in sample: - # Cloud history has measured interval energy, but no cumulative meter or relay history. + # Cloud history has interval energy, but no cumulative meter or relay history. return (sample["id"], sample["source_id"], observed, observed, 1, observed, 0, observed, 0) readings = sample.get("readings", {}) result = [sample["id"], sample["source_id"], observed] diff --git a/solar_relay/history.py b/solar_relay/history.py index fc34918..286033c 100644 --- a/solar_relay/history.py +++ b/solar_relay/history.py @@ -1,19 +1,20 @@ -"""Optional private Solar.web Query API recovery of missing five-minute energy. +"""Optional signed-in Solar.web chart recovery of missing five-minute energy. -GEN24's local Solar API exposes no archive. This reads the independent cloud API -after the inverter has uploaded its internal backlog; it never requests a guest page. +GEN24's local Solar API exposes no archive. Use the same private account login, +chart series and five-minute W-to-Wh conversion as the sibling solar website. """ from datetime import datetime, timedelta, timezone import logging import math import os -from urllib.parse import urlencode from uuid import NAMESPACE_URL, uuid5 +from zoneinfo import ZoneInfo from .config import utc_text from .coverage import milliseconds -from .http import JsonHttp, TransportError +from .http import TransportError +from .solarweb import ChartDataError, SolarWebClient LOG = logging.getLogger(__name__) CHANNELS = { @@ -40,6 +41,7 @@ def history_sample(entry, source_id, system_id): when = utc_text(datetime.fromtimestamp(stamp / 1000, timezone.utc)) return { "schema_version": 1, + # Preserve IDs of intervals already recovered with the former Query API. "id": str(uuid5(NAMESPACE_URL, f"https://swqapi.solarweb.com/{system_id}/{source_id}/{when}")), "source_id": source_id, "observed_at": when, "readings": dict.fromkeys([ @@ -54,68 +56,80 @@ def history_sample(entry, source_id, system_id): } +def chart_entries(production, consumption): + """Pair real chart samples; assume constant W over five minutes, as solar does.""" + for stamp in sorted(production.keys() & consumption.keys()): + generated, consumed = production[stamp], consumption[stamp] + values = {"production": generated / 12, "consumption": consumed / 12, + "self_consumption": min(generated, consumed) / 12, + "bought": max(0, consumed - generated) / 12, "sold": max(0, generated - consumed) / 12} + yield {"logDateTime": utc_text(datetime.fromtimestamp(stamp / 1000, timezone.utc)), "logDuration": 300, + "channels": [{"channelName": name, "channelType": "Energy", "unit": "Wh", "value": values[field]} + for field, name in CHANNELS.items()], + "chart_power_w": {"production": generated, "consumption": consumed}, + "energy_method": "constant_power_over_five_minutes"} + + class HistoryRecovery: - def __init__(self, config, source_id, store, *, http=None, credentials=None): + def __init__(self, config, source_id, store, *, timezone_name="Europe/Zurich", client=None, credentials=None): self.config, self.source_id, self.store = config, source_id, store - names = (config.access_key_id_env, config.access_key_value_env) - values = credentials if credentials is not None else [os.environ.get(name, "") for name in names] - if len(values) != 2 or any(not isinstance(value, str) or not value or any(ord(c) < 33 or ord(c) > 126 for c in value) for value in values): - raise ValueError(f"Set {names[0]} and {names[1]} before enabling Solar.web recovery") - self.headers = dict(zip(("AccessKeyId", "AccessKeyValue"), values)) - self.http = http or JsonHttp(config.request_timeout_seconds) + self.zone = ZoneInfo(timezone_name) + if client is None: + names = (config.username_env, config.password_env) + if credentials is not None: + values = credentials + elif config.username or config.password: + values = (config.username, config.password) + else: + values = [os.environ.get(name, "") for name in names] + if (len(values) != 2 or any(not isinstance(value, str) or not value for value in values) + or not values[0].strip()): + raise ValueError("Set history.username and history.password in config.toml, " + f"or {names[0]} and {names[1]} before enabling Solar.web recovery") + client = SolarWebClient(*values, timeout=config.request_timeout_seconds) + self.client = client def recover(self, start=None, until=None, *, now=None, stop=None): now = now or datetime.now(timezone.utc) latest = int((now - timedelta(seconds=self.config.lag_seconds)).timestamp()) // 300 * 300 - first = self.store.status()["first_observation"] - earliest = max(int((now - timedelta(days=self.config.lookback_days)).timestamp()) // 300 * 300, - milliseconds(first) // 300_000 * 300 if first else latest) + # Check the whole retained window, including days before any live observation. + # Round up so recovery never requests an interval older than the cutoff. + earliest = math.ceil((now - timedelta(days=self.store.retention_days)).timestamp() / 300) * 300 lower = milliseconds(start) // 300_000 * 300 if start else earliest upper = min(milliseconds(until) // 300_000 * 300, latest) if until else latest if lower >= upper: return 0 - if upper - lower > 30 * 86400: + if (start is not None or until is not None) and upper - lower > 30 * 86400: raise ValueError("Recover at most 30 days at once") stamp = lambda epoch: utc_text(datetime.fromtimestamp(epoch, timezone.utc)) present = self.store.energy_slots(self.source_id, stamp(lower), stamp(upper)) missing = set(range(lower, upper, 300)) - present - # Query only UTC days containing gaps. No request spans the API's 24-hour limit. - days = sorted({epoch // 86400 * 86400 for epoch in missing}) + LOG.info("Checking %d historical energy intervals; %d are missing", (upper - lower) // 300, len(missing)) + # Chart dates use the installation's local timezone, including 23/25-hour DST days. + days = sorted({datetime.fromtimestamp(epoch, self.zone).date() for epoch in missing}) recovered = 0 for day in days: if stop and stop.is_set(): break - begin, end = max(day, lower), min(day + 86400, upper) - page = 0 - while True: + LOG.info("Recovering Solar.web energy history for %s", day) + try: + production = self.client.get_chart(self.config.pv_system_id, "production", day) if stop and stop.is_set(): break - query = urlencode({"from": stamp(begin), "to": stamp(end), "timezone": "zulu", - "channel": ",".join(CHANNELS.values()), "offset": page * 300, "limit": 300}) - url = f"https://swqapi.solarweb.com/pvsystems/{self.config.pv_system_id}/histdata?{query}" - response = self.http.request(url, extra_headers=self.headers) - system_id = response.get("pvSystemId") - if not isinstance(system_id, str) or system_id.lower() != self.config.pv_system_id.lower() or not isinstance(response.get("data"), list): - raise ValueError("Unexpected Solar.web history response") - entries = response["data"] - if len(entries) > 300: - raise ValueError("Solar.web history response exceeds the requested page size") - for entry in entries: - try: - sample = history_sample(entry, self.source_id, self.config.pv_system_id) - epoch = milliseconds(sample["observed_at"]) // 1000 - except (KeyError, TypeError, ValueError, AttributeError): - # Partial/unavailable cloud records remain missing and are tried again later. - continue - if begin <= epoch < end and epoch in missing: - self.store.add(sample) - missing.remove(epoch) - recovered += 1 - if len(entries) < 300: - break - page += 1 - if page > 2: - raise ValueError("Solar.web history pagination did not finish") + consumption = self.client.get_chart(self.config.pv_system_id, "consumption", day) + except ChartDataError: + # A day with no usable cloud data must not block recovery of newer gaps. + LOG.warning("Solar.web chart data unavailable for %s; leaving its gaps for the next check", day) + continue + for entry in chart_entries(production, consumption): + epoch = milliseconds(entry["logDateTime"]) // 1000 + if epoch not in missing or datetime.fromtimestamp(epoch, self.zone).date() != day: + continue + sample = history_sample(entry, self.source_id, self.config.pv_system_id) + self.store.add(sample) + missing.remove(epoch) + recovered += 1 + LOG.info("History check finished: %d intervals recovered, %d still missing", recovered, len(missing)) return recovered def run(self, stop): diff --git a/solar_relay/solarweb.py b/solar_relay/solarweb.py new file mode 100644 index 0000000..fec03ff --- /dev/null +++ b/solar_relay/solarweb.py @@ -0,0 +1,254 @@ +"""Private, cookie-based Solar.web chart access, matching solar's SolarWebClient.""" + +from dataclasses import dataclass, field +from html.parser import HTMLParser +from http.client import HTTPException +from http.cookiejar import CookieJar +import json +import math +import re +from urllib.error import HTTPError, URLError +from urllib.parse import unquote, urlencode, urljoin, urlsplit +from urllib.request import HTTPCookieProcessor, Request, build_opener + +from .http import NoRedirects, TransportError, _invalid_constant + +API_SERVER = "https://www.solarweb.com" +LOGIN_SERVER = "https://login.fronius.com" +LOGIN_ERROR = ("Solar.web login failed. Check the configured username and password. " + "MFA, CAPTCHA or consent may require an interactive login.") +DATASETS = {"production": ("consumption", "FromGen"), "consumption": ("production", "ToConsumer")} + + +class ChartDataError(TransportError): + """A chart day is unavailable or cannot be safely decoded; login errors are separate.""" + + +@dataclass +class Response: + status: int + url: str = field(repr=False) + body: bytes = field(default=b"", repr=False) + location: str = field(default="", repr=False) + + +class Forms(HTMLParser): + def __init__(self, body): + super().__init__() + self.forms, self.current = [], None + try: + self.feed(body.decode("utf-8")) + self.close() + except (UnicodeError, ValueError): + raise TransportError("Solar.web did not return a valid sign-in page") from None + + def handle_starttag(self, tag, attrs): + attrs = dict(attrs) + if tag == "form": + self.current = {"action": attrs.get("action") or "", "method": (attrs.get("method") or "get").lower(), + "fields": {}, "password": False, "user_input": False, "visible": False, "captcha": False} + self.forms.append(self.current) + form = self.current + if form is None: + return + if "data-sitekey" in attrs or "frc-captcha" in (attrs.get("class") or ""): + form["captcha"] = True + if tag != "input" or "disabled" in attrs: + return + name, kind = attrs.get("name"), (attrs.get("type") or "text").lower() + form["password"] |= kind == "password" and name == "password" + form["user_input"] |= name == "usernameUserInput" + form["visible"] |= kind not in ("hidden", "submit", "button") + if kind == "hidden" and name: + form["fields"][name] = attrs.get("value") or "" + + def handle_endtag(self, tag): + if tag == "form": + self.current = None + + +def origin(url): + parsed = urlsplit(url) + return f"{parsed.scheme}://{parsed.hostname}" + + +def validate_url(url): + try: + parsed = urlsplit(url) + valid = (parsed.scheme == "https" and origin(url) in (API_SERVER, LOGIN_SERVER) + and parsed.username is None and parsed.password is None and parsed.port in (None, 443) + and not any(ord(c) <= 32 or ord(c) == 127 or c == "\\" for c in url) + and not re.search(r"GuestLogOn|DemoLogOn", unquote(parsed.path), re.I)) + except ValueError: + valid = False + if not valid: + raise TransportError("Solar.web attempted an untrusted URL or guest access. Request blocked.") + + +class SolarWebClient: + MAX_RESPONSE_BYTES = 8 * 1024 * 1024 + + def __init__(self, username, password, timeout=45): + if not isinstance(username, str) or not username.strip() or not isinstance(password, str) or not password: + raise ValueError("Set the Solar.web account username and password before recovery") + self._username, self._password = username.strip(), password + self.timeout = timeout + self.cookies = CookieJar() # In memory only; sessions are never written to disk. + self.opener = build_opener(NoRedirects(), HTTPCookieProcessor(self.cookies)) + self.authenticated = False + self.chart_systems = set() + + def _send(self, url, form, referer, as_json): + headers = {"Accept": "application/json" if as_json else "text/html", + "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 " + "(KHTML, like Gecko) Chrome/156.0.0.0 Safari/537.36"} + if as_json: + headers["X-Requested-With"] = "XMLHttpRequest" + if referer: + # SSO query parameters must never appear in a cross-origin Referer. + headers["Referer"] = origin(referer) + "/" + body = None + if form is not None: + body = urlencode(form).encode("utf-8") + headers.update({"Content-Type": "application/x-www-form-urlencoded", "Origin": origin(referer)}) + request = Request(url, data=body, headers=headers) + try: + try: + response = self.opener.open(request, timeout=self.timeout) + except HTTPError as error: + response = error + with response: + raw = response.read(self.MAX_RESPONSE_BYTES + 1) + result = Response(response.code, url, raw, response.headers.get("Location", "")) + except (URLError, OSError, HTTPException, ValueError) as error: + raise TransportError(f"Solar.web network request failed ({type(error).__name__})") from None + if len(raw) > self.MAX_RESPONSE_BYTES: + raise TransportError("Solar.web response exceeds 8 MiB") + return result + + def _request(self, url, form=None, referer="", as_json=False): + for _ in range(12): + validate_url(url) + response = self._send(url, form, referer, as_json) + if response.status not in (301, 302, 303, 307, 308): + return response + if not response.location: + raise TransportError("Solar.web returned a redirect without a destination") + if form is not None and response.status in (307, 308): + raise TransportError("Solar.web attempted to redirect a sensitive POST. Request blocked.") + referer, url = url, urljoin(url, response.location) + form = None + raise TransportError("Solar.web exceeded the redirect limit") + + @staticmethod + def _signed_out(response): + path = urlsplit(response.url).path.lower() + return (response.status in (401, 403) or origin(response.url) != API_SERVER + or path.startswith("/account/externallogin") + or bool(re.search(rb'''(?:type\s*=\s*["']password["']|href\s*=\s*["']/Account/ExternalLogin(?:[?"']))''', + response.body, re.I))) + + @staticmethod + def _check_status(response): + if response.status != 200: + raise TransportError(f"Solar.web returned HTTP {response.status}. No response content was logged.") + + def _login(self): + response = self._request(API_SERVER + "/Account/ExternalLogin") + submitted = callback = False + for _ in range(6): + self._check_status(response) + if callback and not self._signed_out(response): + self.authenticated = True + return + next_form = None + for form in Forms(response.body).forms: + if form["password"]: + if submitted or form["captcha"]: + raise TransportError(LOGIN_ERROR) + action = urljoin(response.url, form["action"]) + validate_url(action) + if (origin(response.url) != LOGIN_SERVER or origin(action) != LOGIN_SERVER + or urlsplit(action).path != "/commonauth" or form["method"] != "post" + or not form["fields"].get("sessionDataKey")): + raise TransportError("Solar.web sign-in form changed; credentials were not submitted.") + fields = dict(form["fields"], username=self._username, password=self._password) + if form["user_input"]: + fields["usernameUserInput"] = self._username + next_form = action, fields + submitted = True + break + if (submitted and not form["visible"] and "state" in form["fields"] + and ("id_token" in form["fields"] or "code" in form["fields"])): + action = urljoin(response.url, form["action"]) + validate_url(action) + if (origin(response.url) != LOGIN_SERVER or origin(action) != API_SERVER + or urlsplit(action).path != "/Account/ExternalLoginCallback" or form["method"] != "post"): + raise TransportError("Solar.web sign-in callback changed; authentication was stopped.") + next_form = action, form["fields"] + callback = True + break + if next_form is None: + raise TransportError(LOGIN_ERROR) + response = self._request(*next_form, referer=response.url) + raise TransportError(LOGIN_ERROR) + + def _reset(self): + self.authenticated = False + self.chart_systems.clear() + self.cookies.clear() + + def get_chart(self, system_id, dataset, date): + view, series_id = DATASETS[dataset] + chart_url = API_SERVER + "/Chart/Chart?" + urlencode({"pvSystemId": system_id}) + query = urlencode({"pvSystemId": system_id, "year": date.year, "month": date.month, + "day": date.day, "interval": "day", "view": view}) + for _ in range(2): # Reauthenticate an expired session once; never retry rejected credentials here. + if not self.authenticated: + self._login() + if system_id not in self.chart_systems: + page = self._request(chart_url) + if self._signed_out(page): + self._reset() + continue + self._check_status(page) + self.chart_systems.add(system_id) + response = self._request(API_SERVER + "/Chart/GetChartNew?" + query, referer=chart_url, as_json=True) + if self._signed_out(response): + self._reset() + continue + self._check_status(response) + return decode_chart(response.body, series_id) + raise TransportError("Solar.web session expired or access to this PV system was denied. No guest access was attempted.") + + +def decode_chart(body, series_id): + try: + data = json.loads(body, parse_constant=_invalid_constant) + series = data["settings"]["series"] + if not isinstance(series, list): + raise ValueError + matches = [item for item in series if isinstance(item, dict) and item.get("id") == series_id] + if len(matches) != 1: + raise ValueError + item = matches[0] + if item["tooltip"]["valueSuffix"].strip() != "W" or not isinstance(item["data"], list): + raise ValueError + points, seen = {}, set() + for point in item["data"]: + if not isinstance(point, list) or len(point) != 2: + raise ValueError + stamp, watts = point + if (isinstance(stamp, bool) or not isinstance(stamp, (int, float)) + or not 0 <= stamp <= 253402300799000 or stamp % 300_000 or stamp in seen): + raise ValueError + seen.add(stamp) + if watts is None: + continue + if (isinstance(watts, bool) or not isinstance(watts, (int, float)) + or not 0 <= watts <= 1e18 or not math.isfinite(watts)): + raise ValueError + points[int(stamp)] = watts + return points + except (KeyError, TypeError, ValueError, AttributeError, UnicodeError, RecursionError): + raise ChartDataError("Solar.web chart series, samples or units changed. Import stopped; no response content was logged.") from None diff --git a/solar_relay/status.py b/solar_relay/status.py index 7f9a667..4e780ca 100644 --- a/solar_relay/status.py +++ b/solar_relay/status.py @@ -121,7 +121,7 @@ class StatusServer: "poll_interval_seconds": self.config.collector.poll_interval_seconds}, "archive": {"samples": archive["samples"], "pending": archive["pending"], "last_upload": archive["last_upload"], "upload_enabled": bool(self.config.upload.url), "format": "parquet", "retention_days": self.config.collector.retention_days}, - "history": {"enabled": self.config.history.enabled, "provider": "solarweb_query_api", + "history": {"enabled": self.config.history.enabled, "provider": "solarweb_account_charts", "boiler_history_available": False}, "days": self.store.coverage(self.config.collector.source_id, self.config.collector.timezone, days, now), } diff --git a/solar_relay/upload.py b/solar_relay/upload.py index 4d6e26d..3e553de 100644 --- a/solar_relay/upload.py +++ b/solar_relay/upload.py @@ -20,9 +20,9 @@ class Uploader: raise ValueError("Set upload.url to the website's ingestion endpoint first") self.config = config self.store = store - self.token = token if token is not None else os.environ.get(config.token_env, "") - if not self.token or any(ord(c) < 33 or ord(c) > 126 for c in self.token): - raise ValueError(f"Set {config.token_env} to a nonempty bearer token without whitespace") + self.token = token if token is not None else config.token or os.environ.get(config.token_env, "") + if not isinstance(self.token, str) or not self.token or any(ord(c) < 33 or ord(c) > 126 for c in self.token): + raise ValueError(f"Set upload.token in config.toml or {config.token_env} to a nonempty bearer token without whitespace") self.http = http or JsonHttp(config.request_timeout_seconds) def upload_once(self) -> int: diff --git a/tests/test_history.py b/tests/test_history.py index fa947c8..6348d0f 100644 --- a/tests/test_history.py +++ b/tests/test_history.py @@ -1,14 +1,16 @@ -from copy import deepcopy +from dataclasses import replace from datetime import datetime, timedelta, timezone from pathlib import Path import tempfile import unittest +from unittest.mock import Mock, patch import pyarrow.parquet as pq -from urllib.parse import parse_qs, urlsplit +from zoneinfo import ZoneInfo from solar_relay.config import HistoryConfig, utc_text -from solar_relay.history import CHANNELS, HistoryRecovery, history_sample +from solar_relay.history import CHANNELS, HistoryRecovery, chart_entries, history_sample from solar_relay.http import TransportError +from solar_relay.solarweb import ChartDataError from solar_relay.storage import Store from tests.test_status import observed @@ -22,13 +24,16 @@ def entry(when): for field, channel in CHANNELS.items()]} -class FakeHistoryHttp: - def __init__(self, entries): - self.entries, self.requests = entries, [] +class FakeCharts: + def __init__(self, when): + self.points = {dataset: {int((when + timedelta(minutes=offset)).timestamp() * 1000): watts + for offset in [0, 5, 10]} + for dataset, watts in [("production", 1200), ("consumption", 1440)]} + self.requests = [] - def request(self, url, **kwargs): - self.requests.append((url, kwargs)) - return {"pvSystemId": SYSTEM, "data": deepcopy(self.entries)} + def get_chart(self, system_id, dataset, date): + self.requests.append((system_id, dataset, date)) + return dict(self.points[dataset]) class HistoryTests(unittest.TestCase): @@ -38,9 +43,9 @@ class HistoryTests(unittest.TestCase): self.store = Store(Path(self.directory.name) / "archive") self.when = datetime(2026, 10, 30, 10, tzinfo=timezone.utc) self.source = "test-house" - self.http = FakeHistoryHttp([entry(self.when + timedelta(minutes=offset)) for offset in [0, 5, 10]]) + self.client = FakeCharts(self.when) self.config = HistoryConfig(enabled=True, pv_system_id=SYSTEM) - self.recovery = HistoryRecovery(self.config, self.source, self.store, http=self.http, credentials=("id-test", "value-test")) + self.recovery = HistoryRecovery(self.config, self.source, self.store, client=self.client) def recover(self): return self.recovery.recover(utc_text(self.when), utc_text(self.when + timedelta(minutes=15)), now=self.when + timedelta(hours=4)) @@ -56,12 +61,12 @@ class HistoryTests(unittest.TestCase): self.assertIsNone(recovered[0]["readings"]["boiler_enabled"]) self.assertNotIn("boiler", recovered[0]["source_timestamps"]) self.assertIn("boiler", recovered[0]["errors"]) - url, kwargs = self.http.requests[0] - self.assertEqual(urlsplit(url).hostname, "swqapi.solarweb.com") - self.assertEqual(kwargs["extra_headers"], {"AccessKeyId": "id-test", "AccessKeyValue": "value-test"}) - self.assertNotIn("value-test", url) + self.assertEqual(self.client.requests, [(SYSTEM, dataset, self.when.date()) + for dataset in ("production", "consumption")]) + self.assertEqual(recovered[0]["raw"]["solarweb_history"]["chart_power_w"], + {"production": 1200, "consumption": 1440}) self.assertEqual(self.recover(), 0) - self.assertEqual(len(self.http.requests), 1) + self.assertEqual(len(self.client.requests), 2) def test_cloud_energy_does_not_mark_boiler_or_meter_history_complete(self): self.assertEqual(self.recover(), 3) @@ -76,10 +81,11 @@ class HistoryTests(unittest.TestCase): self.assertEqual(row["interval_energy_wh"]["production"], 100) def test_partial_cloud_records_remain_missing_for_retry(self): - self.http.entries[1]["channels"][0]["value"] = None + stamp = int((self.when + timedelta(minutes=5)).timestamp() * 1000) + del self.client.points["production"][stamp] self.assertEqual(self.recover(), 2) self.assertEqual(self.recover(), 0) - self.http.entries[1] = entry(self.when + timedelta(minutes=5)) + self.client.points["production"][stamp] = 1200 self.assertEqual(self.recover(), 1) def test_recovered_identifiers_are_stable_for_duplicate_delivery(self): @@ -99,23 +105,129 @@ class HistoryTests(unittest.TestCase): history_sample(value, self.source, SYSTEM) def test_missing_private_credentials_fail_before_requests(self): - with self.assertRaisesRegex(ValueError, "SOLARWEB_ACCESS_KEY"): + with self.assertRaisesRegex(ValueError, "SOLARWEB_USERNAME and SOLARWEB_PASSWORD"): HistoryRecovery(self.config, self.source, self.store, credentials=("", "")) - def test_requests_are_split_at_utc_midnight_even_across_dst(self): - self.http.entries = [] + def test_config_credentials_take_precedence_over_environment(self): + config = replace(self.config, username="private@example.test", password="synthetic password é &") + with patch("solar_relay.history.os.environ", {"SOLARWEB_USERNAME": "env@example.test", "SOLARWEB_PASSWORD": "env-password"}), \ + patch("solar_relay.history.SolarWebClient", return_value=self.client) as client: + recovery = HistoryRecovery(config, self.source, self.store) + self.assertEqual(recovery.recover(utc_text(self.when), utc_text(self.when + timedelta(minutes=15)), + now=self.when + timedelta(hours=4)), 3) + client.assert_called_once_with("private@example.test", "synthetic password é &", timeout=config.request_timeout_seconds) + + def test_empty_config_credentials_use_environment_fallback(self): + config = replace(self.config, username_env="BACKUP_USER", password_env="BACKUP_PASSWORD") + with patch("solar_relay.history.os.environ", {"BACKUP_USER": "env@example.test", "BACKUP_PASSWORD": "env-password"}), \ + patch("solar_relay.history.SolarWebClient", return_value=self.client) as client: + HistoryRecovery(config, self.source, self.store) + client.assert_called_once_with("env@example.test", "env-password", timeout=config.request_timeout_seconds) + + def test_partial_config_credentials_never_mix_with_environment(self): + for changes in [{"username": "private@example.test"}, {"password": "private-password"}]: + with self.subTest(changes=changes), \ + patch("solar_relay.history.os.environ", {"SOLARWEB_USERNAME": "env@example.test", "SOLARWEB_PASSWORD": "env-password"}), \ + patch("solar_relay.history.SolarWebClient") as client: + with self.assertRaisesRegex(ValueError, "history.username and history.password"): + HistoryRecovery(replace(self.config, **changes), self.source, self.store) + client.assert_not_called() + + def test_requests_use_local_dates_across_dst(self): self.recovery.recover("2026-10-24T22:00:00Z", "2026-10-26T00:00:00Z", now=datetime(2026, 10, 27, tzinfo=timezone.utc)) - self.assertEqual(len(self.http.requests), 2) - for url, _ in self.http.requests: - query = parse_qs(urlsplit(url).query) - start, end = [datetime.fromisoformat(query[name][0]) for name in ("from", "to")] - self.assertLessEqual((end - start).total_seconds(), 86400) + self.assertEqual([date.isoformat() for _, dataset, date in self.client.requests if dataset == "production"], + ["2026-10-25", "2026-10-26"]) def test_recent_intervals_wait_for_the_hourly_cloud_upload(self): - self.http.entries = [] - self.recovery.recover(utc_text(self.when), utc_text(self.when + timedelta(hours=4)), now=self.when + timedelta(hours=4)) - query = parse_qs(urlsplit(self.http.requests[0][0]).query) - self.assertEqual(query["to"], [utc_text(self.when + timedelta(hours=2))]) + for points in self.client.points.values(): + points.clear() + for offset in [0, 115, 120, 235]: + points[int((self.when + timedelta(minutes=offset)).timestamp() * 1000)] = 1200 + self.assertEqual(self.recovery.recover(utc_text(self.when), utc_text(self.when + timedelta(hours=4)), + now=self.when + timedelta(hours=4)), 2) + + def test_automatic_recovery_checks_retention_before_first_observation_and_respects_cutoff(self): + now = datetime(2026, 11, 2, 12, 2, 17, tzinfo=timezone.utc) + cutoff = datetime(2026, 10, 3, 12, 5, tzinfo=timezone.utc) + latest = datetime(2026, 11, 2, 10, tzinfo=timezone.utc) + recoverable = [cutoff, now.replace(minute=0, second=0) - timedelta(days=25), latest - timedelta(minutes=5)] + points = {int(when.timestamp() * 1000): 1200 for when in + [cutoff - timedelta(minutes=5), *recoverable, latest]} + self.client.points = {"production": points, "consumption": points} + self.store.add(observed(now.replace(minute=0, second=0))) + self.assertEqual(self.recovery.recover(now=now), 3) + recovered = [record for record in self.store.records() if "history_interval" in record] + self.assertEqual([record["observed_at"] for record in recovered], [utc_text(when) for when in recoverable]) + self.assertEqual(min(day for _, _, day in self.client.requests), cutoff.date()) + + def test_automatic_recovery_checks_entirely_empty_days_and_archives(self): + self.store.retention_days = 2 + self.assertEqual(self.recovery.recover(now=self.when + timedelta(days=2)), 3) + self.assertEqual({day for _, _, day in self.client.requests}, + {(self.when + timedelta(days=offset)).date() for offset in range(3)}) + + def test_automatic_recovery_supports_retention_longer_than_manual_limit(self): + self.store.retention_days = 45 + now = self.when + timedelta(days=40) + self.assertEqual(self.recovery.recover(now=now), 3) + self.assertEqual(min(day for _, _, day in self.client.requests), (now - timedelta(days=45)).date()) + with self.assertRaisesRegex(ValueError, "at most 30 days"): + self.recovery.recover(utc_text(now - timedelta(days=45)), utc_text(now), now=now) + + def test_complete_retention_window_does_not_request_solarweb(self): + now = self.when + timedelta(days=1) + lower = int(self.when.timestamp()) + upper = int((now - timedelta(seconds=self.config.lag_seconds)).timestamp()) + store = Mock(retention_days=1) + store.energy_slots.return_value = set(range(lower, upper, 300)) + recovery = HistoryRecovery(self.config, self.source, store, client=self.client) + self.assertEqual(recovery.recover(now=now), 0) + self.assertEqual(self.client.requests, []) + store.add.assert_not_called() + + def test_unavailable_cloud_day_does_not_block_newer_missing_days(self): + self.store.retention_days = 2 + available = self.when + timedelta(days=1) + self.client.points = {dataset: {int(available.timestamp() * 1000): 1200} + for dataset in ("production", "consumption")} + get_chart = self.client.get_chart + + def charts(system, dataset, day): + if day == self.when.date(): + raise ChartDataError("synthetic secret response must not be logged") + return get_chart(system, dataset, day) + + with patch.object(self.client, "get_chart", side_effect=charts), self.assertLogs("solar_relay.history", level="WARNING") as logs: + self.assertEqual(self.recovery.recover(now=self.when + timedelta(days=2)), 1) + self.assertNotIn("synthetic secret response", " ".join(logs.output)) + self.assertEqual(next(self.store.records())["observed_at"], utc_text(available)) + + def test_login_or_transport_failure_stops_the_scan_without_repeated_attempts(self): + with patch.object(self.client, "get_chart", side_effect=TransportError("Login failed")) as client: + with self.assertRaisesRegex(TransportError, "Login failed"): + self.recovery.recover(now=self.when + timedelta(days=2)) + self.assertEqual(client.call_count, 1) + + def test_chart_conversion_matches_solar_and_retains_provenance(self): + for generated, consumed in [(1200, 1440), (2400, 1200), (0, 0)]: + with self.subTest(generated=generated, consumed=consumed): + stamp = int(self.when.timestamp() * 1000) + record = next(chart_entries({stamp: generated}, {stamp: consumed})) + values = history_sample(record, self.source, SYSTEM)["history_interval"]["energy_wh"] + self.assertEqual(values, {"production": generated / 12, "consumption": consumed / 12, + "self_consumption": min(generated, consumed) / 12, + "bought": max(0, consumed - generated) / 12, + "sold": max(0, generated - consumed) / 12}) + self.assertEqual(record["energy_method"], "constant_power_over_five_minutes") + + def test_dst_days_recover_every_actual_five_minute_slot(self): + for day, expected in [("2026-03-29", 276), ("2026-10-25", 300)]: + with self.subTest(day=day): + start = datetime.fromisoformat(day).replace(tzinfo=ZoneInfo("Europe/Zurich")) + end = start + timedelta(days=1) + points = {stamp: 1200 for stamp in range(int(start.timestamp() * 1000), int(end.timestamp() * 1000), 300_000)} + self.client.points = {"production": points, "consumption": points} + self.assertEqual(self.recovery.recover(utc_text(start), utc_text(end), now=end + timedelta(days=1)), expected) if __name__ == "__main__": diff --git a/tests/test_relay.py b/tests/test_relay.py index e8926c3..fd51129 100644 --- a/tests/test_relay.py +++ b/tests/test_relay.py @@ -10,7 +10,7 @@ import threading import unittest import subprocess import sys -from unittest.mock import patch +from unittest.mock import Mock, patch from urllib.parse import urlsplit from solar_relay.config import InverterConfig, UploadConfig, date_bound, load_config @@ -244,7 +244,7 @@ class DeliveryTests(unittest.TestCase): reading = client.collect("test") self.assertEqual(reading["errors"], {}) self.store.add(reading) - uploader = Uploader(UploadConfig(url=base + "/ingest"), self.store, token="test-token") + uploader = Uploader(UploadConfig(url=base + "/ingest", token="test-token"), self.store) httpd.fail_once = True with self.assertRaises(UploadError): uploader.upload_once() @@ -323,6 +323,27 @@ class DeliveryTests(unittest.TestCase): with self.assertRaisesRegex(ValueError, "SOLAR_RELAY_TOKEN"): Uploader(UploadConfig(url="https://example.test/ingest"), self.store) + def test_config_upload_token_takes_precedence_over_environment(self): + reading = sample() + self.store.add(reading) + http = Mock() + http.request.return_value = {"accepted_ids": [reading["id"]]} + with patch("solar_relay.upload.os.environ", {"SOLAR_RELAY_TOKEN": "env-token"}): + uploader = Uploader(UploadConfig(url="https://example.test/ingest", token="config-token"), self.store, http=http) + self.assertEqual(uploader.upload_once(), 1) + self.assertEqual(http.request.call_args.kwargs["token"], "config-token") + + def test_empty_config_token_uses_environment_fallback(self): + with patch("solar_relay.upload.os.environ", {"BACKUP_TOKEN": "env-token"}): + uploader = Uploader(UploadConfig(url="https://example.test/ingest", token_env="BACKUP_TOKEN"), self.store) + self.assertEqual(uploader.token, "env-token") + + def test_invalid_config_token_does_not_fall_back_or_leak_its_value(self): + with patch("solar_relay.upload.os.environ", {"SOLAR_RELAY_TOKEN": "env-token"}): + with self.assertRaises(ValueError) as raised: + Uploader(UploadConfig(url="https://example.test/ingest", token="invalid secret token"), self.store) + self.assertNotIn("invalid secret token", str(raised.exception)) + def test_cli_sample_and_export_use_the_persistent_archive(self): with server() as (_, base): config = Path(self.directory.name) / "config.toml" @@ -381,6 +402,51 @@ class ConfigTests(unittest.TestCase): with self.assertRaises(ValueError): self.load('\n[http]\nbase_path="/../config"\n') + def test_history_account_settings_and_migration_from_query_api_keys(self): + config, _ = self.load('\n[history]\nusername_env="BACKUP_USER"\npassword_env="BACKUP_PASSWORD"\n') + self.assertEqual(config.history.username_env, "BACKUP_USER") + self.assertEqual(config.history.password_env, "BACKUP_PASSWORD") + self.assertFalse(config.history.enabled) + config, _ = self.load('\n[history]\naccess_key_id_env="OLD_ID"\naccess_key_value_env="OLD_KEY"\n') + self.assertEqual(config.history.username_env, "SOLARWEB_USERNAME") + self.assertEqual(config.history.password_env, "SOLARWEB_PASSWORD") + for name in ("username_env", "password_env"): + with self.subTest(name=name), self.assertRaisesRegex(ValueError, "environment variable"): + self.load(f'\n[history]\n{name}="invalid name"\n') + + def test_history_uses_retention_instead_of_legacy_lookback_setting(self): + with tempfile.TemporaryDirectory() as directory: + path = Path(directory) / "synthetic-test.toml" + path.write_text('[collector]\nretention_days=2\n[history]\nenabled=true\n' + 'pv_system_id="73733052-6f10-47a5-9746-7f7e76ebcb8c"\nlookback_days=7\n') + config = load_config(path) + self.assertEqual(config.collector.retention_days, 2) + self.assertTrue(config.history.enabled) + self.assertFalse(hasattr(config.history, "lookback_days")) + + def test_private_config_accepts_credentials_and_redacts_them_from_repr(self): + config, _ = self.load('\n[upload]\ntoken="synthetic-shared-token"\n' + '\n[history]\nusername="synthetic@example.test"\npassword="synthetic password é &"\n') + self.assertEqual(config.upload.token, "synthetic-shared-token") + self.assertEqual(config.history.username, "synthetic@example.test") + self.assertEqual(config.history.password, "synthetic password é &") + for secret in (config.upload.token, config.history.username, config.history.password): + self.assertNotIn(secret, repr(config)) + + def test_partial_or_mistyped_private_credentials_are_rejected_without_values(self): + for fields in ['username="synthetic@example.test"', 'password="synthetic-secret"', + 'username=" "\npassword="synthetic-secret"', 'username=42', 'password=42']: + with self.subTest(fields=fields): + with self.assertRaises(ValueError) as raised: + self.load(f'\n[history]\n{fields}\n') + self.assertNotIn("synthetic-secret", str(raised.exception)) + self.assertNotIn("synthetic@example.test", str(raised.exception)) + for value in ['42', '"invalid secret token"']: + with self.subTest(value=value): + with self.assertRaisesRegex(ValueError, "upload.token") as raised: + self.load(f'\n[upload]\ntoken={value}\n') + self.assertNotIn("invalid secret token", str(raised.exception)) + if __name__ == "__main__": unittest.main() diff --git a/tests/test_schedule.py b/tests/test_schedule.py index 8cee8d7..527b378 100644 --- a/tests/test_schedule.py +++ b/tests/test_schedule.py @@ -63,6 +63,11 @@ class ScheduleTests(unittest.TestCase): self.assertEqual(clock.waits[0], (301, 299)) self.assertTrue(all(stamp % 300 == 0 for stamp in recorded)) + def test_default_live_collection_never_initializes_cloud_recovery(self): + with patch("solar_relay.cli.HistoryRecovery", side_effect=AssertionError("Unexpected cloud recovery")): + recorded, _ = self.run_collector(301) + self.assertEqual(recorded, [600, 900, 1200]) + def test_restart_in_the_same_interval_adds_no_extra_observation(self): before, _ = self.run_collector(590, count=1) after, _ = self.run_collector(615, count=2) diff --git a/tests/test_solarweb.py b/tests/test_solarweb.py new file mode 100644 index 0000000..acfc1d7 --- /dev/null +++ b/tests/test_solarweb.py @@ -0,0 +1,203 @@ +"""Exercise browser authentication with synthetic credentials and responses only.""" + +from datetime import date +from io import BytesIO +import json +import unittest +from urllib.error import URLError +from urllib.parse import parse_qs, urlsplit +from unittest.mock import Mock + +from solar_relay.http import TransportError +from solar_relay.solarweb import API_SERVER, LOGIN_SERVER, Response, SolarWebClient, decode_chart, validate_url + +SYSTEM = "73733052-6f10-47a5-9746-7f7e76ebcb8c" +DAY = date(2026, 10, 30) +STAMP = 1793354400000 +PASSWORD = "synthetic password & Unicode é" +LOGIN_FORM = b'''
+ + +
''' +CALLBACK_FORM = b'''
+ +
''' + + +def chart(series_id="FromGen", points=None, suffix=" W"): + return json.dumps({"settings": {"series": [{"id": series_id, "name": "Localized display name", + "tooltip": {"valueSuffix": suffix}, + "data": points if points is not None else [[STAMP, 1200], [STAMP + 300_000, None]]}]}}).encode() + + +def login_steps(): + return [(302, b"", LOGIN_SERVER + "/authenticationendpoint/login.do?sessionDataKey=synthetic-session"), + (200, LOGIN_FORM, ""), (200, CALLBACK_FORM, ""), + (302, b"", API_SERVER + "/"), (200, b"Account", "")] + + +class ScriptedClient(SolarWebClient): + def __init__(self, steps): + super().__init__("account@example.test", PASSWORD) + self.steps, self.requests = list(steps), [] + + def _send(self, url, form, referer, as_json): + self.requests.append((url, form, referer, as_json)) + if not self.steps: + raise AssertionError("Unexpected request") + status, body, location = self.steps.pop(0) + return Response(status, url, body, location) + + +class LoginTests(unittest.TestCase): + def test_real_account_flow_and_authenticated_charts_reuse_session(self): + client = ScriptedClient(login_steps() + [(200, b"Chart", ""), (200, chart(), ""), + (200, chart("ToConsumer"), "")]) + self.assertEqual(client.get_chart(SYSTEM, "production", DAY), {STAMP: 1200}) + self.assertEqual(client.get_chart(SYSTEM, "consumption", DAY), {STAMP: 1200}) + self.assertTrue(client.authenticated) + self.assertFalse(client.steps) + posts = [(url, form) for url, form, _, _ in client.requests if form is not None] + self.assertEqual(posts, [(LOGIN_SERVER + "/commonauth", {"sessionDataKey": "synthetic-session", + "username": "account@example.test", "usernameUserInput": "account@example.test", + "password": PASSWORD}), + (API_SERVER + "/Account/ExternalLoginCallback", + {"state": "synthetic-state", "id_token": "synthetic-token"})]) + for request, view in zip(client.requests[-2:], ["consumption", "production"]): + url, form, referer, as_json = request + query = parse_qs(urlsplit(url).query) + self.assertEqual(query, {"pvSystemId": [SYSTEM], "year": ["2026"], "month": ["10"], + "day": ["30"], "interval": ["day"], "view": [view]}) + self.assertIsNone(form) + self.assertTrue(as_json) + self.assertEqual(urlsplit(referer).path, "/Chart/Chart") + for url, _, _, _ in client.requests: + self.assertNotIn(PASSWORD, url) + self.assertNotIn("GuestLogOn", url) + self.assertNotIn("swqapi", url) + + def test_expired_chart_session_is_renewed_once(self): + client = ScriptedClient(login_steps() + [(200, b"Chart", ""), (401, b"", "")] + + login_steps() + [(200, b"Chart", ""), (200, chart(), "")]) + self.assertEqual(client.get_chart(SYSTEM, "production", DAY), {STAMP: 1200}) + self.assertEqual(sum(form is not None and "password" in form for _, form, _, _ in client.requests), 2) + + def test_persistent_denial_stops_after_one_session_renewal(self): + client = ScriptedClient(login_steps() + [(403, b"", "")] + login_steps() + [(403, b"", "")]) + with self.assertRaisesRegex(TransportError, "No guest access"): + client.get_chart(SYSTEM, "production", DAY) + self.assertFalse(client.steps) + + def test_rejected_credentials_are_submitted_only_once(self): + client = ScriptedClient(login_steps()[:2] + [(200, LOGIN_FORM, "")]) + with self.assertRaisesRegex(TransportError, "login failed"): + client.get_chart(SYSTEM, "production", DAY) + self.assertEqual(sum(form is not None for _, form, _, _ in client.requests), 1) + + def test_captcha_or_mfa_does_not_trigger_password_retries(self): + for body in [LOGIN_FORM.replace(b"", b'
'), + b'
']: + with self.subTest(body=body): + client = ScriptedClient([login_steps()[0], (200, body, "")]) + with self.assertRaisesRegex(TransportError, "login failed"): + client.get_chart(SYSTEM, "production", DAY) + self.assertFalse(any(form is not None for _, form, _, _ in client.requests)) + + def test_changed_or_untrusted_password_form_is_not_submitted(self): + bodies = [LOGIN_FORM.replace(b"/commonauth", b"https://evil.example/commonauth"), + LOGIN_FORM.replace(b"/commonauth", b"/other"), LOGIN_FORM.replace(b'"post"', b'"get"'), + LOGIN_FORM.replace(b"sessionDataKey", b"differentKey")] + for body in bodies: + with self.subTest(body=body): + client = ScriptedClient([login_steps()[0], (200, body, "")]) + with self.assertRaises(TransportError): + client.get_chart(SYSTEM, "production", DAY) + self.assertFalse(any(form is not None for _, form, _, _ in client.requests)) + + def test_untrusted_callback_is_never_sent_tokens(self): + for target in [b"https://evil.example/callback", b"https://login.fronius.com/commonauth", + b"https://www.solarweb.com/other"]: + with self.subTest(target=target): + body = CALLBACK_FORM.replace(b"https://www.solarweb.com/Account/ExternalLoginCallback", target) + client = ScriptedClient(login_steps()[:2] + [(200, body, "")]) + with self.assertRaises(TransportError): + client.get_chart(SYSTEM, "production", DAY) + self.assertEqual(sum(form is not None for _, form, _, _ in client.requests), 1) + + def test_guest_and_untrusted_redirects_are_blocked_before_network(self): + for target in [API_SERVER + "/Account/GuestLogOn?pvSystemId=x", API_SERVER + "/Account/DemoLogOn", + "https://evil.example/", "http://www.solarweb.com/"]: + with self.subTest(target=target): + client = ScriptedClient([(302, b"", target)]) + with self.assertRaisesRegex(TransportError, "Request blocked"): + client.get_chart(SYSTEM, "production", DAY) + self.assertEqual(len(client.requests), 1) + + def test_post_redirect_cannot_forward_credentials(self): + for status in [307, 308]: + with self.subTest(status=status): + client = ScriptedClient(login_steps()[:2] + [(status, b"", API_SERVER + "/")]) + with self.assertRaisesRegex(TransportError, "sensitive POST"): + client.get_chart(SYSTEM, "production", DAY) + self.assertEqual(len(client.requests), 3) + + +class TransportTests(unittest.TestCase): + def test_url_allowlist(self): + for url in [API_SERVER + "/", LOGIN_SERVER + "/commonauth", API_SERVER + ":443/Chart/Chart"]: + validate_url(url) + for url in ["https://www.solarweb.com.evil.example/", "https://user:pass@www.solarweb.com/", + API_SERVER + ":444/", "http://www.solarweb.com/", API_SERVER + "/%47uestLogOn", + API_SERVER + "/DemoLogOn", API_SERVER + "/bad\npath", API_SERVER + "\\@evil.example/"]: + with self.subTest(url=url), self.assertRaises(TransportError): + validate_url(url) + + def test_transport_redacts_network_errors_and_response_repr(self): + client = SolarWebClient("account@example.test", PASSWORD) + client.opener = Mock() + client.opener.open.side_effect = URLError("synthetic-token " + PASSWORD) + with self.assertRaises(TransportError) as raised: + client._request(API_SERVER + "/") + self.assertNotIn(PASSWORD, str(raised.exception)) + self.assertNotIn("synthetic-token", str(raised.exception)) + self.assertEqual(repr(Response(200, API_SERVER + "/?token=synthetic-token", b"secret", "secret")), "Response(status=200)") + + def test_transport_keeps_sso_queries_out_of_referer_and_encodes_password(self): + client = SolarWebClient("account@example.test", PASSWORD) + response = BytesIO(b"success") + response.code, response.headers = 200, {} + client.opener = Mock() + client.opener.open.return_value = response + client._request(LOGIN_SERVER + "/commonauth", {"password": PASSWORD}, LOGIN_SERVER + "/?state=synthetic-token") + request = client.opener.open.call_args.args[0] + self.assertEqual(request.get_header("Referer"), LOGIN_SERVER + "/") + self.assertEqual(request.get_header("Origin"), LOGIN_SERVER) + self.assertEqual(parse_qs(request.data.decode())["password"], [PASSWORD]) + + def test_responses_are_bounded(self): + client = SolarWebClient("account@example.test", PASSWORD) + response = BytesIO(b"x" * (client.MAX_RESPONSE_BYTES + 1)) + response.code, response.headers = 200, {} + client.opener = Mock() + client.opener.open.return_value = response + with self.assertRaisesRegex(TransportError, "exceeds 8 MiB"): + client._request(API_SERVER + "/") + + +class ChartTests(unittest.TestCase): + def test_series_uses_id_and_skips_null_instead_of_zeroing(self): + self.assertEqual(decode_chart(chart(), "FromGen"), {STAMP: 1200}) + + def test_bad_units_missing_series_and_invalid_points_stop_import(self): + bodies = [b"not JSON", b'{"settings":{"series":[]}}', chart("Other"), chart(suffix="kW"), + chart(points=[[STAMP + 1, 1200]]), chart(points=[[STAMP, True]]), + chart(points=[[STAMP, -1]]), chart(points=[[STAMP, float("nan")]]), + chart(points=[[STAMP, 1], [STAMP, 2]]), chart(points=[[True, 1]]), + chart(points=[[STAMP, "1200"]]), chart(points=[[STAMP]])] + for body in bodies: + with self.subTest(body=body), self.assertRaises(TransportError): + decode_chart(body, "FromGen") + + +if __name__ == "__main__": + unittest.main()