This commit is contained in:
1 parent
cd7ddd3cf6
commit
89621bc7c3
15 files changed
+289
-74
No files matched your search
@@ -20,8 +20,9 @@ Scheduled collection remains anchored to `:00`, `:05`, `:10`, … boundaries.
|
||||
|
||||
Observations are saved durably to **Parquet** before upload. Separate workers handle
|
||||
delivery and optional history recovery; an internet outage does not block recording.
|
||||
Acknowledged observations are deleted **30 days after acknowledgement**. Pending
|
||||
observations never expire, even if their measurement dates are much older.
|
||||
Acknowledged observations are retained for **30 full local calendar days after
|
||||
acknowledgement**. Cleanup removes whole observation days. A pending or recently
|
||||
acknowledged record keeps its whole local day, even when its measurements are much older.
|
||||
|
||||
## Installation
|
||||
|
||||
@@ -110,10 +111,17 @@ acknowledgements mark only the accepted records; remaining records retry with
|
||||
exponential backoff. Delivery is **at least once**: the website deduplicates stable
|
||||
UUIDs when a response is lost, so retries and replays do not double-count energy.
|
||||
|
||||
`collector.retention_days` defaults to 30. Automatic cleanup runs at service startup
|
||||
and hourly, deleting records acknowledged more than that many days ago. Old pending
|
||||
records remain. Replay clears acknowledgement and therefore prevents expiry until
|
||||
another successful transfer. The main website retains its own imported history.
|
||||
`collector.retention_days` defaults to 30. All retention cutoffs are midnight in
|
||||
`collector.timezone`. For example, on November 2 with 30 days retained, the cutoff
|
||||
is October 3 at 00:00 local time, regardless of the execution time. The entire
|
||||
October 3 day is kept and eligible for history recovery.
|
||||
|
||||
Automatic cleanup runs at service startup and hourly. It removes an older local
|
||||
observation day only when every record in that day was acknowledged before the
|
||||
cutoff. Pending or recently acknowledged records protect the entire day, including
|
||||
its rows in adjacent UTC Parquet partitions. Replay clears acknowledgement and
|
||||
therefore prevents expiry until another successful transfer. The main website
|
||||
retains its own imported history.
|
||||
|
||||
```bash
|
||||
.venv/bin/python -m solar_relay --config config.toml prune
|
||||
@@ -179,9 +187,9 @@ SOLARWEB_PASSWORD=your-account-password
|
||||
```
|
||||
|
||||
While live collection runs, the separate recovery worker checks at startup and hourly
|
||||
by default. It checks every five-minute energy interval back to
|
||||
`collector.retention_days` (30 days by default), including entirely empty days and
|
||||
dates before the first local observation. A two-hour delay allows the inverter to
|
||||
by default. It checks every five-minute energy interval from midnight at the start
|
||||
of the date `collector.retention_days` ago (30 days by default), including empty
|
||||
days and dates before the first local observation. A two-hour delay allows the inverter to
|
||||
upload its recent history. Only local dates containing gaps are fetched, and existing
|
||||
live or recovered energy is preserved. When all eligible intervals are present, no
|
||||
Solar.web requests are made. Unavailable chart days remain missing for retry and do
|
||||
@@ -189,7 +197,7 @@ not prevent other dates from being recovered. Login and network failures stop th
|
||||
attempt and retry on the next scheduled check.
|
||||
|
||||
Set `collector.timezone` to the installation's Solar.web timezone; local chart dates
|
||||
include 23/25-hour DST days. The old `history.lookback_days` setting is accepted but
|
||||
include whole 23/25-hour DST days. The old `history.lookback_days` setting is accepted but
|
||||
ignored; `collector.retention_days` controls the automatic recovery window.
|
||||
|
||||
Recovery pairs the `FromGen` Production series from the consumption view with the
|
||||
@@ -220,8 +228,8 @@ Recovery requires the account credentials even if API keys remain set.
|
||||
--from 2026-10-29 --until 2026-10-31
|
||||
```
|
||||
|
||||
Manual recovery with explicit date bounds accepts at most 30 days per invocation and
|
||||
can query dates older than the automatic retention window. Unbounded recovery checks
|
||||
Manual recovery with explicit date bounds accepts at most 30 local calendar days
|
||||
per invocation and can query dates older than the automatic retention window. Unbounded recovery checks
|
||||
the whole retention window, even when longer than 30 days. Only aligned five-minute
|
||||
chart samples in W are accepted;
|
||||
longer aggregates and unexpected units are rejected.
|
||||
@@ -327,9 +335,13 @@ extraction/last success, upload backlog, retained observations, and daily five-m
|
||||
completeness with a selectable interval grid and gap ranges. Dates use the configured
|
||||
timezone. DST days have 276 or 300 intervals; ordinary days have 288.
|
||||
|
||||
Complete means at least one fresh, successful extraction for energy, meter and boiler
|
||||
in that five-minute slot. It does not mean continuously observed power or switch state.
|
||||
The unfinished current slot is excluded from missing counts. Historical recovery
|
||||
The five-minute interval card appears above daily coverage. Status shows Energy and
|
||||
Boiler; meter counters remain archived but do not affect status, completeness, or
|
||||
interval details. Complete means valid energy and boiler data in the same five-minute
|
||||
slot. It does not mean continuously observed power or switch state.
|
||||
The current slot shows its saved extraction immediately, including partial or failed
|
||||
results. Until a record arrives, it is pending and excluded from missing counts.
|
||||
Future slots remain upcoming. Historical recovery
|
||||
contributes energy only. After retention cleanup, pruned records no longer contribute
|
||||
to this local page; the website keeps the transferred history. Refresh is every 15 seconds.
|
||||
|
||||
|
||||
+2
-1
@@ -3,7 +3,7 @@ source_id = "home-gen24"
|
||||
archive = "data/archive" # Parquet directory, relative to this configuration file.
|
||||
timezone = "Europe/Zurich" # Used for date-only export/replay arguments.
|
||||
poll_interval_seconds = 300
|
||||
retention_days = 30 # Days AFTER acknowledgement; pending records never expire.
|
||||
retention_days = 30 # Full local days after acknowledgement; pending days never expire.
|
||||
|
||||
[inverter]
|
||||
base_url = "http://fronius.local"
|
||||
@@ -32,6 +32,7 @@ enabled = false
|
||||
pv_system_id = "" # Your Solar.web PV system UUID; required when enabled.
|
||||
username = "" # Account email; enter real credentials only in private config.toml.
|
||||
password = ""
|
||||
# Recover from local midnight collector.retention_days ago, including empty days.
|
||||
lag_seconds = 7200 # Allow the inverter to upload its hourly historical data.
|
||||
check_interval_seconds = 3600
|
||||
request_timeout_seconds = 45
|
||||
@@ -1,4 +1,4 @@
|
||||
"""Local Fronius recorder and durable HTTP relay (AGPL-3.0-or-later)."""
|
||||
|
||||
__version__ = "0.2.0"
|
||||
__version__ = "0.3.0"
|
||||
|
||||
+3
-2
@@ -102,7 +102,7 @@ def _parser():
|
||||
commands.add_parser("sample", help="Record one observation and print its normalized readings")
|
||||
commands.add_parser("upload", help="Upload the pending queue, stopping on the first failure")
|
||||
commands.add_parser("status", help="Show local archive/queue status without contacting the inverter")
|
||||
commands.add_parser("prune", help="Delete records acknowledged more than retention_days ago")
|
||||
commands.add_parser("prune", help="Delete expired whole local days after acknowledgement retention")
|
||||
for name, help_text in [
|
||||
("export", "Export archived samples as JSON Lines, including already uploaded samples"),
|
||||
("replay", "Requeue archived samples for upload, preserving their IDs"),
|
||||
@@ -125,7 +125,8 @@ def main(argv=None):
|
||||
until = date_bound(args.until, config.collector.timezone)
|
||||
if start is not None and until is not None and start >= until:
|
||||
raise ValueError("--from must be earlier than --until")
|
||||
store = Store(config.collector.archive, config.collector.retention_days)
|
||||
store = Store(config.collector.archive, config.collector.retention_days,
|
||||
timezone_name=config.collector.timezone)
|
||||
if args.command == "run":
|
||||
return _run(config, store)
|
||||
if args.command == "sample":
|
||||
|
||||
+10
-1
@@ -1,7 +1,7 @@
|
||||
"""Private runtime configuration; credential fields are excluded from reprs."""
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import datetime, timezone
|
||||
from datetime import datetime, timedelta, timezone
|
||||
import math
|
||||
from pathlib import Path
|
||||
import re
|
||||
@@ -18,6 +18,15 @@ def utc_text(value: datetime) -> str:
|
||||
return value.astimezone(timezone.utc).isoformat(timespec="milliseconds").replace("+00:00", "Z")
|
||||
|
||||
|
||||
def retention_cutoff(now: datetime, days: int, timezone_name: str) -> datetime:
|
||||
"""Start of the oldest retained local calendar day, returned in UTC."""
|
||||
if now.tzinfo is None:
|
||||
raise ValueError("Retention time must include its timezone")
|
||||
zone = ZoneInfo(timezone_name)
|
||||
first = now.astimezone(zone).date() - timedelta(days=days)
|
||||
return datetime.combine(first, datetime.min.time(), zone).astimezone(timezone.utc)
|
||||
|
||||
|
||||
def date_bound(value: str | None, timezone_name: str) -> str | None:
|
||||
"""Date bounds are local midnights; timestamp bounds may specify an offset."""
|
||||
if value is None:
|
||||
|
||||
@@ -61,7 +61,8 @@ def daily_coverage(rows, timezone_name, days, now=None):
|
||||
date = datetime.fromtimestamp(observed / 1000, zone).date().isoformat()
|
||||
observations[date] += 1
|
||||
bucket = observed // BUCKET_MS * BUCKET_MS
|
||||
mask = sum(1 << index for index in range(3) if row[2 + index * 2])
|
||||
# Archive rows retain meter flags, but status coverage tracks Energy and Boiler only.
|
||||
mask = int(bool(row[2])) | (int(bool(row[6])) << 1)
|
||||
slots_by_time[bucket] = slots_by_time.get(bucket, 0) | mask
|
||||
result = []
|
||||
for offset in range(days + 1):
|
||||
@@ -71,18 +72,22 @@ def daily_coverage(rows, timezone_name, days, now=None):
|
||||
start_ms = round(start.timestamp() * 1000)
|
||||
expected = round((end.timestamp() - start.timestamp()) * 1000) // BUCKET_MS
|
||||
elapsed = expected if date < today else max(0, min(expected, (upper - start_ms) // BUCKET_MS))
|
||||
# This is extraction coverage: show the current slot as soon as its
|
||||
# record is saved, without waiting for the five-minute period to end.
|
||||
# An unrecorded current slot stays pending and is not counted missing.
|
||||
if date == today and elapsed < expected and start_ms + elapsed * BUCKET_MS in slots_by_time:
|
||||
elapsed += 1
|
||||
slots = []
|
||||
for index in range(expected):
|
||||
slots.append(slots_by_time.get(start_ms + index * BUCKET_MS, 0) if index < elapsed else -1)
|
||||
complete = sum(mask == 7 for mask in slots)
|
||||
complete = sum(mask == 3 for mask in slots)
|
||||
count = observations[date.isoformat()]
|
||||
status = "in_progress" if date == today else "complete" if complete == expected else "partial" if count else "missing"
|
||||
result.append({
|
||||
"date": date.isoformat(), "day_start": utc_text(start), "expected": expected, "elapsed": elapsed,
|
||||
"complete": complete, "missing": elapsed - complete, "samples": count, "status": status,
|
||||
"energy_complete": sum(mask >= 0 and bool(mask & 1) for mask in slots),
|
||||
"meter_complete": sum(mask >= 0 and bool(mask & 2) for mask in slots),
|
||||
"boiler_complete": sum(mask >= 0 and bool(mask & 4) for mask in slots),
|
||||
"boiler_complete": sum(mask >= 0 and bool(mask & 2) for mask in slots),
|
||||
"slots": slots,
|
||||
})
|
||||
return result
|
||||
@@ -11,7 +11,7 @@ import os
|
||||
from uuid import NAMESPACE_URL, uuid5
|
||||
from zoneinfo import ZoneInfo
|
||||
|
||||
from .config import utc_text
|
||||
from .config import retention_cutoff, utc_text
|
||||
from .coverage import milliseconds
|
||||
from .http import TransportError
|
||||
from .solarweb import ChartDataError, SolarWebClient
|
||||
@@ -92,14 +92,14 @@ class HistoryRecovery:
|
||||
def recover(self, start=None, until=None, *, now=None, stop=None):
|
||||
now = now or datetime.now(timezone.utc)
|
||||
latest = int((now - timedelta(seconds=self.config.lag_seconds)).timestamp()) // 300 * 300
|
||||
# Check the whole retained window, including days before any live observation.
|
||||
# Round up so recovery never requests an interval older than the cutoff.
|
||||
earliest = math.ceil((now - timedelta(days=self.store.retention_days)).timestamp() / 300) * 300
|
||||
# Always include the entire oldest local day, even across a DST change.
|
||||
earliest = int(retention_cutoff(now, self.store.retention_days, self.zone.key).timestamp())
|
||||
lower = milliseconds(start) // 300_000 * 300 if start else earliest
|
||||
upper = min(milliseconds(until) // 300_000 * 300, latest) if until else latest
|
||||
if lower >= upper:
|
||||
return 0
|
||||
if (start is not None or until is not None) and upper - lower > 30 * 86400:
|
||||
manual_limit = datetime.fromtimestamp(lower, self.zone) + timedelta(days=30)
|
||||
if (start is not None or until is not None) and upper > manual_limit.timestamp():
|
||||
raise ValueError("Recover at most 30 days at once")
|
||||
stamp = lambda epoch: utc_text(datetime.fromtimestamp(epoch, timezone.utc))
|
||||
present = self.store.energy_slots(self.source_id, stamp(lower), stamp(upper))
|
||||
|
||||
@@ -109,7 +109,7 @@ class StatusServer:
|
||||
now = datetime.now(timezone.utc)
|
||||
attempt = archive["last_observation"]
|
||||
age = (now.timestamp() * 1000 - milliseconds(attempt)) / 1000 if attempt else None
|
||||
errors = list(archive["last_errors"])
|
||||
errors = [source for source in archive["last_errors"] if source != "meter"]
|
||||
success = self.store.last_success(self.config.collector.source_id)
|
||||
status = "waiting" if age is None else "stale" if age > max(90, self.config.collector.poll_interval_seconds * 3) or age < -90 else "partial" if errors or success != attempt else "ok"
|
||||
body = {
|
||||
|
||||
+27
-10
@@ -13,11 +13,12 @@ from pathlib import Path
|
||||
import tempfile
|
||||
import threading
|
||||
import time
|
||||
from zoneinfo import ZoneInfo
|
||||
|
||||
import pyarrow as pa
|
||||
import pyarrow.parquet as pq
|
||||
|
||||
from .config import utc_text
|
||||
from .config import retention_cutoff, utc_text
|
||||
from .coverage import daily_coverage, milliseconds, sample_coverage
|
||||
|
||||
NUMBERS = (
|
||||
@@ -45,12 +46,13 @@ class StoreError(OSError):
|
||||
|
||||
|
||||
class Store:
|
||||
def __init__(self, path: Path, retention_days=30):
|
||||
def __init__(self, path: Path, retention_days=30, *, timezone_name="Europe/Zurich"):
|
||||
self.path = Path(path)
|
||||
if self.path.is_file():
|
||||
raise StoreError("collector.archive must be a directory")
|
||||
self.path.mkdir(parents=True, exist_ok=True, mode=0o700)
|
||||
self.retention_days = retention_days
|
||||
self.zone = ZoneInfo(timezone_name)
|
||||
self._thread_lock = threading.RLock()
|
||||
descriptor = os.open(self.path / ".lock", os.O_CREAT | os.O_WRONLY, 0o600)
|
||||
os.close(descriptor)
|
||||
@@ -190,16 +192,32 @@ class Store:
|
||||
return count
|
||||
|
||||
def prune(self, *, now=None):
|
||||
"""Retain pending records; age acknowledged records from their receipt time."""
|
||||
cutoff = utc_text((now or datetime.now(timezone.utc)) - timedelta(days=self.retention_days))
|
||||
"""Remove whole local observation days only after all their acknowledgements expire."""
|
||||
boundary = retention_cutoff(now or datetime.now(timezone.utc), self.retention_days, self.zone.key)
|
||||
cutoff, first_day = utc_text(boundary), boundary.astimezone(self.zone).date()
|
||||
deleted = 0
|
||||
local_day = lambda row: datetime.fromtimestamp(row["observed_ms"] / 1000, self.zone).date()
|
||||
with self._locked(write=True):
|
||||
partitions, expired, protected = [], set(), set()
|
||||
# A local day can span two UTC partitions. One pending or recently
|
||||
# acknowledged record protects the rest of its local day as well.
|
||||
for path in self._files():
|
||||
index = self._read(path, ["uploaded_at"])
|
||||
if not any(row["uploaded_at"] is not None and row["uploaded_at"] < cutoff for row in index):
|
||||
index = self._read(path, ["observed_ms", "uploaded_at"])
|
||||
days = set()
|
||||
for row in index:
|
||||
day = local_day(row)
|
||||
days.add(day)
|
||||
if day < first_day and row["uploaded_at"] is not None and row["uploaded_at"] < cutoff:
|
||||
expired.add(day)
|
||||
else:
|
||||
protected.add(day)
|
||||
partitions.append((path, days))
|
||||
expired -= protected
|
||||
for path, days in partitions:
|
||||
if not days & expired:
|
||||
continue
|
||||
rows = self._read(path)
|
||||
retained = [row for row in rows if row["uploaded_at"] is None or row["uploaded_at"] >= cutoff]
|
||||
retained = [row for row in rows if local_day(row) not in expired]
|
||||
self._write(path, retained)
|
||||
deleted += len(rows) - len(retained)
|
||||
return deleted
|
||||
@@ -229,7 +247,6 @@ class Store:
|
||||
}
|
||||
|
||||
def coverage(self, source_id, timezone_name, days, now=None):
|
||||
from zoneinfo import ZoneInfo
|
||||
now = now or datetime.now(timezone.utc)
|
||||
first = now.astimezone(ZoneInfo(timezone_name)).date() - timedelta(days=days)
|
||||
lower = round(datetime.combine(first, datetime.min.time(), ZoneInfo(timezone_name)).timestamp() * 1000)
|
||||
@@ -247,8 +264,8 @@ class Store:
|
||||
def last_success(self, source_id):
|
||||
with self._locked():
|
||||
for path in reversed(self._files()):
|
||||
rows = self._read(path, ["source_id", "observed_ms", "power_ok", "meter_ok", "boiler_ok"])
|
||||
stamps = [row["observed_ms"] for row in rows if row["source_id"] == source_id and row["power_ok"] and row["meter_ok"] and row["boiler_ok"]]
|
||||
rows = self._read(path, ["source_id", "observed_ms", "power_ok", "boiler_ok"])
|
||||
stamps = [row["observed_ms"] for row in rows if row["source_id"] == source_id and row["power_ok"] and row["boiler_ok"]]
|
||||
if stamps:
|
||||
return utc_text(datetime.fromtimestamp(max(stamps) / 1000, timezone.utc))
|
||||
return None
|
||||
|
||||
+11
-12
@@ -24,19 +24,9 @@
|
||||
<p id="notice" class="notice" role="alert" hidden></p>
|
||||
<section class="metrics" aria-label="Current collection status">
|
||||
<article class="metric"><p class="metric-label">Last extraction</p><strong id="last-extraction">—</strong><p id="last-extraction-date">No observation received yet</p><small id="last-success">Last successful extraction: —</small></article>
|
||||
<article class="metric"><p class="metric-label">Yesterday’s coverage</p><strong id="yesterday">— <span>/ —</span></strong><p id="yesterday-detail">Five-minute intervals</p><small>Energy, meter and boiler status</small></article>
|
||||
<article class="metric"><p class="metric-label">Yesterday’s coverage</p><strong id="yesterday">— <span>/ —</span></strong><p id="yesterday-detail">Five-minute intervals</p><small>Energy and boiler status</small></article>
|
||||
<article class="metric"><p class="metric-label">Waiting to upload</p><strong id="pending">—</strong><p id="upload-detail">Checking delivery</p><small id="last-upload">Last acknowledgement: —</small></article>
|
||||
<article class="metric"><p class="metric-label">Local archive</p><strong id="samples">—</strong><p>Observations retained in Parquet</p><small id="retention">Acknowledged readings kept for 30 days</small></article>
|
||||
</section>
|
||||
<section class="panel">
|
||||
<div class="panel-heading"><div><p class="eyebrow">DAILY COVERAGE</p><h2>Five-minute completeness</h2></div><label class="range">History <select id="history"><option value="7">7 days</option><option value="14" selected>14 days</option><option value="30">30 days</option><option value="90">90 days</option></select></label></div>
|
||||
<p class="panel-note">Each count shows five-minute intervals with valid data out of the elapsed intervals for that day. Complete intervals have all three sources. Polling does not guarantee continuous observation. Solar.web recovery fills Energy only.</p>
|
||||
<dl class="coverage-definitions">
|
||||
<div><dt>Energy</dt><dd id="energy-description">Solar production, household consumption and electricity imported from or exported to the grid. Available from live readings or recovered Solar.web history.</dd></div>
|
||||
<div><dt>Meter</dt><dd id="meter-description">Running totals of electricity bought from and sold to the grid. Requires live meter readings; Solar.web history cannot fill these counters.</dd></div>
|
||||
<div><dt>Boiler</dt><dd id="boiler-description">Boiler control output: enabled or disabled. Both states count as valid data. Heating energy is not measured, and Solar.web cannot recover past switch states.</dd></div>
|
||||
</dl>
|
||||
<div class="table-scroll"><table><thead><tr><th scope="col">Day</th><th scope="col">Complete intervals</th><th scope="col" class="source-column" aria-describedby="energy-description">Energy</th><th scope="col" class="source-column" aria-describedby="meter-description">Meter</th><th scope="col" class="source-column" aria-describedby="boiler-description">Boiler</th><th scope="col">Status</th></tr></thead><tbody id="days"><tr><td colspan="6" class="empty">Waiting for the local archive…</td></tr></tbody></table></div>
|
||||
<article class="metric"><p class="metric-label">Local archive</p><strong id="samples">—</strong><p>Observations retained in Parquet</p><small id="retention">Retention: 30 full calendar days</small></article>
|
||||
</section>
|
||||
<section class="panel detail" aria-label="Selected day interval details">
|
||||
<div class="panel-heading"><div><p class="eyebrow">FIVE-MINUTE INTERVALS</p><h2 id="selected-date">Select a day</h2></div><div class="legend"><span><i class="complete"></i>Complete</span><span><i class="partial"></i>Partial</span><span><i class="missing"></i>Missing</span><span><i class="future"></i>Upcoming</span></div></div>
|
||||
@@ -44,6 +34,15 @@
|
||||
<div id="slots" class="slots" aria-label="Five-minute coverage"></div>
|
||||
<p id="missing-ranges" class="gap-summary"></p>
|
||||
</section>
|
||||
<section class="panel">
|
||||
<div class="panel-heading"><div><p class="eyebrow">DAILY COVERAGE</p><h2>Five-minute completeness</h2></div><label class="range">History <select id="history"><option value="7">7 days</option><option value="14" selected>14 days</option><option value="30">30 days</option><option value="90">90 days</option></select></label></div>
|
||||
<p class="panel-note">Each count shows five-minute intervals with valid data out of the intervals assessed so far. Saved extractions appear immediately, including the current interval. Complete intervals have both Energy and Boiler data. Polling does not guarantee continuous observation. Solar.web recovery fills Energy only.</p>
|
||||
<dl class="coverage-definitions">
|
||||
<div><dt>Energy</dt><dd id="energy-description">Solar production, household consumption and electricity imported from or exported to the grid. Available from live readings or recovered Solar.web history.</dd></div>
|
||||
<div><dt>Boiler</dt><dd id="boiler-description">Boiler control output: enabled or disabled. Both states count as valid data. Heating energy is not measured, and Solar.web cannot recover past switch states.</dd></div>
|
||||
</dl>
|
||||
<div class="table-scroll"><table><thead><tr><th scope="col">Day</th><th scope="col">Complete intervals</th><th scope="col" class="source-column" aria-describedby="energy-description">Energy</th><th scope="col" class="source-column" aria-describedby="boiler-description">Boiler</th><th scope="col">Status</th></tr></thead><tbody id="days"><tr><td colspan="5" class="empty">Waiting for the local archive…</td></tr></tbody></table></div>
|
||||
</section>
|
||||
<footer><span>Read-only status · refreshes every 15 seconds</span><span id="version">solar-relay</span></footer>
|
||||
<noscript><p class="notice">Enable JavaScript to load extraction and coverage status.</p></noscript>
|
||||
</main>
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
:root{color-scheme:dark;--bg:#101518;--panel:#171e22;--line:#2b353a;--ink:#eef1ed;--muted:#9ba9ac;--amber:#e8ba70;--green:#7ad6ac;--red:#e68d86;--radius:12px;font-family:Inter,ui-sans-serif,system-ui,-apple-system,"Segoe UI",sans-serif;font-synthesis:none}
|
||||
*{box-sizing:border-box}body{margin:0;background:var(--bg);color:var(--ink)}button,select{font:inherit}button{cursor:pointer}button:focus-visible,select:focus-visible{outline:2px solid var(--amber);outline-offset:4px}main{max-width:1216px;margin:auto;padding:0 32px}.masthead{height:100px;display:flex;align-items:center;justify-content:space-between;border-bottom:1px solid var(--line)}.brand{display:flex;align-items:center;gap:12px;font-size:13px;letter-spacing:.14em;font-weight:750}.brand small{display:block;color:var(--muted);font-size:9px;letter-spacing:.22em;margin-top:5px}.sun{font-size:36px;color:var(--amber);line-height:1}.header-actions{display:flex;align-items:center;gap:16px}.badge{display:inline-flex;align-items:center;gap:7px;white-space:nowrap;border-radius:20px;padding:6px 11px;font-size:11px;font-weight:650;line-height:1.3;background:#252e32;color:var(--muted)}.badge::before{content:"";width:5px;height:5px;border-radius:50%;background:currentColor}.badge.ok,.badge.complete{color:var(--green);background:#193a2e}.badge.partial,.badge.in_progress{color:var(--amber);background:#3a3121}.badge.stale{color:var(--red);background:#3d2829}.badge.missing,.badge.waiting,.badge.neutral{color:var(--muted);background:#252e32}.refresh{border:1px solid var(--line);background:transparent;color:var(--ink);border-radius:7px;padding:8px 12px;font-size:13px}.refresh:hover{border-color:var(--muted)}.refresh:disabled{opacity:.45;cursor:wait}.intro{padding:42px 0 28px}.eyebrow{color:var(--amber);font-size:10px;font-weight:700;letter-spacing:.17em;margin:0 0 13px}h1{font-size:clamp(27px,3.4vw,42px);letter-spacing:-.045em;font-weight:550;line-height:1.14;margin:0 0 13px}.description{color:var(--muted);font-size:14px;line-height:1.6;margin:0}.context{display:flex;align-items:center;gap:10px;font-size:11px;color:var(--muted);margin-top:20px;flex-wrap:wrap}.separator{color:#48565c}.metrics{display:grid;grid-template-columns:repeat(4,minmax(0,1fr));gap:14px;margin:0 0 28px}.metric{border:1px solid var(--line);border-radius:var(--radius);padding:23px 20px;background:var(--panel)}.metric-label{font-size:11px;font-weight:550;color:var(--muted);margin:0 0 18px}.metric strong{display:block;font-size:30px;font-weight:550;font-variant-numeric:tabular-nums;letter-spacing:-.04em;line-height:1.1;min-height:34px}.metric strong span{color:var(--muted);font-size:17px;font-weight:400;letter-spacing:normal}.metric p:not(.metric-label){font-size:11px;margin:12px 0 7px;color:#d1d9d7}.metric small{font-size:10px;color:var(--muted);line-height:1.6;display:block}.panel{background:var(--panel);border:1px solid var(--line);border-radius:var(--radius);margin-bottom:22px;overflow:hidden}.panel-heading{display:flex;align-items:center;justify-content:space-between;gap:16px;padding:24px 24px 0}.panel-heading .eyebrow{margin-bottom:7px}h2{font-size:19px;font-weight:550;letter-spacing:-.025em;margin:0}.panel-note{font-size:11px;line-height:1.7;color:var(--muted);padding:0 24px;margin:12px 0 22px}.range{display:flex;align-items:center;gap:9px;color:var(--muted);font-size:11px;white-space:nowrap}select{background:#20292e;border:1px solid #354149;border-radius:6px;color:var(--ink);padding:7px 27px 7px 9px;font-size:11px}.table-scroll{overflow:auto}table{width:100%;border-collapse:collapse;text-align:left;font-size:12px;font-variant-numeric:tabular-nums}th{background:#1c252a;border-top:1px solid var(--line);border-bottom:1px solid var(--line);padding:11px 18px;font-size:9px;font-weight:650;letter-spacing:.07em;text-transform:uppercase;color:var(--muted)}th:first-child,td:first-child{padding-left:24px}td{padding:13px 18px;border-bottom:1px solid #283137;white-space:nowrap}tbody tr:last-child td{border-bottom:0}tr.selected{background:#222d32}tr:hover{background:#202a2f}.day-select{display:block;text-align:left;color:var(--ink);background:transparent;border:0;padding:0;line-height:1.55;font-size:12px}.day-select small{display:block;color:var(--muted);font-size:10px}.coverage-number{display:flex;align-items:baseline;gap:5px;font-size:12px}.coverage-number small{font-size:10px;color:var(--muted)}.progress{height:3px;width:126px;border-radius:2px;background:#303c42;margin-top:7px;overflow:hidden}.progress span{display:block;height:100%;background:var(--green)}.source-count{color:#c1ceca;font-size:11px}.source-count.incomplete{color:var(--amber)}.empty{padding:40px!important;text-align:center;color:var(--muted)}.legend{display:flex;gap:14px;flex-wrap:wrap;font-size:10px;color:var(--muted)}.legend span{display:flex;align-items:center;gap:6px}.legend i{display:block;width:7px;height:7px;border-radius:2px}.slots{display:flex;gap:5px;padding:0 24px;overflow-x:auto}.hour{display:grid;grid-template-rows:16px repeat(12,5px);gap:3px;flex:1;min-width:14px}.hour-label{font-size:9px;color:var(--muted);font-variant-numeric:tabular-nums;text-align:center}.slot{display:block;border-radius:1px;min-height:5px}.complete{background:#76c9a4}.partial{background:#bd9558}.missing{bacLine truncated
|
||||
.coverage-definitions{display:grid;grid-template-columns:repeat(3,minmax(0,1fr));gap:20px;margin:0;padding:0 24px 24px}.coverage-definitions dt{font-size:11px;font-weight:650;margin-bottom:6px}.coverage-definitions dd{margin:0;color:var(--muted);font-size:11px;line-height:1.7}
|
||||
.coverage-definitions{display:grid;grid-template-columns:repeat(2,minmax(0,1fr));gap:20px;margin:0;padding:0 24px 24px}.coverage-definitions dt{font-size:11px;font-weight:650;margin-bottom:6px}.coverage-definitions dd{margin:0;color:var(--muted);font-size:11px;line-height:1.7}
|
||||
@media(max-width:900px){.metrics{grid-template-columns:repeat(2,minmax(0,1fr))}.metric{padding:20px}.source-column{display:none}.legend{max-width:215px;gap:10px}.slots{gap:4px}.progress{width:115px}}
|
||||
@media(max-width:560px){.coverage-definitions{grid-template-columns:1fr;gap:14px;padding:0 16px 20px}.coverage-definitions dt,.coverage-definitions dd{font-size:10px}}
|
||||
@media(max-width:560px){main{padding:0 16px}.masthead{height:82px}.brand{font-size:11px;gap:8px}.sun{font-size:30px}.header-actions{gap:8px}.refresh span{display:none}.refresh{font-size:17px;padding:4px 9px}.badge{font-size:10px;padding:6px 8px}.intro{padding-top:30px}.metrics{gap:10px;margin-bottom:20px}.metric{padding:17px 14px}.metric strong{font-size:26px}.metric strong span{font-size:14px}.metric-label{margin-bottom:14px}.metric small{font-size:9px}.panel-heading{padding:20px 16px 0;align-items:flex-start}h2{font-size:16px}.panel-note{padding:0 16px;font-size:10px}th,td{padding-left:11px;padding-right:11px}th:first-child,td:first-child{padding-left:16px}.progress{width:88px}.range{flex-direction:column;align-items:flex-end;gap:5px}.legend{font-size:9px;gap:7px;max-width:150px}.slots{padding:0 16px;gap:3px}.hour{min-width:10px;grid-template-rows:16px repeat(12,5px);gap:3px}.hour-label{font-size:8px}.gap-summary{padding:16px;font-size:10px}.day-select{font-size:11px}footer{font-size:9px;flex-wrap:wrap}.detail .panel-heading{gap:8px}}
|
||||
@@ -48,7 +48,7 @@
|
||||
const fill = addText(progress,'span','');
|
||||
fill.style.width = (day.elapsed ? day.complete/day.elapsed*100:0)+'%';
|
||||
row.append(amount);
|
||||
for(const field of ['energy_complete','meter_complete','boiler_complete']) {
|
||||
for(const field of ['energy_complete','boiler_complete']) {
|
||||
const cell = td('source-column');
|
||||
addText(cell,'span',number(day[field])+'/'+number(day.elapsed),'source-count'+(day[field]<day.elapsed?' incomplete':''));
|
||||
row.append(cell);
|
||||
@@ -63,8 +63,9 @@
|
||||
const day = current.days.find(item => item.date===selected);
|
||||
if(!day) return;
|
||||
$('selected-date').textContent = format(day.date+'T12:00:00Z',{weekday:'long',day:'numeric',month:'long',year:'numeric'});
|
||||
$('selected-summary').textContent = `${number(day.complete)} complete of ${number(day.elapsed)} elapsed intervals · ${number(day.expected)} intervals in this calendar day. Each cell is five minutes.`;
|
||||
$('selected-summary').textContent = `${number(day.complete)} complete of ${number(day.elapsed)} assessed intervals · ${number(day.expected)} intervals in this calendar day. Each cell is five minutes.`;
|
||||
const start = Date.parse(day.day_start);
|
||||
const currentIndex = day.status==='in_progress' ? Math.floor((Date.parse(current.generated_at)-start)/300000) : -1;
|
||||
const stamp = index => format(new Date(start+index*300000),{hour:'2-digit',minute:'2-digit',timeZoneName:'shortOffset',hour12:false});
|
||||
$('slots').replaceChildren();
|
||||
let hour;
|
||||
@@ -74,9 +75,9 @@
|
||||
addText(hour,'span',format(new Date(start+index*300000),{hour:'2-digit',hour12:false}),'hour-label');
|
||||
}
|
||||
const mask = day.slots[index];
|
||||
const state = mask===7?'complete':mask===-1?'future':mask>0?'partial':'missing';
|
||||
const missing = ['Energy','Meter','Boiler'].filter((_,bit) => !(mask & (1<<bit)));
|
||||
const text = `${stamp(index)}–${stamp(index+1)} · ${mask===7?'Complete':mask===-1?'Upcoming':'Incomplete: '+missing.join(', ')}`;
|
||||
const state = mask===3?'complete':mask===-1?'future':mask>0?'partial':'missing';
|
||||
const missing = ['Energy','Boiler'].filter((_,bit) => !(mask & (1<<bit)));
|
||||
const text = `${stamp(index)}–${stamp(index+1)} · ${mask===3?'Complete':mask===-1?(index===currentIndex?'Awaiting extraction':'Upcoming'):'Incomplete: '+missing.join(', ')}`;
|
||||
const cell = addText(hour,'span','','slot '+state);
|
||||
cell.title = text;
|
||||
cell.setAttribute('role','img');
|
||||
@@ -84,12 +85,12 @@
|
||||
}
|
||||
const ranges = [];
|
||||
for(let index=0;index<day.elapsed;) {
|
||||
if(day.slots[index]===7) {index++;continue;}
|
||||
if(day.slots[index]===3) {index++;continue;}
|
||||
const from = index;
|
||||
while(index<day.elapsed && day.slots[index]!==7) index++;
|
||||
while(index<day.elapsed && day.slots[index]!==3) index++;
|
||||
ranges.push(`${stamp(from)}–${stamp(index)}`);
|
||||
}
|
||||
$('missing-ranges').textContent = ranges.length ? `${number(day.missing)} incomplete intervals. Gaps: ${ranges.slice(0,6).join('; ')}${ranges.length>6?`; plus ${number(ranges.length-6)} more ranges.`:'.'}` : day.status==='in_progress'?'All elapsed intervals are complete. The current interval is still being recorded.':'All five-minute intervals have complete data.';
|
||||
$('missing-ranges').textContent = ranges.length ? `${number(day.missing)} incomplete intervals. Gaps: ${ranges.slice(0,6).join('; ')}${ranges.length>6?`; plus ${number(ranges.length-6)} more ranges.`:'.'}` : day.status==='in_progress'?(day.slots[currentIndex]===-1?'All previous intervals are complete. Waiting for the current extraction.':'All intervals checked so far have complete data.'):'All five-minute intervals have complete data.';
|
||||
}
|
||||
function render(data) {
|
||||
current = data;
|
||||
@@ -112,7 +113,7 @@
|
||||
$('upload-detail').textContent = !data.archive.upload_enabled?'Uploads not configured':data.archive.pending?'Saved locally, awaiting delivery':'All observations acknowledged';
|
||||
$('last-upload').textContent = 'Last acknowledgement: '+(data.archive.last_upload?date(data.archive.last_upload)+' '+time(data.archive.last_upload):'none yet');
|
||||
$('samples').textContent = number(data.archive.samples);
|
||||
$('retention').textContent = `Acknowledged readings kept for ${number(data.archive.retention_days)} days · pending readings retained`;
|
||||
$('retention').textContent = `Retention: ${number(data.archive.retention_days)} full calendar days · pending days retained`;
|
||||
$('version').textContent = 'solar-relay '+data.version;
|
||||
if(!selected || !data.days.some(day=>day.date===selected)) selected=(data.days.find(day=>day.samples>0)||yesterday||data.days[0]).date;
|
||||
renderDays();
|
||||
|
||||
+29
-5
@@ -68,13 +68,13 @@ class HistoryTests(unittest.TestCase):
|
||||
self.assertEqual(self.recover(), 0)
|
||||
self.assertEqual(len(self.client.requests), 2)
|
||||
|
||||
def test_cloud_energy_does_not_mark_boiler_or_meter_history_complete(self):
|
||||
def test_cloud_energy_does_not_mark_boiler_history_complete(self):
|
||||
self.assertEqual(self.recover(), 3)
|
||||
rows = self.store.coverage(self.source, "UTC", 1, self.when + timedelta(days=1))
|
||||
day = rows[1]
|
||||
self.assertEqual(day["energy_complete"], 3)
|
||||
self.assertEqual(day["boiler_complete"], 0)
|
||||
self.assertEqual(day["meter_complete"], 0)
|
||||
self.assertNotIn("meter_complete", day)
|
||||
self.assertEqual(day["complete"], 0)
|
||||
row = pq.ParquetFile(next(self.store.path.glob("date=*/samples.parquet"))).read().to_pylist()[0]
|
||||
self.assertEqual(row["record_kind"], "history")
|
||||
@@ -148,7 +148,7 @@ class HistoryTests(unittest.TestCase):
|
||||
|
||||
def test_automatic_recovery_checks_retention_before_first_observation_and_respects_cutoff(self):
|
||||
now = datetime(2026, 11, 2, 12, 2, 17, tzinfo=timezone.utc)
|
||||
cutoff = datetime(2026, 10, 3, 12, 5, tzinfo=timezone.utc)
|
||||
cutoff = datetime(2026, 10, 3, tzinfo=ZoneInfo("Europe/Zurich")).astimezone(timezone.utc)
|
||||
latest = datetime(2026, 11, 2, 10, tzinfo=timezone.utc)
|
||||
recoverable = [cutoff, now.replace(minute=0, second=0) - timedelta(days=25), latest - timedelta(minutes=5)]
|
||||
points = {int(when.timestamp() * 1000): 1200 for when in
|
||||
@@ -158,7 +158,31 @@ class HistoryTests(unittest.TestCase):
|
||||
self.assertEqual(self.recovery.recover(now=now), 3)
|
||||
recovered = [record for record in self.store.records() if "history_interval" in record]
|
||||
self.assertEqual([record["observed_at"] for record in recovered], [utc_text(when) for when in recoverable])
|
||||
self.assertEqual(min(day for _, _, day in self.client.requests), cutoff.date())
|
||||
self.assertEqual(min(day for _, _, day in self.client.requests), cutoff.astimezone(self.recovery.zone).date())
|
||||
|
||||
def test_oldest_retained_day_recovers_all_288_intervals_including_midnight(self):
|
||||
zone = ZoneInfo("Europe/Zurich")
|
||||
start = datetime(2026, 10, 3, tzinfo=zone)
|
||||
end = start + timedelta(days=1)
|
||||
points = {stamp: 1200 for stamp in range(int(start.timestamp() * 1000), int(end.timestamp() * 1000), 300_000)}
|
||||
self.client.points = {"production": points, "consumption": points}
|
||||
self.assertEqual(self.recovery.recover(now=datetime(2026, 11, 2, 22, 30, tzinfo=zone)), 288)
|
||||
records = list(self.store.records())
|
||||
self.assertEqual(records[0]["observed_at"], utc_text(start))
|
||||
self.assertEqual(records[-1]["observed_at"], utc_text(end - timedelta(minutes=5)))
|
||||
|
||||
def test_retention_uses_the_configured_timezone_instead_of_utc(self):
|
||||
now = datetime(2026, 11, 2, 1, 30, tzinfo=timezone.utc)
|
||||
recovery = HistoryRecovery(self.config, self.source, self.store, timezone_name="America/New_York", client=self.client)
|
||||
recovery.recover(now=now)
|
||||
self.assertEqual(min(day for _, _, day in self.client.requests).isoformat(), "2026-10-02")
|
||||
|
||||
def test_thirty_calendar_day_manual_range_can_cross_a_25_hour_dst_day(self):
|
||||
zone = ZoneInfo("Europe/Zurich")
|
||||
self.client.points = {"production": {}, "consumption": {}}
|
||||
self.assertEqual(self.recovery.recover(utc_text(datetime(2026, 10, 3, tzinfo=zone)),
|
||||
utc_text(datetime(2026, 11, 2, tzinfo=zone)),
|
||||
now=datetime(2026, 11, 3, tzinfo=zone)), 0)
|
||||
|
||||
def test_automatic_recovery_checks_entirely_empty_days_and_archives(self):
|
||||
self.store.retention_days = 2
|
||||
@@ -176,7 +200,7 @@ class HistoryTests(unittest.TestCase):
|
||||
|
||||
def test_complete_retention_window_does_not_request_solarweb(self):
|
||||
now = self.when + timedelta(days=1)
|
||||
lower = int(self.when.timestamp())
|
||||
lower = int(datetime(2026, 10, 30, tzinfo=ZoneInfo("Europe/Zurich")).timestamp())
|
||||
upper = int((now - timedelta(seconds=self.config.lag_seconds)).timestamp())
|
||||
store = Mock(retention_days=1)
|
||||
store.energy_slots.return_value = set(range(lower, upper, 300))
|
||||
|
||||
+65
-2
@@ -5,10 +5,12 @@ from pathlib import Path
|
||||
import tempfile
|
||||
import unittest
|
||||
from unittest.mock import patch
|
||||
from zoneinfo import ZoneInfo
|
||||
|
||||
import pyarrow.parquet as pq
|
||||
|
||||
from solar_relay.cli import next_poll_at
|
||||
from solar_relay.config import retention_cutoff
|
||||
from solar_relay.storage import Store, StoreError
|
||||
from tests.test_status import observed
|
||||
|
||||
@@ -34,8 +36,8 @@ class ParquetTests(unittest.TestCase):
|
||||
self.assertEqual(json.loads(row["payload"]), value)
|
||||
self.assertEqual(self.store.status()["format"], "parquet")
|
||||
|
||||
def test_retention_uses_ack_time_and_never_deletes_pending(self):
|
||||
items = [observed(self.now - timedelta(days=90, minutes=index)) for index in range(4)]
|
||||
def test_retention_uses_ack_calendar_days_and_never_deletes_pending(self):
|
||||
items = [observed(self.now - timedelta(days=90 + index)) for index in range(4)]
|
||||
for item in items:
|
||||
self.store.add(item)
|
||||
self.store.acknowledge([items[0]["id"]], now=self.now - timedelta(days=31))
|
||||
@@ -47,6 +49,67 @@ class ParquetTests(unittest.TestCase):
|
||||
self.assertEqual(self.store.prune(now=self.now + timedelta(days=31)), 2)
|
||||
self.assertEqual(list(self.store.records()), [items[3]])
|
||||
|
||||
def test_retention_boundary_is_local_midnight_regardless_of_execution_time(self):
|
||||
zone = ZoneInfo("Europe/Zurich")
|
||||
for hour in (0, 12, 23):
|
||||
now = datetime(2026, 11, 2, hour, 30, tzinfo=zone)
|
||||
self.assertEqual(retention_cutoff(now, 30, zone.key), datetime(2026, 10, 2, 22, tzinfo=timezone.utc))
|
||||
with self.assertRaisesRegex(ValueError, "timezone"):
|
||||
retention_cutoff(datetime(2026, 11, 2), 30, zone.key)
|
||||
|
||||
def test_prune_uses_the_configured_timezone_at_the_date_boundary(self):
|
||||
store = Store(self.path, timezone_name="America/New_York")
|
||||
values = [observed(self.now - timedelta(days=60 + offset)) for offset in range(2)]
|
||||
for value in values:
|
||||
store.add(value)
|
||||
store.acknowledge([values[0]["id"]], now=datetime(2026, 10, 2, 3, 59, tzinfo=timezone.utc))
|
||||
store.acknowledge([values[1]["id"]], now=datetime(2026, 10, 2, 4, tzinfo=timezone.utc))
|
||||
self.assertEqual(store.prune(now=datetime(2026, 11, 2, 1, 30, tzinfo=timezone.utc)), 1)
|
||||
self.assertEqual(list(store.records()), [values[1]])
|
||||
|
||||
def test_prune_keeps_the_entire_cutoff_acknowledgement_day(self):
|
||||
zone = ZoneInfo("Europe/Zurich")
|
||||
start = datetime(2026, 10, 3, tzinfo=zone)
|
||||
values = [observed(start - timedelta(days=10) + timedelta(minutes=offset)) for offset in (0, 5)]
|
||||
for value, ack in zip(values, (start, start + timedelta(hours=23, minutes=55))):
|
||||
self.store.add(value)
|
||||
self.store.acknowledge([value["id"]], now=ack)
|
||||
self.assertEqual(self.store.prune(now=datetime(2026, 11, 2, 23, 59, tzinfo=zone)), 0)
|
||||
self.assertEqual(self.store.prune(now=datetime(2026, 11, 3, tzinfo=zone)), 2)
|
||||
|
||||
def test_prune_removes_whole_local_days_across_utc_partitions(self):
|
||||
zone = ZoneInfo("Europe/Zurich")
|
||||
first = datetime(2026, 10, 3, tzinfo=zone)
|
||||
expired = [observed(first - timedelta(hours=23, minutes=55)), observed(first - timedelta(minutes=5))]
|
||||
retained = [observed(first), observed(first + timedelta(hours=23, minutes=55))]
|
||||
for value in expired + retained:
|
||||
self.store.add(value)
|
||||
self.store.acknowledge([value["id"] for value in expired], now=first - timedelta(minutes=1))
|
||||
self.store.acknowledge([value["id"] for value in retained], now=first + timedelta(days=1))
|
||||
self.assertEqual(self.store.prune(now=datetime(2026, 11, 2, 22, tzinfo=zone)), 2)
|
||||
self.assertEqual(list(self.store.records()), retained)
|
||||
|
||||
def test_one_pending_record_protects_its_entire_local_day_across_partitions(self):
|
||||
zone = ZoneInfo("Europe/Zurich")
|
||||
day = datetime(2026, 10, 2, tzinfo=zone)
|
||||
values = [observed(day + timedelta(minutes=5)), observed(day + timedelta(hours=23, minutes=55))]
|
||||
for value in values:
|
||||
self.store.add(value)
|
||||
self.store.acknowledge([values[0]["id"]], now=day + timedelta(hours=23, minutes=59))
|
||||
self.assertEqual(self.store.prune(now=datetime(2026, 11, 2, 22, tzinfo=zone)), 0)
|
||||
self.assertEqual(list(self.store.records()), values)
|
||||
|
||||
def test_daylight_saving_retention_boundaries_preserve_complete_local_days(self):
|
||||
zone = ZoneInfo("Europe/Zurich")
|
||||
for day, expected_hours in [(datetime(2026, 3, 29, tzinfo=zone), 23),
|
||||
(datetime(2026, 10, 25, tzinfo=zone), 25)]:
|
||||
with self.subTest(day=day):
|
||||
now = day + timedelta(days=30, hours=12)
|
||||
cutoff = retention_cutoff(now, 30, zone.key)
|
||||
self.assertEqual(cutoff, day.astimezone(timezone.utc))
|
||||
next_day = (day + timedelta(days=1)).astimezone(timezone.utc)
|
||||
self.assertEqual((next_day - cutoff).total_seconds(), expected_hours * 3600)
|
||||
|
||||
def test_replay_cancels_expiry_until_the_next_acknowledgement(self):
|
||||
value = observed(self.now)
|
||||
self.store.add(value)
|
||||
|
||||
+88
-5
@@ -42,8 +42,56 @@ class CoverageTests(unittest.TestCase):
|
||||
self.assertEqual(yesterday["expected"], 288)
|
||||
self.assertEqual(yesterday["complete"], 288)
|
||||
self.assertEqual(yesterday["status"], "complete")
|
||||
self.assertEqual(result[0]["elapsed"], 0)
|
||||
self.assertTrue(all(mask == -1 for mask in result[0]["slots"]))
|
||||
self.assertEqual(result[0]["elapsed"], 1)
|
||||
self.assertEqual(result[0]["slots"][0], 3)
|
||||
self.assertTrue(all(mask == -1 for mask in result[0]["slots"][1:]))
|
||||
|
||||
def test_successful_2305_extraction_is_green_at_2308(self):
|
||||
zone = ZoneInfo("Europe/Zurich")
|
||||
now = datetime(2026, 11, 2, 23, 8, tzinfo=zone)
|
||||
stamp = round(now.replace(minute=5).timestamp() * 1000)
|
||||
day = daily_coverage([(stamp, stamp, 1, stamp, 1, stamp, 1)], zone.key, 1, now)[0]
|
||||
self.assertEqual(day["slots"][277], 3)
|
||||
self.assertEqual(day["slots"][278], -1)
|
||||
self.assertEqual(day["complete"], 1)
|
||||
self.assertEqual(day["energy_complete"], 1)
|
||||
self.assertEqual(day["boiler_complete"], 1)
|
||||
self.assertEqual(day["elapsed"], 278)
|
||||
self.assertEqual(day["missing"], 277)
|
||||
|
||||
def test_current_slot_without_extraction_stays_pending_until_it_ends(self):
|
||||
zone = ZoneInfo("Europe/Zurich")
|
||||
now = datetime(2026, 11, 2, 23, 8, tzinfo=zone)
|
||||
day = daily_coverage([], zone.key, 1, now)[0]
|
||||
self.assertEqual(day["slots"][277], -1)
|
||||
self.assertEqual(day["elapsed"], 277)
|
||||
self.assertEqual(day["missing"], 277)
|
||||
after = daily_coverage([], zone.key, 1, now.replace(minute=10))[0]
|
||||
self.assertEqual(after["slots"][277], 0)
|
||||
self.assertEqual(after["slots"][278], -1)
|
||||
self.assertEqual(after["missing"], 278)
|
||||
|
||||
def test_partial_and_failed_current_extractions_are_shown_immediately(self):
|
||||
zone = ZoneInfo("Europe/Zurich")
|
||||
now = datetime(2026, 11, 2, 23, 8, tzinfo=zone)
|
||||
stamp = round(now.replace(minute=5).timestamp() * 1000)
|
||||
for energy, boiler, expected_mask in [(1, 0, 1), (0, 1, 2), (0, 0, 0)]:
|
||||
with self.subTest(energy=energy, boiler=boiler):
|
||||
day = daily_coverage([(stamp, stamp, energy, stamp, 0, stamp, boiler)], zone.key, 1, now)[0]
|
||||
self.assertEqual(day["slots"][277], expected_mask)
|
||||
self.assertEqual(day["elapsed"], 278)
|
||||
self.assertEqual(day["complete"], 0)
|
||||
|
||||
def test_current_extraction_at_midnight_is_counted_without_waiting(self):
|
||||
zone = ZoneInfo("Europe/Zurich")
|
||||
now = datetime(2026, 11, 2, 0, 0, 10, tzinfo=zone)
|
||||
stamp = round(now.replace(second=0).timestamp() * 1000)
|
||||
day = daily_coverage([(stamp, stamp, 1, stamp, 0, stamp, 1)], zone.key, 1, now)[0]
|
||||
self.assertEqual(day["slots"][0], 3)
|
||||
self.assertEqual(day["elapsed"], 1)
|
||||
self.assertEqual(day["complete"], 1)
|
||||
self.assertEqual(day["missing"], 0)
|
||||
self.assertTrue(all(mask == -1 for mask in day["slots"][1:]))
|
||||
|
||||
def test_dst_days_expect_276_and_300_intervals(self):
|
||||
for date, expected in [("2026-03-29", 276), ("2026-10-25", 300)]:
|
||||
@@ -61,17 +109,27 @@ class CoverageTests(unittest.TestCase):
|
||||
self.assertEqual(yesterday["missing"], 3)
|
||||
self.assertEqual(yesterday["status"], "partial")
|
||||
|
||||
def test_energy_failure_does_not_erase_meter_or_boiler_coverage(self):
|
||||
def test_energy_failure_does_not_erase_boiler_coverage(self):
|
||||
rows, end = rows_for_day("2026-10-30")
|
||||
stamp = rows[1][0]
|
||||
rows[1] = (stamp, stamp, 0, stamp, 1, stamp, 1)
|
||||
yesterday = daily_coverage(rows, "Europe/Zurich", 1, end + timedelta(minutes=3))[1]
|
||||
self.assertEqual(yesterday["slots"][1], 6)
|
||||
self.assertEqual(yesterday["slots"][1], 2)
|
||||
self.assertEqual(yesterday["energy_complete"], 287)
|
||||
self.assertEqual(yesterday["meter_complete"], 288)
|
||||
self.assertNotIn("meter_complete", yesterday)
|
||||
self.assertEqual(yesterday["boiler_complete"], 288)
|
||||
self.assertEqual(yesterday["missing"], 1)
|
||||
|
||||
def test_missing_meter_readings_do_not_affect_visible_completeness(self):
|
||||
rows, end = rows_for_day("2026-10-30")
|
||||
rows = [(observed, power_ms, power_ok, meter_ms, 0, boiler_ms, boiler_ok)
|
||||
for observed, power_ms, power_ok, meter_ms, meter_ok, boiler_ms, boiler_ok in rows]
|
||||
yesterday = daily_coverage(rows, "Europe/Zurich", 1, end + timedelta(minutes=3))[1]
|
||||
self.assertEqual(yesterday["complete"], 288)
|
||||
self.assertEqual(yesterday["status"], "complete")
|
||||
self.assertEqual(set(yesterday["slots"]), {3})
|
||||
self.assertNotIn("meter_complete", yesterday)
|
||||
|
||||
def test_source_errors_stale_readings_and_false_boiler_state(self):
|
||||
when = datetime(2026, 10, 30, tzinfo=timezone.utc)
|
||||
value = observed(when)
|
||||
@@ -150,6 +208,31 @@ class StatusHttpTests(unittest.TestCase):
|
||||
self.assertEqual(result["collector"]["source_errors"], ["boiler"])
|
||||
self.assertNotIn("secret canary", json.dumps(result))
|
||||
|
||||
def test_meter_failure_does_not_change_current_status_or_last_success(self):
|
||||
value = observed(datetime.now(timezone.utc))
|
||||
value["errors"] = {"meter": "meter is unavailable"}
|
||||
value["readings"]["grid_import_total_wh"] = None
|
||||
value["readings"]["grid_export_total_wh"] = None
|
||||
self.store.add(value)
|
||||
result = json.loads(self.get("/api/status")[2])
|
||||
self.assertEqual(result["status"], "ok")
|
||||
self.assertEqual(result["collector"]["source_errors"], [])
|
||||
self.assertEqual(result["collector"]["last_success"], value["observed_at"])
|
||||
self.assertNotIn("meter_complete", result["days"][0])
|
||||
|
||||
def test_latest_success_and_current_grid_cell_agree(self):
|
||||
zone = ZoneInfo("Europe/Zurich")
|
||||
now = datetime(2026, 11, 2, 23, 8, tzinfo=zone)
|
||||
value = observed(now.replace(minute=5))
|
||||
self.store.add(value)
|
||||
with unittest.mock.patch("solar_relay.status.datetime", wraps=datetime) as dates:
|
||||
dates.now.return_value = now.astimezone(timezone.utc)
|
||||
result = json.loads(self.get("/api/status?days=1")[2])
|
||||
self.assertEqual(result["status"], "ok")
|
||||
self.assertEqual(result["collector"]["last_success"], value["observed_at"])
|
||||
self.assertEqual(result["days"][0]["slots"][277], 3)
|
||||
self.assertEqual(result["days"][0]["slots"][278], -1)
|
||||
|
||||
def test_only_explicit_read_only_routes_are_available(self):
|
||||
for path in ["/config.toml", "/.env", "/data/archive/date=2026-11-01/samples.parquet", "/../config.toml", "/solar-relay/.git/config"]:
|
||||
self.assertEqual(self.get(path)[0], 404, path)
|
||||
|
||||
Reference in new issue
Block a user