This commit is contained in:
1 parent
cd7ddd3cf6
commit
89621bc7c3
15 files changed
+289
-74
No files matched your search
@@ -20,8 +20,9 @@ Scheduled collection remains anchored to `:00`, `:05`, `:10`, … boundaries.
|
|||||||
|
|
||||||
Observations are saved durably to **Parquet** before upload. Separate workers handle
|
Observations are saved durably to **Parquet** before upload. Separate workers handle
|
||||||
delivery and optional history recovery; an internet outage does not block recording.
|
delivery and optional history recovery; an internet outage does not block recording.
|
||||||
Acknowledged observations are deleted **30 days after acknowledgement**. Pending
|
Acknowledged observations are retained for **30 full local calendar days after
|
||||||
observations never expire, even if their measurement dates are much older.
|
acknowledgement**. Cleanup removes whole observation days. A pending or recently
|
||||||
|
acknowledged record keeps its whole local day, even when its measurements are much older.
|
||||||
|
|
||||||
## Installation
|
## Installation
|
||||||
|
|
||||||
@@ -110,10 +111,17 @@ acknowledgements mark only the accepted records; remaining records retry with
|
|||||||
exponential backoff. Delivery is **at least once**: the website deduplicates stable
|
exponential backoff. Delivery is **at least once**: the website deduplicates stable
|
||||||
UUIDs when a response is lost, so retries and replays do not double-count energy.
|
UUIDs when a response is lost, so retries and replays do not double-count energy.
|
||||||
|
|
||||||
`collector.retention_days` defaults to 30. Automatic cleanup runs at service startup
|
`collector.retention_days` defaults to 30. All retention cutoffs are midnight in
|
||||||
and hourly, deleting records acknowledged more than that many days ago. Old pending
|
`collector.timezone`. For example, on November 2 with 30 days retained, the cutoff
|
||||||
records remain. Replay clears acknowledgement and therefore prevents expiry until
|
is October 3 at 00:00 local time, regardless of the execution time. The entire
|
||||||
another successful transfer. The main website retains its own imported history.
|
October 3 day is kept and eligible for history recovery.
|
||||||
|
|
||||||
|
Automatic cleanup runs at service startup and hourly. It removes an older local
|
||||||
|
observation day only when every record in that day was acknowledged before the
|
||||||
|
cutoff. Pending or recently acknowledged records protect the entire day, including
|
||||||
|
its rows in adjacent UTC Parquet partitions. Replay clears acknowledgement and
|
||||||
|
therefore prevents expiry until another successful transfer. The main website
|
||||||
|
retains its own imported history.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
.venv/bin/python -m solar_relay --config config.toml prune
|
.venv/bin/python -m solar_relay --config config.toml prune
|
||||||
@@ -179,9 +187,9 @@ SOLARWEB_PASSWORD=your-account-password
|
|||||||
```
|
```
|
||||||
|
|
||||||
While live collection runs, the separate recovery worker checks at startup and hourly
|
While live collection runs, the separate recovery worker checks at startup and hourly
|
||||||
by default. It checks every five-minute energy interval back to
|
by default. It checks every five-minute energy interval from midnight at the start
|
||||||
`collector.retention_days` (30 days by default), including entirely empty days and
|
of the date `collector.retention_days` ago (30 days by default), including empty
|
||||||
dates before the first local observation. A two-hour delay allows the inverter to
|
days and dates before the first local observation. A two-hour delay allows the inverter to
|
||||||
upload its recent history. Only local dates containing gaps are fetched, and existing
|
upload its recent history. Only local dates containing gaps are fetched, and existing
|
||||||
live or recovered energy is preserved. When all eligible intervals are present, no
|
live or recovered energy is preserved. When all eligible intervals are present, no
|
||||||
Solar.web requests are made. Unavailable chart days remain missing for retry and do
|
Solar.web requests are made. Unavailable chart days remain missing for retry and do
|
||||||
@@ -189,7 +197,7 @@ not prevent other dates from being recovered. Login and network failures stop th
|
|||||||
attempt and retry on the next scheduled check.
|
attempt and retry on the next scheduled check.
|
||||||
|
|
||||||
Set `collector.timezone` to the installation's Solar.web timezone; local chart dates
|
Set `collector.timezone` to the installation's Solar.web timezone; local chart dates
|
||||||
include 23/25-hour DST days. The old `history.lookback_days` setting is accepted but
|
include whole 23/25-hour DST days. The old `history.lookback_days` setting is accepted but
|
||||||
ignored; `collector.retention_days` controls the automatic recovery window.
|
ignored; `collector.retention_days` controls the automatic recovery window.
|
||||||
|
|
||||||
Recovery pairs the `FromGen` Production series from the consumption view with the
|
Recovery pairs the `FromGen` Production series from the consumption view with the
|
||||||
@@ -220,8 +228,8 @@ Recovery requires the account credentials even if API keys remain set.
|
|||||||
--from 2026-10-29 --until 2026-10-31
|
--from 2026-10-29 --until 2026-10-31
|
||||||
```
|
```
|
||||||
|
|
||||||
Manual recovery with explicit date bounds accepts at most 30 days per invocation and
|
Manual recovery with explicit date bounds accepts at most 30 local calendar days
|
||||||
can query dates older than the automatic retention window. Unbounded recovery checks
|
per invocation and can query dates older than the automatic retention window. Unbounded recovery checks
|
||||||
the whole retention window, even when longer than 30 days. Only aligned five-minute
|
the whole retention window, even when longer than 30 days. Only aligned five-minute
|
||||||
chart samples in W are accepted;
|
chart samples in W are accepted;
|
||||||
longer aggregates and unexpected units are rejected.
|
longer aggregates and unexpected units are rejected.
|
||||||
@@ -327,9 +335,13 @@ extraction/last success, upload backlog, retained observations, and daily five-m
|
|||||||
completeness with a selectable interval grid and gap ranges. Dates use the configured
|
completeness with a selectable interval grid and gap ranges. Dates use the configured
|
||||||
timezone. DST days have 276 or 300 intervals; ordinary days have 288.
|
timezone. DST days have 276 or 300 intervals; ordinary days have 288.
|
||||||
|
|
||||||
Complete means at least one fresh, successful extraction for energy, meter and boiler
|
The five-minute interval card appears above daily coverage. Status shows Energy and
|
||||||
in that five-minute slot. It does not mean continuously observed power or switch state.
|
Boiler; meter counters remain archived but do not affect status, completeness, or
|
||||||
The unfinished current slot is excluded from missing counts. Historical recovery
|
interval details. Complete means valid energy and boiler data in the same five-minute
|
||||||
|
slot. It does not mean continuously observed power or switch state.
|
||||||
|
The current slot shows its saved extraction immediately, including partial or failed
|
||||||
|
results. Until a record arrives, it is pending and excluded from missing counts.
|
||||||
|
Future slots remain upcoming. Historical recovery
|
||||||
contributes energy only. After retention cleanup, pruned records no longer contribute
|
contributes energy only. After retention cleanup, pruned records no longer contribute
|
||||||
to this local page; the website keeps the transferred history. Refresh is every 15 seconds.
|
to this local page; the website keeps the transferred history. Refresh is every 15 seconds.
|
||||||
|
|
||||||
|
|||||||
+2
-1
@@ -3,7 +3,7 @@ source_id = "home-gen24"
|
|||||||
archive = "data/archive" # Parquet directory, relative to this configuration file.
|
archive = "data/archive" # Parquet directory, relative to this configuration file.
|
||||||
timezone = "Europe/Zurich" # Used for date-only export/replay arguments.
|
timezone = "Europe/Zurich" # Used for date-only export/replay arguments.
|
||||||
poll_interval_seconds = 300
|
poll_interval_seconds = 300
|
||||||
retention_days = 30 # Days AFTER acknowledgement; pending records never expire.
|
retention_days = 30 # Full local days after acknowledgement; pending days never expire.
|
||||||
|
|
||||||
[inverter]
|
[inverter]
|
||||||
base_url = "http://fronius.local"
|
base_url = "http://fronius.local"
|
||||||
@@ -32,6 +32,7 @@ enabled = false
|
|||||||
pv_system_id = "" # Your Solar.web PV system UUID; required when enabled.
|
pv_system_id = "" # Your Solar.web PV system UUID; required when enabled.
|
||||||
username = "" # Account email; enter real credentials only in private config.toml.
|
username = "" # Account email; enter real credentials only in private config.toml.
|
||||||
password = ""
|
password = ""
|
||||||
|
# Recover from local midnight collector.retention_days ago, including empty days.
|
||||||
lag_seconds = 7200 # Allow the inverter to upload its hourly historical data.
|
lag_seconds = 7200 # Allow the inverter to upload its hourly historical data.
|
||||||
check_interval_seconds = 3600
|
check_interval_seconds = 3600
|
||||||
request_timeout_seconds = 45
|
request_timeout_seconds = 45
|
||||||
@@ -1,4 +1,4 @@
|
|||||||
"""Local Fronius recorder and durable HTTP relay (AGPL-3.0-or-later)."""
|
"""Local Fronius recorder and durable HTTP relay (AGPL-3.0-or-later)."""
|
||||||
|
|
||||||
__version__ = "0.2.0"
|
__version__ = "0.3.0"
|
||||||
|
|
||||||
+3
-2
@@ -102,7 +102,7 @@ def _parser():
|
|||||||
commands.add_parser("sample", help="Record one observation and print its normalized readings")
|
commands.add_parser("sample", help="Record one observation and print its normalized readings")
|
||||||
commands.add_parser("upload", help="Upload the pending queue, stopping on the first failure")
|
commands.add_parser("upload", help="Upload the pending queue, stopping on the first failure")
|
||||||
commands.add_parser("status", help="Show local archive/queue status without contacting the inverter")
|
commands.add_parser("status", help="Show local archive/queue status without contacting the inverter")
|
||||||
commands.add_parser("prune", help="Delete records acknowledged more than retention_days ago")
|
commands.add_parser("prune", help="Delete expired whole local days after acknowledgement retention")
|
||||||
for name, help_text in [
|
for name, help_text in [
|
||||||
("export", "Export archived samples as JSON Lines, including already uploaded samples"),
|
("export", "Export archived samples as JSON Lines, including already uploaded samples"),
|
||||||
("replay", "Requeue archived samples for upload, preserving their IDs"),
|
("replay", "Requeue archived samples for upload, preserving their IDs"),
|
||||||
@@ -125,7 +125,8 @@ def main(argv=None):
|
|||||||
until = date_bound(args.until, config.collector.timezone)
|
until = date_bound(args.until, config.collector.timezone)
|
||||||
if start is not None and until is not None and start >= until:
|
if start is not None and until is not None and start >= until:
|
||||||
raise ValueError("--from must be earlier than --until")
|
raise ValueError("--from must be earlier than --until")
|
||||||
store = Store(config.collector.archive, config.collector.retention_days)
|
store = Store(config.collector.archive, config.collector.retention_days,
|
||||||
|
timezone_name=config.collector.timezone)
|
||||||
if args.command == "run":
|
if args.command == "run":
|
||||||
return _run(config, store)
|
return _run(config, store)
|
||||||
if args.command == "sample":
|
if args.command == "sample":
|
||||||
|
|||||||
+10
-1
@@ -1,7 +1,7 @@
|
|||||||
"""Private runtime configuration; credential fields are excluded from reprs."""
|
"""Private runtime configuration; credential fields are excluded from reprs."""
|
||||||
|
|
||||||
from dataclasses import dataclass, field
|
from dataclasses import dataclass, field
|
||||||
from datetime import datetime, timezone
|
from datetime import datetime, timedelta, timezone
|
||||||
import math
|
import math
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
import re
|
import re
|
||||||
@@ -18,6 +18,15 @@ def utc_text(value: datetime) -> str:
|
|||||||
return value.astimezone(timezone.utc).isoformat(timespec="milliseconds").replace("+00:00", "Z")
|
return value.astimezone(timezone.utc).isoformat(timespec="milliseconds").replace("+00:00", "Z")
|
||||||
|
|
||||||
|
|
||||||
|
def retention_cutoff(now: datetime, days: int, timezone_name: str) -> datetime:
|
||||||
|
"""Start of the oldest retained local calendar day, returned in UTC."""
|
||||||
|
if now.tzinfo is None:
|
||||||
|
raise ValueError("Retention time must include its timezone")
|
||||||
|
zone = ZoneInfo(timezone_name)
|
||||||
|
first = now.astimezone(zone).date() - timedelta(days=days)
|
||||||
|
return datetime.combine(first, datetime.min.time(), zone).astimezone(timezone.utc)
|
||||||
|
|
||||||
|
|
||||||
def date_bound(value: str | None, timezone_name: str) -> str | None:
|
def date_bound(value: str | None, timezone_name: str) -> str | None:
|
||||||
"""Date bounds are local midnights; timestamp bounds may specify an offset."""
|
"""Date bounds are local midnights; timestamp bounds may specify an offset."""
|
||||||
if value is None:
|
if value is None:
|
||||||
|
|||||||
@@ -61,7 +61,8 @@ def daily_coverage(rows, timezone_name, days, now=None):
|
|||||||
date = datetime.fromtimestamp(observed / 1000, zone).date().isoformat()
|
date = datetime.fromtimestamp(observed / 1000, zone).date().isoformat()
|
||||||
observations[date] += 1
|
observations[date] += 1
|
||||||
bucket = observed // BUCKET_MS * BUCKET_MS
|
bucket = observed // BUCKET_MS * BUCKET_MS
|
||||||
mask = sum(1 << index for index in range(3) if row[2 + index * 2])
|
# Archive rows retain meter flags, but status coverage tracks Energy and Boiler only.
|
||||||
|
mask = int(bool(row[2])) | (int(bool(row[6])) << 1)
|
||||||
slots_by_time[bucket] = slots_by_time.get(bucket, 0) | mask
|
slots_by_time[bucket] = slots_by_time.get(bucket, 0) | mask
|
||||||
result = []
|
result = []
|
||||||
for offset in range(days + 1):
|
for offset in range(days + 1):
|
||||||
@@ -71,18 +72,22 @@ def daily_coverage(rows, timezone_name, days, now=None):
|
|||||||
start_ms = round(start.timestamp() * 1000)
|
start_ms = round(start.timestamp() * 1000)
|
||||||
expected = round((end.timestamp() - start.timestamp()) * 1000) // BUCKET_MS
|
expected = round((end.timestamp() - start.timestamp()) * 1000) // BUCKET_MS
|
||||||
elapsed = expected if date < today else max(0, min(expected, (upper - start_ms) // BUCKET_MS))
|
elapsed = expected if date < today else max(0, min(expected, (upper - start_ms) // BUCKET_MS))
|
||||||
|
# This is extraction coverage: show the current slot as soon as its
|
||||||
|
# record is saved, without waiting for the five-minute period to end.
|
||||||
|
# An unrecorded current slot stays pending and is not counted missing.
|
||||||
|
if date == today and elapsed < expected and start_ms + elapsed * BUCKET_MS in slots_by_time:
|
||||||
|
elapsed += 1
|
||||||
slots = []
|
slots = []
|
||||||
for index in range(expected):
|
for index in range(expected):
|
||||||
slots.append(slots_by_time.get(start_ms + index * BUCKET_MS, 0) if index < elapsed else -1)
|
slots.append(slots_by_time.get(start_ms + index * BUCKET_MS, 0) if index < elapsed else -1)
|
||||||
complete = sum(mask == 7 for mask in slots)
|
complete = sum(mask == 3 for mask in slots)
|
||||||
count = observations[date.isoformat()]
|
count = observations[date.isoformat()]
|
||||||
status = "in_progress" if date == today else "complete" if complete == expected else "partial" if count else "missing"
|
status = "in_progress" if date == today else "complete" if complete == expected else "partial" if count else "missing"
|
||||||
result.append({
|
result.append({
|
||||||
"date": date.isoformat(), "day_start": utc_text(start), "expected": expected, "elapsed": elapsed,
|
"date": date.isoformat(), "day_start": utc_text(start), "expected": expected, "elapsed": elapsed,
|
||||||
"complete": complete, "missing": elapsed - complete, "samples": count, "status": status,
|
"complete": complete, "missing": elapsed - complete, "samples": count, "status": status,
|
||||||
"energy_complete": sum(mask >= 0 and bool(mask & 1) for mask in slots),
|
"energy_complete": sum(mask >= 0 and bool(mask & 1) for mask in slots),
|
||||||
"meter_complete": sum(mask >= 0 and bool(mask & 2) for mask in slots),
|
"boiler_complete": sum(mask >= 0 and bool(mask & 2) for mask in slots),
|
||||||
"boiler_complete": sum(mask >= 0 and bool(mask & 4) for mask in slots),
|
|
||||||
"slots": slots,
|
"slots": slots,
|
||||||
})
|
})
|
||||||
return result
|
return result
|
||||||
@@ -11,7 +11,7 @@ import os
|
|||||||
from uuid import NAMESPACE_URL, uuid5
|
from uuid import NAMESPACE_URL, uuid5
|
||||||
from zoneinfo import ZoneInfo
|
from zoneinfo import ZoneInfo
|
||||||
|
|
||||||
from .config import utc_text
|
from .config import retention_cutoff, utc_text
|
||||||
from .coverage import milliseconds
|
from .coverage import milliseconds
|
||||||
from .http import TransportError
|
from .http import TransportError
|
||||||
from .solarweb import ChartDataError, SolarWebClient
|
from .solarweb import ChartDataError, SolarWebClient
|
||||||
@@ -92,14 +92,14 @@ class HistoryRecovery:
|
|||||||
def recover(self, start=None, until=None, *, now=None, stop=None):
|
def recover(self, start=None, until=None, *, now=None, stop=None):
|
||||||
now = now or datetime.now(timezone.utc)
|
now = now or datetime.now(timezone.utc)
|
||||||
latest = int((now - timedelta(seconds=self.config.lag_seconds)).timestamp()) // 300 * 300
|
latest = int((now - timedelta(seconds=self.config.lag_seconds)).timestamp()) // 300 * 300
|
||||||
# Check the whole retained window, including days before any live observation.
|
# Always include the entire oldest local day, even across a DST change.
|
||||||
# Round up so recovery never requests an interval older than the cutoff.
|
earliest = int(retention_cutoff(now, self.store.retention_days, self.zone.key).timestamp())
|
||||||
earliest = math.ceil((now - timedelta(days=self.store.retention_days)).timestamp() / 300) * 300
|
|
||||||
lower = milliseconds(start) // 300_000 * 300 if start else earliest
|
lower = milliseconds(start) // 300_000 * 300 if start else earliest
|
||||||
upper = min(milliseconds(until) // 300_000 * 300, latest) if until else latest
|
upper = min(milliseconds(until) // 300_000 * 300, latest) if until else latest
|
||||||
if lower >= upper:
|
if lower >= upper:
|
||||||
return 0
|
return 0
|
||||||
if (start is not None or until is not None) and upper - lower > 30 * 86400:
|
manual_limit = datetime.fromtimestamp(lower, self.zone) + timedelta(days=30)
|
||||||
|
if (start is not None or until is not None) and upper > manual_limit.timestamp():
|
||||||
raise ValueError("Recover at most 30 days at once")
|
raise ValueError("Recover at most 30 days at once")
|
||||||
stamp = lambda epoch: utc_text(datetime.fromtimestamp(epoch, timezone.utc))
|
stamp = lambda epoch: utc_text(datetime.fromtimestamp(epoch, timezone.utc))
|
||||||
present = self.store.energy_slots(self.source_id, stamp(lower), stamp(upper))
|
present = self.store.energy_slots(self.source_id, stamp(lower), stamp(upper))
|
||||||
|
|||||||
@@ -109,7 +109,7 @@ class StatusServer:
|
|||||||
now = datetime.now(timezone.utc)
|
now = datetime.now(timezone.utc)
|
||||||
attempt = archive["last_observation"]
|
attempt = archive["last_observation"]
|
||||||
age = (now.timestamp() * 1000 - milliseconds(attempt)) / 1000 if attempt else None
|
age = (now.timestamp() * 1000 - milliseconds(attempt)) / 1000 if attempt else None
|
||||||
errors = list(archive["last_errors"])
|
errors = [source for source in archive["last_errors"] if source != "meter"]
|
||||||
success = self.store.last_success(self.config.collector.source_id)
|
success = self.store.last_success(self.config.collector.source_id)
|
||||||
status = "waiting" if age is None else "stale" if age > max(90, self.config.collector.poll_interval_seconds * 3) or age < -90 else "partial" if errors or success != attempt else "ok"
|
status = "waiting" if age is None else "stale" if age > max(90, self.config.collector.poll_interval_seconds * 3) or age < -90 else "partial" if errors or success != attempt else "ok"
|
||||||
body = {
|
body = {
|
||||||
|
|||||||
+27
-10
@@ -13,11 +13,12 @@ from pathlib import Path
|
|||||||
import tempfile
|
import tempfile
|
||||||
import threading
|
import threading
|
||||||
import time
|
import time
|
||||||
|
from zoneinfo import ZoneInfo
|
||||||
|
|
||||||
import pyarrow as pa
|
import pyarrow as pa
|
||||||
import pyarrow.parquet as pq
|
import pyarrow.parquet as pq
|
||||||
|
|
||||||
from .config import utc_text
|
from .config import retention_cutoff, utc_text
|
||||||
from .coverage import daily_coverage, milliseconds, sample_coverage
|
from .coverage import daily_coverage, milliseconds, sample_coverage
|
||||||
|
|
||||||
NUMBERS = (
|
NUMBERS = (
|
||||||
@@ -45,12 +46,13 @@ class StoreError(OSError):
|
|||||||
|
|
||||||
|
|
||||||
class Store:
|
class Store:
|
||||||
def __init__(self, path: Path, retention_days=30):
|
def __init__(self, path: Path, retention_days=30, *, timezone_name="Europe/Zurich"):
|
||||||
self.path = Path(path)
|
self.path = Path(path)
|
||||||
if self.path.is_file():
|
if self.path.is_file():
|
||||||
raise StoreError("collector.archive must be a directory")
|
raise StoreError("collector.archive must be a directory")
|
||||||
self.path.mkdir(parents=True, exist_ok=True, mode=0o700)
|
self.path.mkdir(parents=True, exist_ok=True, mode=0o700)
|
||||||
self.retention_days = retention_days
|
self.retention_days = retention_days
|
||||||
|
self.zone = ZoneInfo(timezone_name)
|
||||||
self._thread_lock = threading.RLock()
|
self._thread_lock = threading.RLock()
|
||||||
descriptor = os.open(self.path / ".lock", os.O_CREAT | os.O_WRONLY, 0o600)
|
descriptor = os.open(self.path / ".lock", os.O_CREAT | os.O_WRONLY, 0o600)
|
||||||
os.close(descriptor)
|
os.close(descriptor)
|
||||||
@@ -190,16 +192,32 @@ class Store:
|
|||||||
return count
|
return count
|
||||||
|
|
||||||
def prune(self, *, now=None):
|
def prune(self, *, now=None):
|
||||||
"""Retain pending records; age acknowledged records from their receipt time."""
|
"""Remove whole local observation days only after all their acknowledgements expire."""
|
||||||
cutoff = utc_text((now or datetime.now(timezone.utc)) - timedelta(days=self.retention_days))
|
boundary = retention_cutoff(now or datetime.now(timezone.utc), self.retention_days, self.zone.key)
|
||||||
|
cutoff, first_day = utc_text(boundary), boundary.astimezone(self.zone).date()
|
||||||
deleted = 0
|
deleted = 0
|
||||||
|
local_day = lambda row: datetime.fromtimestamp(row["observed_ms"] / 1000, self.zone).date()
|
||||||
with self._locked(write=True):
|
with self._locked(write=True):
|
||||||
|
partitions, expired, protected = [], set(), set()
|
||||||
|
# A local day can span two UTC partitions. One pending or recently
|
||||||
|
# acknowledged record protects the rest of its local day as well.
|
||||||
for path in self._files():
|
for path in self._files():
|
||||||
index = self._read(path, ["uploaded_at"])
|
index = self._read(path, ["observed_ms", "uploaded_at"])
|
||||||
if not any(row["uploaded_at"] is not None and row["uploaded_at"] < cutoff for row in index):
|
days = set()
|
||||||
|
for row in index:
|
||||||
|
day = local_day(row)
|
||||||
|
days.add(day)
|
||||||
|
if day < first_day and row["uploaded_at"] is not None and row["uploaded_at"] < cutoff:
|
||||||
|
expired.add(day)
|
||||||
|
else:
|
||||||
|
protected.add(day)
|
||||||
|
partitions.append((path, days))
|
||||||
|
expired -= protected
|
||||||
|
for path, days in partitions:
|
||||||
|
if not days & expired:
|
||||||
continue
|
continue
|
||||||
rows = self._read(path)
|
rows = self._read(path)
|
||||||
retained = [row for row in rows if row["uploaded_at"] is None or row["uploaded_at"] >= cutoff]
|
retained = [row for row in rows if local_day(row) not in expired]
|
||||||
self._write(path, retained)
|
self._write(path, retained)
|
||||||
deleted += len(rows) - len(retained)
|
deleted += len(rows) - len(retained)
|
||||||
return deleted
|
return deleted
|
||||||
@@ -229,7 +247,6 @@ class Store:
|
|||||||
}
|
}
|
||||||
|
|
||||||
def coverage(self, source_id, timezone_name, days, now=None):
|
def coverage(self, source_id, timezone_name, days, now=None):
|
||||||
from zoneinfo import ZoneInfo
|
|
||||||
now = now or datetime.now(timezone.utc)
|
now = now or datetime.now(timezone.utc)
|
||||||
first = now.astimezone(ZoneInfo(timezone_name)).date() - timedelta(days=days)
|
first = now.astimezone(ZoneInfo(timezone_name)).date() - timedelta(days=days)
|
||||||
lower = round(datetime.combine(first, datetime.min.time(), ZoneInfo(timezone_name)).timestamp() * 1000)
|
lower = round(datetime.combine(first, datetime.min.time(), ZoneInfo(timezone_name)).timestamp() * 1000)
|
||||||
@@ -247,8 +264,8 @@ class Store:
|
|||||||
def last_success(self, source_id):
|
def last_success(self, source_id):
|
||||||
with self._locked():
|
with self._locked():
|
||||||
for path in reversed(self._files()):
|
for path in reversed(self._files()):
|
||||||
rows = self._read(path, ["source_id", "observed_ms", "power_ok", "meter_ok", "boiler_ok"])
|
rows = self._read(path, ["source_id", "observed_ms", "power_ok", "boiler_ok"])
|
||||||
stamps = [row["observed_ms"] for row in rows if row["source_id"] == source_id and row["power_ok"] and row["meter_ok"] and row["boiler_ok"]]
|
stamps = [row["observed_ms"] for row in rows if row["source_id"] == source_id and row["power_ok"] and row["boiler_ok"]]
|
||||||
if stamps:
|
if stamps:
|
||||||
return utc_text(datetime.fromtimestamp(max(stamps) / 1000, timezone.utc))
|
return utc_text(datetime.fromtimestamp(max(stamps) / 1000, timezone.utc))
|
||||||
return None
|
return None
|
||||||
|
|||||||
+11
-12
@@ -24,19 +24,9 @@
|
|||||||
<p id="notice" class="notice" role="alert" hidden></p>
|
<p id="notice" class="notice" role="alert" hidden></p>
|
||||||
<section class="metrics" aria-label="Current collection status">
|
<section class="metrics" aria-label="Current collection status">
|
||||||
<article class="metric"><p class="metric-label">Last extraction</p><strong id="last-extraction">—</strong><p id="last-extraction-date">No observation received yet</p><small id="last-success">Last successful extraction: —</small></article>
|
<article class="metric"><p class="metric-label">Last extraction</p><strong id="last-extraction">—</strong><p id="last-extraction-date">No observation received yet</p><small id="last-success">Last successful extraction: —</small></article>
|
||||||
<article class="metric"><p class="metric-label">Yesterday’s coverage</p><strong id="yesterday">— <span>/ —</span></strong><p id="yesterday-detail">Five-minute intervals</p><small>Energy, meter and boiler status</small></article>
|
<article class="metric"><p class="metric-label">Yesterday’s coverage</p><strong id="yesterday">— <span>/ —</span></strong><p id="yesterday-detail">Five-minute intervals</p><small>Energy and boiler status</small></article>
|
||||||
<article class="metric"><p class="metric-label">Waiting to upload</p><strong id="pending">—</strong><p id="upload-detail">Checking delivery</p><small id="last-upload">Last acknowledgement: —</small></article>
|
<article class="metric"><p class="metric-label">Waiting to upload</p><strong id="pending">—</strong><p id="upload-detail">Checking delivery</p><small id="last-upload">Last acknowledgement: —</small></article>
|
||||||
<article class="metric"><p class="metric-label">Local archive</p><strong id="samples">—</strong><p>Observations retained in Parquet</p><small id="retention">Acknowledged readings kept for 30 days</small></article>
|
<article class="metric"><p class="metric-label">Local archive</p><strong id="samples">—</strong><p>Observations retained in Parquet</p><small id="retention">Retention: 30 full calendar days</small></article>
|
||||||
</section>
|
|
||||||
<section class="panel">
|
|
||||||
<div class="panel-heading"><div><p class="eyebrow">DAILY COVERAGE</p><h2>Five-minute completeness</h2></div><label class="range">History <select id="history"><option value="7">7 days</option><option value="14" selected>14 days</option><option value="30">30 days</option><option value="90">90 days</option></select></label></div>
|
|
||||||
<p class="panel-note">Each count shows five-minute intervals with valid data out of the elapsed intervals for that day. Complete intervals have all three sources. Polling does not guarantee continuous observation. Solar.web recovery fills Energy only.</p>
|
|
||||||
<dl class="coverage-definitions">
|
|
||||||
<div><dt>Energy</dt><dd id="energy-description">Solar production, household consumption and electricity imported from or exported to the grid. Available from live readings or recovered Solar.web history.</dd></div>
|
|
||||||
<div><dt>Meter</dt><dd id="meter-description">Running totals of electricity bought from and sold to the grid. Requires live meter readings; Solar.web history cannot fill these counters.</dd></div>
|
|
||||||
<div><dt>Boiler</dt><dd id="boiler-description">Boiler control output: enabled or disabled. Both states count as valid data. Heating energy is not measured, and Solar.web cannot recover past switch states.</dd></div>
|
|
||||||
</dl>
|
|
||||||
<div class="table-scroll"><table><thead><tr><th scope="col">Day</th><th scope="col">Complete intervals</th><th scope="col" class="source-column" aria-describedby="energy-description">Energy</th><th scope="col" class="source-column" aria-describedby="meter-description">Meter</th><th scope="col" class="source-column" aria-describedby="boiler-description">Boiler</th><th scope="col">Status</th></tr></thead><tbody id="days"><tr><td colspan="6" class="empty">Waiting for the local archive…</td></tr></tbody></table></div>
|
|
||||||
</section>
|
</section>
|
||||||
<section class="panel detail" aria-label="Selected day interval details">
|
<section class="panel detail" aria-label="Selected day interval details">
|
||||||
<div class="panel-heading"><div><p class="eyebrow">FIVE-MINUTE INTERVALS</p><h2 id="selected-date">Select a day</h2></div><div class="legend"><span><i class="complete"></i>Complete</span><span><i class="partial"></i>Partial</span><span><i class="missing"></i>Missing</span><span><i class="future"></i>Upcoming</span></div></div>
|
<div class="panel-heading"><div><p class="eyebrow">FIVE-MINUTE INTERVALS</p><h2 id="selected-date">Select a day</h2></div><div class="legend"><span><i class="complete"></i>Complete</span><span><i class="partial"></i>Partial</span><span><i class="missing"></i>Missing</span><span><i class="future"></i>Upcoming</span></div></div>
|
||||||
@@ -44,6 +34,15 @@
|
|||||||
<div id="slots" class="slots" aria-label="Five-minute coverage"></div>
|
<div id="slots" class="slots" aria-label="Five-minute coverage"></div>
|
||||||
<p id="missing-ranges" class="gap-summary"></p>
|
<p id="missing-ranges" class="gap-summary"></p>
|
||||||
</section>
|
</section>
|
||||||
|
<section class="panel">
|
||||||
|
<div class="panel-heading"><div><p class="eyebrow">DAILY COVERAGE</p><h2>Five-minute completeness</h2></div><label class="range">History <select id="history"><option value="7">7 days</option><option value="14" selected>14 days</option><option value="30">30 days</option><option value="90">90 days</option></select></label></div>
|
||||||
|
<p class="panel-note">Each count shows five-minute intervals with valid data out of the intervals assessed so far. Saved extractions appear immediately, including the current interval. Complete intervals have both Energy and Boiler data. Polling does not guarantee continuous observation. Solar.web recovery fills Energy only.</p>
|
||||||
|
<dl class="coverage-definitions">
|
||||||
|
<div><dt>Energy</dt><dd id="energy-description">Solar production, household consumption and electricity imported from or exported to the grid. Available from live readings or recovered Solar.web history.</dd></div>
|
||||||
|
<div><dt>Boiler</dt><dd id="boiler-description">Boiler control output: enabled or disabled. Both states count as valid data. Heating energy is not measured, and Solar.web cannot recover past switch states.</dd></div>
|
||||||
|
</dl>
|
||||||
|
<div class="table-scroll"><table><thead><tr><th scope="col">Day</th><th scope="col">Complete intervals</th><th scope="col" class="source-column" aria-describedby="energy-description">Energy</th><th scope="col" class="source-column" aria-describedby="boiler-description">Boiler</th><th scope="col">Status</th></tr></thead><tbody id="days"><tr><td colspan="5" class="empty">Waiting for the local archive…</td></tr></tbody></table></div>
|
||||||
|
</section>
|
||||||
<footer><span>Read-only status · refreshes every 15 seconds</span><span id="version">solar-relay</span></footer>
|
<footer><span>Read-only status · refreshes every 15 seconds</span><span id="version">solar-relay</span></footer>
|
||||||
<noscript><p class="notice">Enable JavaScript to load extraction and coverage status.</p></noscript>
|
<noscript><p class="notice">Enable JavaScript to load extraction and coverage status.</p></noscript>
|
||||||
</main>
|
</main>
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
:root{color-scheme:dark;--bg:#101518;--panel:#171e22;--line:#2b353a;--ink:#eef1ed;--muted:#9ba9ac;--amber:#e8ba70;--green:#7ad6ac;--red:#e68d86;--radius:12px;font-family:Inter,ui-sans-serif,system-ui,-apple-system,"Segoe UI",sans-serif;font-synthesis:none}
|
:root{color-scheme:dark;--bg:#101518;--panel:#171e22;--line:#2b353a;--ink:#eef1ed;--muted:#9ba9ac;--amber:#e8ba70;--green:#7ad6ac;--red:#e68d86;--radius:12px;font-family:Inter,ui-sans-serif,system-ui,-apple-system,"Segoe UI",sans-serif;font-synthesis:none}
|
||||||
*{box-sizing:border-box}body{margin:0;background:var(--bg);color:var(--ink)}button,select{font:inherit}button{cursor:pointer}button:focus-visible,select:focus-visible{outline:2px solid var(--amber);outline-offset:4px}main{max-width:1216px;margin:auto;padding:0 32px}.masthead{height:100px;display:flex;align-items:center;justify-content:space-between;border-bottom:1px solid var(--line)}.brand{display:flex;align-items:center;gap:12px;font-size:13px;letter-spacing:.14em;font-weight:750}.brand small{display:block;color:var(--muted);font-size:9px;letter-spacing:.22em;margin-top:5px}.sun{font-size:36px;color:var(--amber);line-height:1}.header-actions{display:flex;align-items:center;gap:16px}.badge{display:inline-flex;align-items:center;gap:7px;white-space:nowrap;border-radius:20px;padding:6px 11px;font-size:11px;font-weight:650;line-height:1.3;background:#252e32;color:var(--muted)}.badge::before{content:"";width:5px;height:5px;border-radius:50%;background:currentColor}.badge.ok,.badge.complete{color:var(--green);background:#193a2e}.badge.partial,.badge.in_progress{color:var(--amber);background:#3a3121}.badge.stale{color:var(--red);background:#3d2829}.badge.missing,.badge.waiting,.badge.neutral{color:var(--muted);background:#252e32}.refresh{border:1px solid var(--line);background:transparent;color:var(--ink);border-radius:7px;padding:8px 12px;font-size:13px}.refresh:hover{border-color:var(--muted)}.refresh:disabled{opacity:.45;cursor:wait}.intro{padding:42px 0 28px}.eyebrow{color:var(--amber);font-size:10px;font-weight:700;letter-spacing:.17em;margin:0 0 13px}h1{font-size:clamp(27px,3.4vw,42px);letter-spacing:-.045em;font-weight:550;line-height:1.14;margin:0 0 13px}.description{color:var(--muted);font-size:14px;line-height:1.6;margin:0}.context{display:flex;align-items:center;gap:10px;font-size:11px;color:var(--muted);margin-top:20px;flex-wrap:wrap}.separator{color:#48565c}.metrics{display:grid;grid-template-columns:repeat(4,minmax(0,1fr));gap:14px;margin:0 0 28px}.metric{border:1px solid var(--line);border-radius:var(--radius);padding:23px 20px;background:var(--panel)}.metric-label{font-size:11px;font-weight:550;color:var(--muted);margin:0 0 18px}.metric strong{display:block;font-size:30px;font-weight:550;font-variant-numeric:tabular-nums;letter-spacing:-.04em;line-height:1.1;min-height:34px}.metric strong span{color:var(--muted);font-size:17px;font-weight:400;letter-spacing:normal}.metric p:not(.metric-label){font-size:11px;margin:12px 0 7px;color:#d1d9d7}.metric small{font-size:10px;color:var(--muted);line-height:1.6;display:block}.panel{background:var(--panel);border:1px solid var(--line);border-radius:var(--radius);margin-bottom:22px;overflow:hidden}.panel-heading{display:flex;align-items:center;justify-content:space-between;gap:16px;padding:24px 24px 0}.panel-heading .eyebrow{margin-bottom:7px}h2{font-size:19px;font-weight:550;letter-spacing:-.025em;margin:0}.panel-note{font-size:11px;line-height:1.7;color:var(--muted);padding:0 24px;margin:12px 0 22px}.range{display:flex;align-items:center;gap:9px;color:var(--muted);font-size:11px;white-space:nowrap}select{background:#20292e;border:1px solid #354149;border-radius:6px;color:var(--ink);padding:7px 27px 7px 9px;font-size:11px}.table-scroll{overflow:auto}table{width:100%;border-collapse:collapse;text-align:left;font-size:12px;font-variant-numeric:tabular-nums}th{background:#1c252a;border-top:1px solid var(--line);border-bottom:1px solid var(--line);padding:11px 18px;font-size:9px;font-weight:650;letter-spacing:.07em;text-transform:uppercase;color:var(--muted)}th:first-child,td:first-child{padding-left:24px}td{padding:13px 18px;border-bottom:1px solid #283137;white-space:nowrap}tbody tr:last-child td{border-bottom:0}tr.selected{background:#222d32}tr:hover{background:#202a2f}.day-select{display:block;text-align:left;color:var(--ink);background:transparent;border:0;padding:0;line-height:1.55;font-size:12px}.day-select small{display:block;color:var(--muted);font-size:10px}.coverage-number{display:flex;align-items:baseline;gap:5px;font-size:12px}.coverage-number small{font-size:10px;color:var(--muted)}.progress{height:3px;width:126px;border-radius:2px;background:#303c42;margin-top:7px;overflow:hidden}.progress span{display:block;height:100%;background:var(--green)}.source-count{color:#c1ceca;font-size:11px}.source-count.incomplete{color:var(--amber)}.empty{padding:40px!important;text-align:center;color:var(--muted)}.legend{display:flex;gap:14px;flex-wrap:wrap;font-size:10px;color:var(--muted)}.legend span{display:flex;align-items:center;gap:6px}.legend i{display:block;width:7px;height:7px;border-radius:2px}.slots{display:flex;gap:5px;padding:0 24px;overflow-x:auto}.hour{display:grid;grid-template-rows:16px repeat(12,5px);gap:3px;flex:1;min-width:14px}.hour-label{font-size:9px;color:var(--muted);font-variant-numeric:tabular-nums;text-align:center}.slot{display:block;border-radius:1px;min-height:5px}.complete{background:#76c9a4}.partial{background:#bd9558}.missing{bacLine truncated
|
*{box-sizing:border-box}body{margin:0;background:var(--bg);color:var(--ink)}button,select{font:inherit}button{cursor:pointer}button:focus-visible,select:focus-visible{outline:2px solid var(--amber);outline-offset:4px}main{max-width:1216px;margin:auto;padding:0 32px}.masthead{height:100px;display:flex;align-items:center;justify-content:space-between;border-bottom:1px solid var(--line)}.brand{display:flex;align-items:center;gap:12px;font-size:13px;letter-spacing:.14em;font-weight:750}.brand small{display:block;color:var(--muted);font-size:9px;letter-spacing:.22em;margin-top:5px}.sun{font-size:36px;color:var(--amber);line-height:1}.header-actions{display:flex;align-items:center;gap:16px}.badge{display:inline-flex;align-items:center;gap:7px;white-space:nowrap;border-radius:20px;padding:6px 11px;font-size:11px;font-weight:650;line-height:1.3;background:#252e32;color:var(--muted)}.badge::before{content:"";width:5px;height:5px;border-radius:50%;background:currentColor}.badge.ok,.badge.complete{color:var(--green);background:#193a2e}.badge.partial,.badge.in_progress{color:var(--amber);background:#3a3121}.badge.stale{color:var(--red);background:#3d2829}.badge.missing,.badge.waiting,.badge.neutral{color:var(--muted);background:#252e32}.refresh{border:1px solid var(--line);background:transparent;color:var(--ink);border-radius:7px;padding:8px 12px;font-size:13px}.refresh:hover{border-color:var(--muted)}.refresh:disabled{opacity:.45;cursor:wait}.intro{padding:42px 0 28px}.eyebrow{color:var(--amber);font-size:10px;font-weight:700;letter-spacing:.17em;margin:0 0 13px}h1{font-size:clamp(27px,3.4vw,42px);letter-spacing:-.045em;font-weight:550;line-height:1.14;margin:0 0 13px}.description{color:var(--muted);font-size:14px;line-height:1.6;margin:0}.context{display:flex;align-items:center;gap:10px;font-size:11px;color:var(--muted);margin-top:20px;flex-wrap:wrap}.separator{color:#48565c}.metrics{display:grid;grid-template-columns:repeat(4,minmax(0,1fr));gap:14px;margin:0 0 28px}.metric{border:1px solid var(--line);border-radius:var(--radius);padding:23px 20px;background:var(--panel)}.metric-label{font-size:11px;font-weight:550;color:var(--muted);margin:0 0 18px}.metric strong{display:block;font-size:30px;font-weight:550;font-variant-numeric:tabular-nums;letter-spacing:-.04em;line-height:1.1;min-height:34px}.metric strong span{color:var(--muted);font-size:17px;font-weight:400;letter-spacing:normal}.metric p:not(.metric-label){font-size:11px;margin:12px 0 7px;color:#d1d9d7}.metric small{font-size:10px;color:var(--muted);line-height:1.6;display:block}.panel{background:var(--panel);border:1px solid var(--line);border-radius:var(--radius);margin-bottom:22px;overflow:hidden}.panel-heading{display:flex;align-items:center;justify-content:space-between;gap:16px;padding:24px 24px 0}.panel-heading .eyebrow{margin-bottom:7px}h2{font-size:19px;font-weight:550;letter-spacing:-.025em;margin:0}.panel-note{font-size:11px;line-height:1.7;color:var(--muted);padding:0 24px;margin:12px 0 22px}.range{display:flex;align-items:center;gap:9px;color:var(--muted);font-size:11px;white-space:nowrap}select{background:#20292e;border:1px solid #354149;border-radius:6px;color:var(--ink);padding:7px 27px 7px 9px;font-size:11px}.table-scroll{overflow:auto}table{width:100%;border-collapse:collapse;text-align:left;font-size:12px;font-variant-numeric:tabular-nums}th{background:#1c252a;border-top:1px solid var(--line);border-bottom:1px solid var(--line);padding:11px 18px;font-size:9px;font-weight:650;letter-spacing:.07em;text-transform:uppercase;color:var(--muted)}th:first-child,td:first-child{padding-left:24px}td{padding:13px 18px;border-bottom:1px solid #283137;white-space:nowrap}tbody tr:last-child td{border-bottom:0}tr.selected{background:#222d32}tr:hover{background:#202a2f}.day-select{display:block;text-align:left;color:var(--ink);background:transparent;border:0;padding:0;line-height:1.55;font-size:12px}.day-select small{display:block;color:var(--muted);font-size:10px}.coverage-number{display:flex;align-items:baseline;gap:5px;font-size:12px}.coverage-number small{font-size:10px;color:var(--muted)}.progress{height:3px;width:126px;border-radius:2px;background:#303c42;margin-top:7px;overflow:hidden}.progress span{display:block;height:100%;background:var(--green)}.source-count{color:#c1ceca;font-size:11px}.source-count.incomplete{color:var(--amber)}.empty{padding:40px!important;text-align:center;color:var(--muted)}.legend{display:flex;gap:14px;flex-wrap:wrap;font-size:10px;color:var(--muted)}.legend span{display:flex;align-items:center;gap:6px}.legend i{display:block;width:7px;height:7px;border-radius:2px}.slots{display:flex;gap:5px;padding:0 24px;overflow-x:auto}.hour{display:grid;grid-template-rows:16px repeat(12,5px);gap:3px;flex:1;min-width:14px}.hour-label{font-size:9px;color:var(--muted);font-variant-numeric:tabular-nums;text-align:center}.slot{display:block;border-radius:1px;min-height:5px}.complete{background:#76c9a4}.partial{background:#bd9558}.missing{bacLine truncated
|
||||||
.coverage-definitions{display:grid;grid-template-columns:repeat(3,minmax(0,1fr));gap:20px;margin:0;padding:0 24px 24px}.coverage-definitions dt{font-size:11px;font-weight:650;margin-bottom:6px}.coverage-definitions dd{margin:0;color:var(--muted);font-size:11px;line-height:1.7}
|
.coverage-definitions{display:grid;grid-template-columns:repeat(2,minmax(0,1fr));gap:20px;margin:0;padding:0 24px 24px}.coverage-definitions dt{font-size:11px;font-weight:650;margin-bottom:6px}.coverage-definitions dd{margin:0;color:var(--muted);font-size:11px;line-height:1.7}
|
||||||
@media(max-width:900px){.metrics{grid-template-columns:repeat(2,minmax(0,1fr))}.metric{padding:20px}.source-column{display:none}.legend{max-width:215px;gap:10px}.slots{gap:4px}.progress{width:115px}}
|
@media(max-width:900px){.metrics{grid-template-columns:repeat(2,minmax(0,1fr))}.metric{padding:20px}.source-column{display:none}.legend{max-width:215px;gap:10px}.slots{gap:4px}.progress{width:115px}}
|
||||||
@media(max-width:560px){.coverage-definitions{grid-template-columns:1fr;gap:14px;padding:0 16px 20px}.coverage-definitions dt,.coverage-definitions dd{font-size:10px}}
|
@media(max-width:560px){.coverage-definitions{grid-template-columns:1fr;gap:14px;padding:0 16px 20px}.coverage-definitions dt,.coverage-definitions dd{font-size:10px}}
|
||||||
@media(max-width:560px){main{padding:0 16px}.masthead{height:82px}.brand{font-size:11px;gap:8px}.sun{font-size:30px}.header-actions{gap:8px}.refresh span{display:none}.refresh{font-size:17px;padding:4px 9px}.badge{font-size:10px;padding:6px 8px}.intro{padding-top:30px}.metrics{gap:10px;margin-bottom:20px}.metric{padding:17px 14px}.metric strong{font-size:26px}.metric strong span{font-size:14px}.metric-label{margin-bottom:14px}.metric small{font-size:9px}.panel-heading{padding:20px 16px 0;align-items:flex-start}h2{font-size:16px}.panel-note{padding:0 16px;font-size:10px}th,td{padding-left:11px;padding-right:11px}th:first-child,td:first-child{padding-left:16px}.progress{width:88px}.range{flex-direction:column;align-items:flex-end;gap:5px}.legend{font-size:9px;gap:7px;max-width:150px}.slots{padding:0 16px;gap:3px}.hour{min-width:10px;grid-template-rows:16px repeat(12,5px);gap:3px}.hour-label{font-size:8px}.gap-summary{padding:16px;font-size:10px}.day-select{font-size:11px}footer{font-size:9px;flex-wrap:wrap}.detail .panel-heading{gap:8px}}
|
@media(max-width:560px){main{padding:0 16px}.masthead{height:82px}.brand{font-size:11px;gap:8px}.sun{font-size:30px}.header-actions{gap:8px}.refresh span{display:none}.refresh{font-size:17px;padding:4px 9px}.badge{font-size:10px;padding:6px 8px}.intro{padding-top:30px}.metrics{gap:10px;margin-bottom:20px}.metric{padding:17px 14px}.metric strong{font-size:26px}.metric strong span{font-size:14px}.metric-label{margin-bottom:14px}.metric small{font-size:9px}.panel-heading{padding:20px 16px 0;align-items:flex-start}h2{font-size:16px}.panel-note{padding:0 16px;font-size:10px}th,td{padding-left:11px;padding-right:11px}th:first-child,td:first-child{padding-left:16px}.progress{width:88px}.range{flex-direction:column;align-items:flex-end;gap:5px}.legend{font-size:9px;gap:7px;max-width:150px}.slots{padding:0 16px;gap:3px}.hour{min-width:10px;grid-template-rows:16px repeat(12,5px);gap:3px}.hour-label{font-size:8px}.gap-summary{padding:16px;font-size:10px}.day-select{font-size:11px}footer{font-size:9px;flex-wrap:wrap}.detail .panel-heading{gap:8px}}
|
||||||
@@ -48,7 +48,7 @@
|
|||||||
const fill = addText(progress,'span','');
|
const fill = addText(progress,'span','');
|
||||||
fill.style.width = (day.elapsed ? day.complete/day.elapsed*100:0)+'%';
|
fill.style.width = (day.elapsed ? day.complete/day.elapsed*100:0)+'%';
|
||||||
row.append(amount);
|
row.append(amount);
|
||||||
for(const field of ['energy_complete','meter_complete','boiler_complete']) {
|
for(const field of ['energy_complete','boiler_complete']) {
|
||||||
const cell = td('source-column');
|
const cell = td('source-column');
|
||||||
addText(cell,'span',number(day[field])+'/'+number(day.elapsed),'source-count'+(day[field]<day.elapsed?' incomplete':''));
|
addText(cell,'span',number(day[field])+'/'+number(day.elapsed),'source-count'+(day[field]<day.elapsed?' incomplete':''));
|
||||||
row.append(cell);
|
row.append(cell);
|
||||||
@@ -63,8 +63,9 @@
|
|||||||
const day = current.days.find(item => item.date===selected);
|
const day = current.days.find(item => item.date===selected);
|
||||||
if(!day) return;
|
if(!day) return;
|
||||||
$('selected-date').textContent = format(day.date+'T12:00:00Z',{weekday:'long',day:'numeric',month:'long',year:'numeric'});
|
$('selected-date').textContent = format(day.date+'T12:00:00Z',{weekday:'long',day:'numeric',month:'long',year:'numeric'});
|
||||||
$('selected-summary').textContent = `${number(day.complete)} complete of ${number(day.elapsed)} elapsed intervals · ${number(day.expected)} intervals in this calendar day. Each cell is five minutes.`;
|
$('selected-summary').textContent = `${number(day.complete)} complete of ${number(day.elapsed)} assessed intervals · ${number(day.expected)} intervals in this calendar day. Each cell is five minutes.`;
|
||||||
const start = Date.parse(day.day_start);
|
const start = Date.parse(day.day_start);
|
||||||
|
const currentIndex = day.status==='in_progress' ? Math.floor((Date.parse(current.generated_at)-start)/300000) : -1;
|
||||||
const stamp = index => format(new Date(start+index*300000),{hour:'2-digit',minute:'2-digit',timeZoneName:'shortOffset',hour12:false});
|
const stamp = index => format(new Date(start+index*300000),{hour:'2-digit',minute:'2-digit',timeZoneName:'shortOffset',hour12:false});
|
||||||
$('slots').replaceChildren();
|
$('slots').replaceChildren();
|
||||||
let hour;
|
let hour;
|
||||||
@@ -74,9 +75,9 @@
|
|||||||
addText(hour,'span',format(new Date(start+index*300000),{hour:'2-digit',hour12:false}),'hour-label');
|
addText(hour,'span',format(new Date(start+index*300000),{hour:'2-digit',hour12:false}),'hour-label');
|
||||||
}
|
}
|
||||||
const mask = day.slots[index];
|
const mask = day.slots[index];
|
||||||
const state = mask===7?'complete':mask===-1?'future':mask>0?'partial':'missing';
|
const state = mask===3?'complete':mask===-1?'future':mask>0?'partial':'missing';
|
||||||
const missing = ['Energy','Meter','Boiler'].filter((_,bit) => !(mask & (1<<bit)));
|
const missing = ['Energy','Boiler'].filter((_,bit) => !(mask & (1<<bit)));
|
||||||
const text = `${stamp(index)}–${stamp(index+1)} · ${mask===7?'Complete':mask===-1?'Upcoming':'Incomplete: '+missing.join(', ')}`;
|
const text = `${stamp(index)}–${stamp(index+1)} · ${mask===3?'Complete':mask===-1?(index===currentIndex?'Awaiting extraction':'Upcoming'):'Incomplete: '+missing.join(', ')}`;
|
||||||
const cell = addText(hour,'span','','slot '+state);
|
const cell = addText(hour,'span','','slot '+state);
|
||||||
cell.title = text;
|
cell.title = text;
|
||||||
cell.setAttribute('role','img');
|
cell.setAttribute('role','img');
|
||||||
@@ -84,12 +85,12 @@
|
|||||||
}
|
}
|
||||||
const ranges = [];
|
const ranges = [];
|
||||||
for(let index=0;index<day.elapsed;) {
|
for(let index=0;index<day.elapsed;) {
|
||||||
if(day.slots[index]===7) {index++;continue;}
|
if(day.slots[index]===3) {index++;continue;}
|
||||||
const from = index;
|
const from = index;
|
||||||
while(index<day.elapsed && day.slots[index]!==7) index++;
|
while(index<day.elapsed && day.slots[index]!==3) index++;
|
||||||
ranges.push(`${stamp(from)}–${stamp(index)}`);
|
ranges.push(`${stamp(from)}–${stamp(index)}`);
|
||||||
}
|
}
|
||||||
$('missing-ranges').textContent = ranges.length ? `${number(day.missing)} incomplete intervals. Gaps: ${ranges.slice(0,6).join('; ')}${ranges.length>6?`; plus ${number(ranges.length-6)} more ranges.`:'.'}` : day.status==='in_progress'?'All elapsed intervals are complete. The current interval is still being recorded.':'All five-minute intervals have complete data.';
|
$('missing-ranges').textContent = ranges.length ? `${number(day.missing)} incomplete intervals. Gaps: ${ranges.slice(0,6).join('; ')}${ranges.length>6?`; plus ${number(ranges.length-6)} more ranges.`:'.'}` : day.status==='in_progress'?(day.slots[currentIndex]===-1?'All previous intervals are complete. Waiting for the current extraction.':'All intervals checked so far have complete data.'):'All five-minute intervals have complete data.';
|
||||||
}
|
}
|
||||||
function render(data) {
|
function render(data) {
|
||||||
current = data;
|
current = data;
|
||||||
@@ -112,7 +113,7 @@
|
|||||||
$('upload-detail').textContent = !data.archive.upload_enabled?'Uploads not configured':data.archive.pending?'Saved locally, awaiting delivery':'All observations acknowledged';
|
$('upload-detail').textContent = !data.archive.upload_enabled?'Uploads not configured':data.archive.pending?'Saved locally, awaiting delivery':'All observations acknowledged';
|
||||||
$('last-upload').textContent = 'Last acknowledgement: '+(data.archive.last_upload?date(data.archive.last_upload)+' '+time(data.archive.last_upload):'none yet');
|
$('last-upload').textContent = 'Last acknowledgement: '+(data.archive.last_upload?date(data.archive.last_upload)+' '+time(data.archive.last_upload):'none yet');
|
||||||
$('samples').textContent = number(data.archive.samples);
|
$('samples').textContent = number(data.archive.samples);
|
||||||
$('retention').textContent = `Acknowledged readings kept for ${number(data.archive.retention_days)} days · pending readings retained`;
|
$('retention').textContent = `Retention: ${number(data.archive.retention_days)} full calendar days · pending days retained`;
|
||||||
$('version').textContent = 'solar-relay '+data.version;
|
$('version').textContent = 'solar-relay '+data.version;
|
||||||
if(!selected || !data.days.some(day=>day.date===selected)) selected=(data.days.find(day=>day.samples>0)||yesterday||data.days[0]).date;
|
if(!selected || !data.days.some(day=>day.date===selected)) selected=(data.days.find(day=>day.samples>0)||yesterday||data.days[0]).date;
|
||||||
renderDays();
|
renderDays();
|
||||||
|
|||||||
+29
-5
@@ -68,13 +68,13 @@ class HistoryTests(unittest.TestCase):
|
|||||||
self.assertEqual(self.recover(), 0)
|
self.assertEqual(self.recover(), 0)
|
||||||
self.assertEqual(len(self.client.requests), 2)
|
self.assertEqual(len(self.client.requests), 2)
|
||||||
|
|
||||||
def test_cloud_energy_does_not_mark_boiler_or_meter_history_complete(self):
|
def test_cloud_energy_does_not_mark_boiler_history_complete(self):
|
||||||
self.assertEqual(self.recover(), 3)
|
self.assertEqual(self.recover(), 3)
|
||||||
rows = self.store.coverage(self.source, "UTC", 1, self.when + timedelta(days=1))
|
rows = self.store.coverage(self.source, "UTC", 1, self.when + timedelta(days=1))
|
||||||
day = rows[1]
|
day = rows[1]
|
||||||
self.assertEqual(day["energy_complete"], 3)
|
self.assertEqual(day["energy_complete"], 3)
|
||||||
self.assertEqual(day["boiler_complete"], 0)
|
self.assertEqual(day["boiler_complete"], 0)
|
||||||
self.assertEqual(day["meter_complete"], 0)
|
self.assertNotIn("meter_complete", day)
|
||||||
self.assertEqual(day["complete"], 0)
|
self.assertEqual(day["complete"], 0)
|
||||||
row = pq.ParquetFile(next(self.store.path.glob("date=*/samples.parquet"))).read().to_pylist()[0]
|
row = pq.ParquetFile(next(self.store.path.glob("date=*/samples.parquet"))).read().to_pylist()[0]
|
||||||
self.assertEqual(row["record_kind"], "history")
|
self.assertEqual(row["record_kind"], "history")
|
||||||
@@ -148,7 +148,7 @@ class HistoryTests(unittest.TestCase):
|
|||||||
|
|
||||||
def test_automatic_recovery_checks_retention_before_first_observation_and_respects_cutoff(self):
|
def test_automatic_recovery_checks_retention_before_first_observation_and_respects_cutoff(self):
|
||||||
now = datetime(2026, 11, 2, 12, 2, 17, tzinfo=timezone.utc)
|
now = datetime(2026, 11, 2, 12, 2, 17, tzinfo=timezone.utc)
|
||||||
cutoff = datetime(2026, 10, 3, 12, 5, tzinfo=timezone.utc)
|
cutoff = datetime(2026, 10, 3, tzinfo=ZoneInfo("Europe/Zurich")).astimezone(timezone.utc)
|
||||||
latest = datetime(2026, 11, 2, 10, tzinfo=timezone.utc)
|
latest = datetime(2026, 11, 2, 10, tzinfo=timezone.utc)
|
||||||
recoverable = [cutoff, now.replace(minute=0, second=0) - timedelta(days=25), latest - timedelta(minutes=5)]
|
recoverable = [cutoff, now.replace(minute=0, second=0) - timedelta(days=25), latest - timedelta(minutes=5)]
|
||||||
points = {int(when.timestamp() * 1000): 1200 for when in
|
points = {int(when.timestamp() * 1000): 1200 for when in
|
||||||
@@ -158,7 +158,31 @@ class HistoryTests(unittest.TestCase):
|
|||||||
self.assertEqual(self.recovery.recover(now=now), 3)
|
self.assertEqual(self.recovery.recover(now=now), 3)
|
||||||
recovered = [record for record in self.store.records() if "history_interval" in record]
|
recovered = [record for record in self.store.records() if "history_interval" in record]
|
||||||
self.assertEqual([record["observed_at"] for record in recovered], [utc_text(when) for when in recoverable])
|
self.assertEqual([record["observed_at"] for record in recovered], [utc_text(when) for when in recoverable])
|
||||||
self.assertEqual(min(day for _, _, day in self.client.requests), cutoff.date())
|
self.assertEqual(min(day for _, _, day in self.client.requests), cutoff.astimezone(self.recovery.zone).date())
|
||||||
|
|
||||||
|
def test_oldest_retained_day_recovers_all_288_intervals_including_midnight(self):
|
||||||
|
zone = ZoneInfo("Europe/Zurich")
|
||||||
|
start = datetime(2026, 10, 3, tzinfo=zone)
|
||||||
|
end = start + timedelta(days=1)
|
||||||
|
points = {stamp: 1200 for stamp in range(int(start.timestamp() * 1000), int(end.timestamp() * 1000), 300_000)}
|
||||||
|
self.client.points = {"production": points, "consumption": points}
|
||||||
|
self.assertEqual(self.recovery.recover(now=datetime(2026, 11, 2, 22, 30, tzinfo=zone)), 288)
|
||||||
|
records = list(self.store.records())
|
||||||
|
self.assertEqual(records[0]["observed_at"], utc_text(start))
|
||||||
|
self.assertEqual(records[-1]["observed_at"], utc_text(end - timedelta(minutes=5)))
|
||||||
|
|
||||||
|
def test_retention_uses_the_configured_timezone_instead_of_utc(self):
|
||||||
|
now = datetime(2026, 11, 2, 1, 30, tzinfo=timezone.utc)
|
||||||
|
recovery = HistoryRecovery(self.config, self.source, self.store, timezone_name="America/New_York", client=self.client)
|
||||||
|
recovery.recover(now=now)
|
||||||
|
self.assertEqual(min(day for _, _, day in self.client.requests).isoformat(), "2026-10-02")
|
||||||
|
|
||||||
|
def test_thirty_calendar_day_manual_range_can_cross_a_25_hour_dst_day(self):
|
||||||
|
zone = ZoneInfo("Europe/Zurich")
|
||||||
|
self.client.points = {"production": {}, "consumption": {}}
|
||||||
|
self.assertEqual(self.recovery.recover(utc_text(datetime(2026, 10, 3, tzinfo=zone)),
|
||||||
|
utc_text(datetime(2026, 11, 2, tzinfo=zone)),
|
||||||
|
now=datetime(2026, 11, 3, tzinfo=zone)), 0)
|
||||||
|
|
||||||
def test_automatic_recovery_checks_entirely_empty_days_and_archives(self):
|
def test_automatic_recovery_checks_entirely_empty_days_and_archives(self):
|
||||||
self.store.retention_days = 2
|
self.store.retention_days = 2
|
||||||
@@ -176,7 +200,7 @@ class HistoryTests(unittest.TestCase):
|
|||||||
|
|
||||||
def test_complete_retention_window_does_not_request_solarweb(self):
|
def test_complete_retention_window_does_not_request_solarweb(self):
|
||||||
now = self.when + timedelta(days=1)
|
now = self.when + timedelta(days=1)
|
||||||
lower = int(self.when.timestamp())
|
lower = int(datetime(2026, 10, 30, tzinfo=ZoneInfo("Europe/Zurich")).timestamp())
|
||||||
upper = int((now - timedelta(seconds=self.config.lag_seconds)).timestamp())
|
upper = int((now - timedelta(seconds=self.config.lag_seconds)).timestamp())
|
||||||
store = Mock(retention_days=1)
|
store = Mock(retention_days=1)
|
||||||
store.energy_slots.return_value = set(range(lower, upper, 300))
|
store.energy_slots.return_value = set(range(lower, upper, 300))
|
||||||
|
|||||||
+65
-2
@@ -5,10 +5,12 @@ from pathlib import Path
|
|||||||
import tempfile
|
import tempfile
|
||||||
import unittest
|
import unittest
|
||||||
from unittest.mock import patch
|
from unittest.mock import patch
|
||||||
|
from zoneinfo import ZoneInfo
|
||||||
|
|
||||||
import pyarrow.parquet as pq
|
import pyarrow.parquet as pq
|
||||||
|
|
||||||
from solar_relay.cli import next_poll_at
|
from solar_relay.cli import next_poll_at
|
||||||
|
from solar_relay.config import retention_cutoff
|
||||||
from solar_relay.storage import Store, StoreError
|
from solar_relay.storage import Store, StoreError
|
||||||
from tests.test_status import observed
|
from tests.test_status import observed
|
||||||
|
|
||||||
@@ -34,8 +36,8 @@ class ParquetTests(unittest.TestCase):
|
|||||||
self.assertEqual(json.loads(row["payload"]), value)
|
self.assertEqual(json.loads(row["payload"]), value)
|
||||||
self.assertEqual(self.store.status()["format"], "parquet")
|
self.assertEqual(self.store.status()["format"], "parquet")
|
||||||
|
|
||||||
def test_retention_uses_ack_time_and_never_deletes_pending(self):
|
def test_retention_uses_ack_calendar_days_and_never_deletes_pending(self):
|
||||||
items = [observed(self.now - timedelta(days=90, minutes=index)) for index in range(4)]
|
items = [observed(self.now - timedelta(days=90 + index)) for index in range(4)]
|
||||||
for item in items:
|
for item in items:
|
||||||
self.store.add(item)
|
self.store.add(item)
|
||||||
self.store.acknowledge([items[0]["id"]], now=self.now - timedelta(days=31))
|
self.store.acknowledge([items[0]["id"]], now=self.now - timedelta(days=31))
|
||||||
@@ -47,6 +49,67 @@ class ParquetTests(unittest.TestCase):
|
|||||||
self.assertEqual(self.store.prune(now=self.now + timedelta(days=31)), 2)
|
self.assertEqual(self.store.prune(now=self.now + timedelta(days=31)), 2)
|
||||||
self.assertEqual(list(self.store.records()), [items[3]])
|
self.assertEqual(list(self.store.records()), [items[3]])
|
||||||
|
|
||||||
|
def test_retention_boundary_is_local_midnight_regardless_of_execution_time(self):
|
||||||
|
zone = ZoneInfo("Europe/Zurich")
|
||||||
|
for hour in (0, 12, 23):
|
||||||
|
now = datetime(2026, 11, 2, hour, 30, tzinfo=zone)
|
||||||
|
self.assertEqual(retention_cutoff(now, 30, zone.key), datetime(2026, 10, 2, 22, tzinfo=timezone.utc))
|
||||||
|
with self.assertRaisesRegex(ValueError, "timezone"):
|
||||||
|
retention_cutoff(datetime(2026, 11, 2), 30, zone.key)
|
||||||
|
|
||||||
|
def test_prune_uses_the_configured_timezone_at_the_date_boundary(self):
|
||||||
|
store = Store(self.path, timezone_name="America/New_York")
|
||||||
|
values = [observed(self.now - timedelta(days=60 + offset)) for offset in range(2)]
|
||||||
|
for value in values:
|
||||||
|
store.add(value)
|
||||||
|
store.acknowledge([values[0]["id"]], now=datetime(2026, 10, 2, 3, 59, tzinfo=timezone.utc))
|
||||||
|
store.acknowledge([values[1]["id"]], now=datetime(2026, 10, 2, 4, tzinfo=timezone.utc))
|
||||||
|
self.assertEqual(store.prune(now=datetime(2026, 11, 2, 1, 30, tzinfo=timezone.utc)), 1)
|
||||||
|
self.assertEqual(list(store.records()), [values[1]])
|
||||||
|
|
||||||
|
def test_prune_keeps_the_entire_cutoff_acknowledgement_day(self):
|
||||||
|
zone = ZoneInfo("Europe/Zurich")
|
||||||
|
start = datetime(2026, 10, 3, tzinfo=zone)
|
||||||
|
values = [observed(start - timedelta(days=10) + timedelta(minutes=offset)) for offset in (0, 5)]
|
||||||
|
for value, ack in zip(values, (start, start + timedelta(hours=23, minutes=55))):
|
||||||
|
self.store.add(value)
|
||||||
|
self.store.acknowledge([value["id"]], now=ack)
|
||||||
|
self.assertEqual(self.store.prune(now=datetime(2026, 11, 2, 23, 59, tzinfo=zone)), 0)
|
||||||
|
self.assertEqual(self.store.prune(now=datetime(2026, 11, 3, tzinfo=zone)), 2)
|
||||||
|
|
||||||
|
def test_prune_removes_whole_local_days_across_utc_partitions(self):
|
||||||
|
zone = ZoneInfo("Europe/Zurich")
|
||||||
|
first = datetime(2026, 10, 3, tzinfo=zone)
|
||||||
|
expired = [observed(first - timedelta(hours=23, minutes=55)), observed(first - timedelta(minutes=5))]
|
||||||
|
retained = [observed(first), observed(first + timedelta(hours=23, minutes=55))]
|
||||||
|
for value in expired + retained:
|
||||||
|
self.store.add(value)
|
||||||
|
self.store.acknowledge([value["id"] for value in expired], now=first - timedelta(minutes=1))
|
||||||
|
self.store.acknowledge([value["id"] for value in retained], now=first + timedelta(days=1))
|
||||||
|
self.assertEqual(self.store.prune(now=datetime(2026, 11, 2, 22, tzinfo=zone)), 2)
|
||||||
|
self.assertEqual(list(self.store.records()), retained)
|
||||||
|
|
||||||
|
def test_one_pending_record_protects_its_entire_local_day_across_partitions(self):
|
||||||
|
zone = ZoneInfo("Europe/Zurich")
|
||||||
|
day = datetime(2026, 10, 2, tzinfo=zone)
|
||||||
|
values = [observed(day + timedelta(minutes=5)), observed(day + timedelta(hours=23, minutes=55))]
|
||||||
|
for value in values:
|
||||||
|
self.store.add(value)
|
||||||
|
self.store.acknowledge([values[0]["id"]], now=day + timedelta(hours=23, minutes=59))
|
||||||
|
self.assertEqual(self.store.prune(now=datetime(2026, 11, 2, 22, tzinfo=zone)), 0)
|
||||||
|
self.assertEqual(list(self.store.records()), values)
|
||||||
|
|
||||||
|
def test_daylight_saving_retention_boundaries_preserve_complete_local_days(self):
|
||||||
|
zone = ZoneInfo("Europe/Zurich")
|
||||||
|
for day, expected_hours in [(datetime(2026, 3, 29, tzinfo=zone), 23),
|
||||||
|
(datetime(2026, 10, 25, tzinfo=zone), 25)]:
|
||||||
|
with self.subTest(day=day):
|
||||||
|
now = day + timedelta(days=30, hours=12)
|
||||||
|
cutoff = retention_cutoff(now, 30, zone.key)
|
||||||
|
self.assertEqual(cutoff, day.astimezone(timezone.utc))
|
||||||
|
next_day = (day + timedelta(days=1)).astimezone(timezone.utc)
|
||||||
|
self.assertEqual((next_day - cutoff).total_seconds(), expected_hours * 3600)
|
||||||
|
|
||||||
def test_replay_cancels_expiry_until_the_next_acknowledgement(self):
|
def test_replay_cancels_expiry_until_the_next_acknowledgement(self):
|
||||||
value = observed(self.now)
|
value = observed(self.now)
|
||||||
self.store.add(value)
|
self.store.add(value)
|
||||||
|
|||||||
+88
-5
@@ -42,8 +42,56 @@ class CoverageTests(unittest.TestCase):
|
|||||||
self.assertEqual(yesterday["expected"], 288)
|
self.assertEqual(yesterday["expected"], 288)
|
||||||
self.assertEqual(yesterday["complete"], 288)
|
self.assertEqual(yesterday["complete"], 288)
|
||||||
self.assertEqual(yesterday["status"], "complete")
|
self.assertEqual(yesterday["status"], "complete")
|
||||||
self.assertEqual(result[0]["elapsed"], 0)
|
self.assertEqual(result[0]["elapsed"], 1)
|
||||||
self.assertTrue(all(mask == -1 for mask in result[0]["slots"]))
|
self.assertEqual(result[0]["slots"][0], 3)
|
||||||
|
self.assertTrue(all(mask == -1 for mask in result[0]["slots"][1:]))
|
||||||
|
|
||||||
|
def test_successful_2305_extraction_is_green_at_2308(self):
|
||||||
|
zone = ZoneInfo("Europe/Zurich")
|
||||||
|
now = datetime(2026, 11, 2, 23, 8, tzinfo=zone)
|
||||||
|
stamp = round(now.replace(minute=5).timestamp() * 1000)
|
||||||
|
day = daily_coverage([(stamp, stamp, 1, stamp, 1, stamp, 1)], zone.key, 1, now)[0]
|
||||||
|
self.assertEqual(day["slots"][277], 3)
|
||||||
|
self.assertEqual(day["slots"][278], -1)
|
||||||
|
self.assertEqual(day["complete"], 1)
|
||||||
|
self.assertEqual(day["energy_complete"], 1)
|
||||||
|
self.assertEqual(day["boiler_complete"], 1)
|
||||||
|
self.assertEqual(day["elapsed"], 278)
|
||||||
|
self.assertEqual(day["missing"], 277)
|
||||||
|
|
||||||
|
def test_current_slot_without_extraction_stays_pending_until_it_ends(self):
|
||||||
|
zone = ZoneInfo("Europe/Zurich")
|
||||||
|
now = datetime(2026, 11, 2, 23, 8, tzinfo=zone)
|
||||||
|
day = daily_coverage([], zone.key, 1, now)[0]
|
||||||
|
self.assertEqual(day["slots"][277], -1)
|
||||||
|
self.assertEqual(day["elapsed"], 277)
|
||||||
|
self.assertEqual(day["missing"], 277)
|
||||||
|
after = daily_coverage([], zone.key, 1, now.replace(minute=10))[0]
|
||||||
|
self.assertEqual(after["slots"][277], 0)
|
||||||
|
self.assertEqual(after["slots"][278], -1)
|
||||||
|
self.assertEqual(after["missing"], 278)
|
||||||
|
|
||||||
|
def test_partial_and_failed_current_extractions_are_shown_immediately(self):
|
||||||
|
zone = ZoneInfo("Europe/Zurich")
|
||||||
|
now = datetime(2026, 11, 2, 23, 8, tzinfo=zone)
|
||||||
|
stamp = round(now.replace(minute=5).timestamp() * 1000)
|
||||||
|
for energy, boiler, expected_mask in [(1, 0, 1), (0, 1, 2), (0, 0, 0)]:
|
||||||
|
with self.subTest(energy=energy, boiler=boiler):
|
||||||
|
day = daily_coverage([(stamp, stamp, energy, stamp, 0, stamp, boiler)], zone.key, 1, now)[0]
|
||||||
|
self.assertEqual(day["slots"][277], expected_mask)
|
||||||
|
self.assertEqual(day["elapsed"], 278)
|
||||||
|
self.assertEqual(day["complete"], 0)
|
||||||
|
|
||||||
|
def test_current_extraction_at_midnight_is_counted_without_waiting(self):
|
||||||
|
zone = ZoneInfo("Europe/Zurich")
|
||||||
|
now = datetime(2026, 11, 2, 0, 0, 10, tzinfo=zone)
|
||||||
|
stamp = round(now.replace(second=0).timestamp() * 1000)
|
||||||
|
day = daily_coverage([(stamp, stamp, 1, stamp, 0, stamp, 1)], zone.key, 1, now)[0]
|
||||||
|
self.assertEqual(day["slots"][0], 3)
|
||||||
|
self.assertEqual(day["elapsed"], 1)
|
||||||
|
self.assertEqual(day["complete"], 1)
|
||||||
|
self.assertEqual(day["missing"], 0)
|
||||||
|
self.assertTrue(all(mask == -1 for mask in day["slots"][1:]))
|
||||||
|
|
||||||
def test_dst_days_expect_276_and_300_intervals(self):
|
def test_dst_days_expect_276_and_300_intervals(self):
|
||||||
for date, expected in [("2026-03-29", 276), ("2026-10-25", 300)]:
|
for date, expected in [("2026-03-29", 276), ("2026-10-25", 300)]:
|
||||||
@@ -61,17 +109,27 @@ class CoverageTests(unittest.TestCase):
|
|||||||
self.assertEqual(yesterday["missing"], 3)
|
self.assertEqual(yesterday["missing"], 3)
|
||||||
self.assertEqual(yesterday["status"], "partial")
|
self.assertEqual(yesterday["status"], "partial")
|
||||||
|
|
||||||
def test_energy_failure_does_not_erase_meter_or_boiler_coverage(self):
|
def test_energy_failure_does_not_erase_boiler_coverage(self):
|
||||||
rows, end = rows_for_day("2026-10-30")
|
rows, end = rows_for_day("2026-10-30")
|
||||||
stamp = rows[1][0]
|
stamp = rows[1][0]
|
||||||
rows[1] = (stamp, stamp, 0, stamp, 1, stamp, 1)
|
rows[1] = (stamp, stamp, 0, stamp, 1, stamp, 1)
|
||||||
yesterday = daily_coverage(rows, "Europe/Zurich", 1, end + timedelta(minutes=3))[1]
|
yesterday = daily_coverage(rows, "Europe/Zurich", 1, end + timedelta(minutes=3))[1]
|
||||||
self.assertEqual(yesterday["slots"][1], 6)
|
self.assertEqual(yesterday["slots"][1], 2)
|
||||||
self.assertEqual(yesterday["energy_complete"], 287)
|
self.assertEqual(yesterday["energy_complete"], 287)
|
||||||
self.assertEqual(yesterday["meter_complete"], 288)
|
self.assertNotIn("meter_complete", yesterday)
|
||||||
self.assertEqual(yesterday["boiler_complete"], 288)
|
self.assertEqual(yesterday["boiler_complete"], 288)
|
||||||
self.assertEqual(yesterday["missing"], 1)
|
self.assertEqual(yesterday["missing"], 1)
|
||||||
|
|
||||||
|
def test_missing_meter_readings_do_not_affect_visible_completeness(self):
|
||||||
|
rows, end = rows_for_day("2026-10-30")
|
||||||
|
rows = [(observed, power_ms, power_ok, meter_ms, 0, boiler_ms, boiler_ok)
|
||||||
|
for observed, power_ms, power_ok, meter_ms, meter_ok, boiler_ms, boiler_ok in rows]
|
||||||
|
yesterday = daily_coverage(rows, "Europe/Zurich", 1, end + timedelta(minutes=3))[1]
|
||||||
|
self.assertEqual(yesterday["complete"], 288)
|
||||||
|
self.assertEqual(yesterday["status"], "complete")
|
||||||
|
self.assertEqual(set(yesterday["slots"]), {3})
|
||||||
|
self.assertNotIn("meter_complete", yesterday)
|
||||||
|
|
||||||
def test_source_errors_stale_readings_and_false_boiler_state(self):
|
def test_source_errors_stale_readings_and_false_boiler_state(self):
|
||||||
when = datetime(2026, 10, 30, tzinfo=timezone.utc)
|
when = datetime(2026, 10, 30, tzinfo=timezone.utc)
|
||||||
value = observed(when)
|
value = observed(when)
|
||||||
@@ -150,6 +208,31 @@ class StatusHttpTests(unittest.TestCase):
|
|||||||
self.assertEqual(result["collector"]["source_errors"], ["boiler"])
|
self.assertEqual(result["collector"]["source_errors"], ["boiler"])
|
||||||
self.assertNotIn("secret canary", json.dumps(result))
|
self.assertNotIn("secret canary", json.dumps(result))
|
||||||
|
|
||||||
|
def test_meter_failure_does_not_change_current_status_or_last_success(self):
|
||||||
|
value = observed(datetime.now(timezone.utc))
|
||||||
|
value["errors"] = {"meter": "meter is unavailable"}
|
||||||
|
value["readings"]["grid_import_total_wh"] = None
|
||||||
|
value["readings"]["grid_export_total_wh"] = None
|
||||||
|
self.store.add(value)
|
||||||
|
result = json.loads(self.get("/api/status")[2])
|
||||||
|
self.assertEqual(result["status"], "ok")
|
||||||
|
self.assertEqual(result["collector"]["source_errors"], [])
|
||||||
|
self.assertEqual(result["collector"]["last_success"], value["observed_at"])
|
||||||
|
self.assertNotIn("meter_complete", result["days"][0])
|
||||||
|
|
||||||
|
def test_latest_success_and_current_grid_cell_agree(self):
|
||||||
|
zone = ZoneInfo("Europe/Zurich")
|
||||||
|
now = datetime(2026, 11, 2, 23, 8, tzinfo=zone)
|
||||||
|
value = observed(now.replace(minute=5))
|
||||||
|
self.store.add(value)
|
||||||
|
with unittest.mock.patch("solar_relay.status.datetime", wraps=datetime) as dates:
|
||||||
|
dates.now.return_value = now.astimezone(timezone.utc)
|
||||||
|
result = json.loads(self.get("/api/status?days=1")[2])
|
||||||
|
self.assertEqual(result["status"], "ok")
|
||||||
|
self.assertEqual(result["collector"]["last_success"], value["observed_at"])
|
||||||
|
self.assertEqual(result["days"][0]["slots"][277], 3)
|
||||||
|
self.assertEqual(result["days"][0]["slots"][278], -1)
|
||||||
|
|
||||||
def test_only_explicit_read_only_routes_are_available(self):
|
def test_only_explicit_read_only_routes_are_available(self):
|
||||||
for path in ["/config.toml", "/.env", "/data/archive/date=2026-11-01/samples.parquet", "/../config.toml", "/solar-relay/.git/config"]:
|
for path in ["/config.toml", "/.env", "/data/archive/date=2026-11-01/samples.parquet", "/../config.toml", "/solar-relay/.git/config"]:
|
||||||
self.assertEqual(self.get(path)[0], 404, path)
|
self.assertEqual(self.get(path)[0], 404, path)
|
||||||
|
|||||||
Reference in new issue
Block a user