Let passive readers reuse the control loop's meter reading

CHANGELOG.md

@@ -2,6 +2,8 @@
 
 ## Next
 
+- **Fixed** the Marstek app and cloud reporting sending extra requests to a polled power meter, such as a Shelly or an HTTP source, on top of the batteries' own polls; they now reuse the batteries' latest reading ([#698](https://github.com/tomquist/astrameter/pull/698)).
+
 - **Added** a Marstek CT002 / CT003 power source (`[MARSTEK_METER]`), so AstraMeter can read the Marstek CT you already own and steer your batteries itself. Pair the batteries with AstraMeter's CT instead of the real one ([#737](https://github.com/tomquist/astrameter/pull/737)).
 
 - **Fixed** the dashboard reporting "No batteries polling" with the default Shelly Pro 3EM emulation while a battery was polling it on port 1010; both ports now show as one emulator listing every battery ([#734](https://github.com/tomquist/astrameter/issues/734), [#736](https://github.com/tomquist/astrameter/pull/736)).

CONTRIBUTING.md

@@ -73,7 +73,7 @@ When you fix a bug in:
 
 `split_balancer_knobs` in `balancer.py` is Python-only too — it sorts the flat tuning knobs a scenario file or a `--set` override names into a `BalancerConfig`, and the firmware takes its configuration from codegen instead.
 
-Fixes to `src/astrameter/powermeter/wrappers/{transform,throttling}.py` have **no** C++ counterpart — those wrappers are delegated to ESPHome's standard `sensor: filters:` (`offset:`, `multiply:`, `throttle:`) on the upstream sensor. `src/astrameter/powermeter/wrappers/health.py` (the outermost `HealthTrackingPowermeter` feeding the MQTT Insights Online sensor) likewise has **no** C++ counterpart — it tracks Python powermeter reads, which the firmware doesn't have.
+Fixes to `src/astrameter/powermeter/wrappers/{transform,throttling}.py` have **no** C++ counterpart — those wrappers are delegated to ESPHome's standard `sensor: filters:` (`offset:`, `multiply:`, `throttle:`) on the upstream sensor. `src/astrameter/powermeter/wrappers/health.py` (the outermost `HealthTrackingPowermeter` feeding the MQTT Insights Online sensor) likewise has **no** C++ counterpart — it tracks Python powermeter reads, which the firmware doesn't have. Nor does `src/astrameter/powermeter/wrappers/last_reading.py` (the innermost `LastReadingPowermeter`, which lets the Marstek responder and cloud reporting reuse the control loop's read of a polled meter): the firmware's passive readers already read its sensor cache (`CT002Component::latest_grid_power()`).
 
 The `power_sensor_lX` unit handling (unit→W auto-conversion, non-power-unit rejection, and the kW-suspicion warning — issue #572) is **ESPHome-only**: it lives in `ct002/__init__.py` (`FINAL_VALIDATE_SCHEMA` + `set_power_unit_scale` codegen) and the sensor-callback cache in `ct002.cpp`, with no counterpart in `src/astrameter/ct002/` — the Python stack receives watts from its powermeter layer, where the equivalent unit handling is implemented per-source (currently `powermeter/homeassistant.py`, which reads the entity's `unit_of_measurement` attribute). Keep the two accepted-unit tables (`POWER_UNIT_SCALES` in `ct002/__init__.py`, `POWER_UNIT_SCALE` in `src/astrameter/power_units.py`) in sync — the Python side keeps its copy in that leaf module because the dashboard's sensor picker reads it too, and drift between the two is what hid working sensors from the form.
 

esphome/components/ct002/sensor_backed.h

@@ -24,7 +24,6 @@ class SensorBackedPowermeter : public Powermeter {
                         uint32_t max_sensor_age_ms);
 
   std::vector<float> get_powermeter_watts() override;
-  std::vector<float> get_powermeter_watts_raw() override { return this->get_powermeter_watts(); }
 
  private:
   uint8_t num_phases_;

esphome/components/ct002/wrapper_base.h

@@ -17,13 +17,6 @@ class Powermeter {
   // 3-element (three-phase) vectors; an empty vector signals "unavailable".
   virtual std::vector<float> get_powermeter_watts() = 0;
 
-  // Raw (unfiltered) per-phase readings — used by mqtt_insights to publish
-  // the unsmoothed signal alongside the filtered one. The base wrapper
-  // delegates to upstream's raw, so the call chain unwinds to the source.
-  virtual std::vector<float> get_powermeter_watts_raw() {
-    return this->get_powermeter_watts();
-  }
-
   // Clears any time-windowed state. Called by CT002 after long gaps where
   // accumulated state would be stale (e.g. Wi-Fi reconnect, sensor outage).
   virtual void reset() {}
@@ -36,9 +29,6 @@ class PowermeterWrapper : public Powermeter {
  public:
   explicit PowermeterWrapper(Powermeter *wrapped) : wrapped_(wrapped) {}
 
-  std::vector<float> get_powermeter_watts_raw() override {
-    return this->wrapped_->get_powermeter_watts_raw();
-  }
   void reset() override { this->wrapped_->reset(); }
 
  protected:

src/astrameter/config/addon_test.py

@@ -12,6 +12,7 @@
 from astrameter.powermeter import HomeAssistant, ThrottledPowermeter
 from astrameter.powermeter.wrappers.base import PowermeterWrapper
 from astrameter.powermeter.wrappers.health import HealthTrackingPowermeter
+from astrameter.powermeter.wrappers.last_reading import LastReadingPowermeter
 
 
 class FakeSupervisor(addon.SupervisorClient):
@@ -174,7 +175,9 @@ def test_single_power_entity_is_read_directly() -> None:
     cfg = config(BASE_OPTIONS)
     meter = cfg.powermeters(cfg.general())[0][0]
     assert isinstance(meter, HealthTrackingPowermeter)
-    source = meter.wrapped_powermeter
+    shared = meter.wrapped_powermeter
+    assert isinstance(shared, LastReadingPowermeter)
+    source = shared.wrapped_powermeter
     assert isinstance(source, HomeAssistant)
     assert source.power_calculate is False
     assert source.current_power_entity == ["sensor.current_power_in"]
@@ -183,10 +186,12 @@ def test_single_power_entity_is_read_directly() -> None:
 
 
 def _inner_source(cfg: addon.AddonAppConfig) -> HomeAssistant:
-    """The Home Assistant source under the health wrapper the loader adds."""
+    """The Home Assistant source under the wrappers the loader always adds."""
     meter = cfg.powermeters(cfg.general())[0][0]
     assert isinstance(meter, HealthTrackingPowermeter)
-    source = meter.wrapped_powermeter
+    shared = meter.wrapped_powermeter
+    assert isinstance(shared, LastReadingPowermeter)
+    source = shared.wrapped_powermeter
     assert isinstance(source, HomeAssistant)
     return source
 
@@ -244,6 +249,7 @@ def test_power_source_is_conditioned_by_the_options() -> None:
         "HampelPowermeter",
         "ThrottledPowermeter",
         "TransformedPowermeter",
+        "LastReadingPowermeter",
     ]
 
 

src/astrameter/config/config_loader.py

@@ -21,6 +21,10 @@
 from astrameter.powermeter.base import Powermeter
 from astrameter.powermeter.wrappers.hampel import HampelPowermeter
 from astrameter.powermeter.wrappers.health import HealthTrackingPowermeter
+from astrameter.powermeter.wrappers.last_reading import (
+    PASSIVE_READING_MAX_AGE_S,
+    LastReadingPowermeter,
+)
 from astrameter.powermeter.wrappers.pid import PidPowermeter
 from astrameter.powermeter.wrappers.smoothing import (
     DeadbandPowermeter,
@@ -356,6 +360,16 @@ def apply_signal_wrappers(
     Shared by every config backend, so a power source behaves the same however
     it was configured. *name* labels it in logs and MQTT Insights.
     """
+    # Innermost, so passive readers see the source's own reading. The throttle
+    # above it lets a control read through at most once per interval, so the
+    # reading may be that much older before a passive reader reads for itself.
+    # A negative interval means no throttle, as below, not a cache that never
+    # expires.
+    powermeter = LastReadingPowermeter(
+        powermeter,
+        max_age=PASSIVE_READING_MAX_AGE_S + max(0.0, signal.throttle_interval),
+    )
+
     if signal.offsets is not None or signal.multipliers is not None:
         offsets = signal.offsets if signal.offsets is not None else [0.0]
         multipliers = signal.multipliers if signal.multipliers is not None else [1.0]

src/astrameter/config/config_loader_test.py

@@ -911,3 +911,28 @@ def test_read_all_configs_wait_for_next_message_section_override() -> None:
     # Section order is preserved by configparser, so SCRIPT_1 (override=false)
     # is first and SCRIPT_2 (inherits global true) is second.
     assert [wait for _, _, wait in powermeters] == [False, True]
+
+
+@pytest.mark.parametrize(("throttle", "max_age"), [(0.0, 5.0), (3.0, 8.0), (-5.0, 5.0)])
+def test_shared_reading_ages_out_with_the_throttle(
+    throttle: float, max_age: float
+) -> None:
+    """Passive readers may reuse a reading for as long as the throttle holds
+    control reads back; a negative interval means no throttle, not a reading
+    reused for ever."""
+    from dataclasses import replace
+
+    from astrameter.config.config_loader import apply_signal_wrappers
+    from astrameter.config.settings import SignalSettings
+    from astrameter.powermeter.script import Script
+    from astrameter.powermeter.wrappers.base import PowermeterWrapper
+    from astrameter.powermeter.wrappers.last_reading import LastReadingPowermeter
+
+    signal = replace(SignalSettings(), throttle_interval=throttle)
+    node = apply_signal_wrappers(Script('echo "100"'), "SCRIPT_1", signal)
+    while isinstance(node, PowermeterWrapper) and not isinstance(
+        node, LastReadingPowermeter
+    ):
+        node = node.wrapped_powermeter
+    assert isinstance(node, LastReadingPowermeter)
+    assert node.max_age == max_age

src/astrameter/main.py

@@ -65,10 +65,16 @@ async def read_ct_powermeter(
     return three_phases(await read_fresh(configured))
 
 
-async def _read_grid_phases(powermeters: list[ConfiguredPowermeter]) -> list[float]:
-    """Raw three-phase reading from the meter that serves every client (or the
-    first one), waiting briefly for a fresh push so an idle meter cannot pin
-    the caller."""
+async def _passive_grid_phases(powermeters: list[ConfiguredPowermeter]) -> list[float]:
+    """Three-phase reading over the passive route, for callers that only
+    report it (the Marstek responder, cloud reporting).
+
+    Reads the meter that serves every client (or the first one) through
+    :meth:`~astrameter.powermeter.Powermeter.get_passive_watts`, so it reuses
+    the reading the batteries' polls last fetched and reads a polled meter only
+    when none did recently. Waits briefly for a fresh push first, so an idle
+    push meter cannot pin the caller.
+    """
     chosen = next(
         (c.powermeter for c in powermeters if c.client_filter.matches("0.0.0.0")), None
     )
@@ -80,7 +86,7 @@ async def _read_grid_phases(powermeters: list[ConfiguredPowermeter]) -> list[flo
         await asyncio.wait_for(
             chosen.wait_for_next_message(), timeout=FRESH_READING_TIMEOUT_S
         )
-    return [float(v) for v in three_phases(await chosen.get_powermeter_watts_raw())]
+    return [float(v) for v in three_phases(await chosen.get_passive_watts())]
 
 
 async def check_powermeter(powermeter: Powermeter, client_filter: ClientFilter) -> None:
@@ -269,7 +275,7 @@ async def _bind_marstek_responder(
             device_id=device_id,
             ct_type=device.ct_type,
             mac=marstek_mac,
-            get_values=lambda: _read_grid_phases(powermeters),
+            get_values=lambda: _passive_grid_phases(powermeters),
             get_connected_slave_count=device.reporting_consumer_count,
             get_cd4_slave_csv=lambda: format_cd4_slave_csv(
                 device.reporting_consumer_rows()
@@ -342,7 +348,7 @@ def _start_cloud_reporting(
 
     async def gather() -> CtMeasurement:
         return _ct_measurement(
-            device, await _read_grid_phases(powermeters), mqtt_connected
+            device, await _passive_grid_phases(powermeters), mqtt_connected
         )
 
     reporter = CloudReporter(

src/astrameter/main_test.py

@@ -1,10 +1,19 @@
 import argparse
+from dataclasses import replace
 from ipaddress import IPv4Network
 
-from astrameter.config.config_loader import ClientFilter, new_config_parser
+from astrameter.config.config_loader import (
+    ClientFilter,
+    apply_signal_wrappers,
+    new_config_parser,
+)
 from astrameter.config.ini_config import IniAppConfig
-from astrameter.config.settings import ConfiguredPowermeter
-from astrameter.main import _resolve_device_config, read_ct_powermeter
+from astrameter.config.settings import ConfiguredPowermeter, SignalSettings
+from astrameter.main import (
+    _passive_grid_phases,
+    _resolve_device_config,
+    read_ct_powermeter,
+)
 from astrameter.powermeter import Powermeter
 
 
@@ -24,7 +33,7 @@ def __init__(
     async def get_powermeter_watts(self) -> list[float]:
         return list(self._values)
 
-    async def get_powermeter_watts_raw(self) -> list[float]:
+    async def get_passive_watts(self) -> list[float]:
         return list(self._values)
 
     async def wait_for_next_message(self, timeout: float = 5) -> None:
@@ -63,10 +72,10 @@ async def test_read_ct_powermeter_calls_wait_with_2s_when_enabled() -> None:
     assert pm._wait_calls == [2]
 
 
-async def test_stub_powermeter_raw_matches_watts() -> None:
+async def test_stub_powermeter_passive_matches_watts() -> None:
     pm = _StubPowermeter([3.0, 4.0, 5.0])
     assert (
-        await pm.get_powermeter_watts_raw()
+        await pm.get_passive_watts()
         == await pm.get_powermeter_watts()
         == [
             3.0,
@@ -87,6 +96,52 @@ async def test_read_ct_powermeter_swallows_timeout_and_serves_cached() -> None:
     assert result == [11.0, 22.0, 33.0]
 
 
+class _PolledMeter(Powermeter):
+    """A pull source that counts how often it is actually read."""
+
+    def __init__(self, values: list[float]) -> None:
+        self.values = values
+        self.reads = 0
+
+    async def get_powermeter_watts(self) -> list[float]:
+        self.reads += 1
+        return list(self.values)
+
+
+_EVERYONE = ClientFilter([IPv4Network("0.0.0.0/0")])
+
+
+def _configured(source: Powermeter) -> list[ConfiguredPowermeter]:
+    """*source* behind the wrappers the config loader builds, smoothing on."""
+    signal = replace(SignalSettings(), smooth_alpha=0.5)
+    return [
+        ConfiguredPowermeter(
+            apply_signal_wrappers(source, "SCRIPT_1", signal), _EVERYONE, False
+        )
+    ]
+
+
+async def test_grid_phases_reuse_the_control_loops_read() -> None:
+    """The Marstek responder and cloud reporting only report the reading, so
+    while a battery polls the meter they must not read it again."""
+    source = _PolledMeter([100.0, 200.0, 300.0])
+    powermeters = _configured(source)
+
+    await read_ct_powermeter(("10.0.0.5", 0), powermeters)
+    source.values = [0.0, 0.0, 0.0]
+
+    assert await _passive_grid_phases(powermeters) == [100.0, 200.0, 300.0]
+    assert source.reads == 1
+
+
+async def test_grid_phases_read_an_idle_meter_themselves() -> None:
+    source = _PolledMeter([100.0, 200.0, 300.0])
+    powermeters = _configured(source)
+
+    assert await _passive_grid_phases(powermeters) == [100.0, 200.0, 300.0]
+    assert source.reads == 1
+
+
 def _resolve(device_type: str) -> tuple[list[str], list[str]]:
     cfg = new_config_parser()
     cfg.add_section("GENERAL")

src/astrameter/meter_pool.py

@@ -12,7 +12,7 @@
 from astrameter.config.logger import logger
 from astrameter.config.settings import ConfiguredPowermeter
 from astrameter.powermeter import Powermeter
-from astrameter.powermeter.wrappers.health import HealthTrackingPowermeter
+from astrameter.powermeter.wrappers.base import PowermeterWrapper
 
 #: How long a poll waits for a pushed reading before serving the last one.
 #: Batteries poll about once a second, so a longer wait would answer after the
@@ -21,13 +21,10 @@
 
 
 def powermeter_name(powermeter: Powermeter) -> str:
-    """The meter class behind the health wrapper every configured meter gets."""
-    inner = (
-        powermeter.wrapped_powermeter
-        if isinstance(powermeter, HealthTrackingPowermeter)
-        else powermeter
-    )
-    return type(inner).__name__
+    """The source class under the wrappers every configured meter gets."""
+    while isinstance(powermeter, PowermeterWrapper):
+        powermeter = powermeter.wrapped_powermeter
+    return type(powermeter).__name__
 
 
 def powermeter_for(

src/astrameter/powermeter/base.py

@@ -37,13 +37,17 @@ class Powermeter(ABC):
     @abstractmethod
     async def get_powermeter_watts(self) -> list[float]: ...
 
-    async def get_powermeter_watts_raw(self) -> list[float]:
-        """Per-phase watts before section/global processing wrappers.
-
-        Used when a consumer (e.g. Marstek MQTT display) should match the physical
-        meter while control still uses :meth:`get_powermeter_watts`. Defaults to
-        the same values as :meth:`get_powermeter_watts` for sources with no inner
-        pipeline.
+    async def get_passive_watts(self) -> list[float]:
+        """Per-phase watts for a reader that only reports them.
+
+        The passive route, for the Marstek MQTT display and cloud reporting:
+        the source's own reading, before offsets and filters, which never
+        feeds the processing chain and must not add load on the meter. The
+        ``LastReadingPowermeter`` the config loader puts around every source
+        answers it from the control loop's recent read, and reads the source
+        only when nothing else has lately. Control uses
+        :meth:`get_powermeter_watts` instead. Defaults to the same values as
+        :meth:`get_powermeter_watts` for a source with no wrappers.
         """
         return await self.get_powermeter_watts()
 

src/astrameter/powermeter/wrappers/__init__.py

@@ -1,15 +1,18 @@
 from .base import PowermeterWrapper
 from .hampel import HampelPowermeter
 from .health import HealthTrackingPowermeter
+from .last_reading import PASSIVE_READING_MAX_AGE_S, LastReadingPowermeter
 from .pid import PidPowermeter
 from .smoothing import DeadbandPowermeter, SmoothedPowermeter
 from .throttling import ThrottledPowermeter
 from .transform import TransformedPowermeter
 
 __all__ = [
+    "PASSIVE_READING_MAX_AGE_S",
     "DeadbandPowermeter",
     "HampelPowermeter",
     "HealthTrackingPowermeter",
+    "LastReadingPowermeter",
     "PidPowermeter",
     "PowermeterWrapper",
     "SmoothedPowermeter",

src/astrameter/powermeter/wrappers/base.py

@@ -7,8 +7,8 @@ class PowermeterWrapper(Powermeter):
     def __init__(self, wrapped_powermeter: Powermeter) -> None:
         self.wrapped_powermeter = wrapped_powermeter
 
-    async def get_powermeter_watts_raw(self) -> list[float]:
-        return await self.wrapped_powermeter.get_powermeter_watts_raw()
+    async def get_passive_watts(self) -> list[float]:
+        return await self.wrapped_powermeter.get_passive_watts()
 
     def stream_online(self) -> bool | None:
         return self.wrapped_powermeter.stream_online()

src/astrameter/powermeter/wrappers/conftest.py

@@ -25,7 +25,7 @@ def set(self, values: list[float]) -> None:
     async def get_powermeter_watts(self) -> list[float]:
         return list(self._values)
 
-    async def get_powermeter_watts_raw(self) -> list[float]:
+    async def get_passive_watts(self) -> list[float]:
         return list(self._values)
 
     async def wait_for_message(self, timeout: float = 5) -> None: