Skip to content

fix: degradation watch regresses on the whole lookback, not the newest 40 sessions - #43

Merged
zebraengine merged 3 commits into
mainfrom
fix/drift-full-history
Oct 19, 2026
Merged

zebraengine merged 3 commits into
mainfrom
fix/drift-full-history

Conversation

@zebraengine

@zebraengine zebraengine commented Oct 19, 2026 •

Copy link
Copy Markdown
Owner

Problem

The heat-rise degradation alert ("Handle heat rise increasing") was raised on 10-14 with a "statistically confirmed" +8.1 °C over 34 days (95% CI 2.8..13.3, n=29). The connector isn't degrading. fit_sessions kept only the newest 40 sessions ([:40], there since the first thermal commit), so the drift regression's baseline was whatever happened ~40 charges ago. On this install that edge fell on one week of unusually low fits (Sep 14–20: median rise 28.9 °C against 35–37 °C before and after). The "climb" was the recovery from that week.

window fits Δ over span 95% CI
deployed (newest 40 sessions) 34 days 29 +8.1 °C 2.8..13.3
whole history since the verified anchor 80 days 45 −0.04 °C −5.9..5.8

Garage ambient isn't the explanation: the regression's ambient coefficient is +0.12 ± 0.16 °C/°C, and the garage was warmer in mid-September than during the recent warm days.

Code touched

  • wallmonitor/thermal.py
    • fit_sessions fits every session in the lookback (default 120 days) and tags fits from the newest FORECAST_SESSIONS (40) with forecast: true. Older sessions are processed after the newest 40, so the newest 40's fits come out exactly as before. The tau_prior used for window sizing grows in the same order.
    • New forecast_fits(fits) returns the tagged subset.
    • fit_history takes tau, rise and the current exponent from forecast_fits only, and re-normalizes every fit's rise_ref_c with that exponent, so the drift check's older fits share the forecast's current law.
  • wallmonitor/web.py: /api/thermal serves a stale fit right away and refits in the background. The fit now covers the whole lookback and takes ~27 s on the live install, up from 9–10 s, and the amp controller polls this endpoint with a 10 s timeout. A fit is stale after THERMAL_REFIT_S (6 h, as before) or when the idle-offset model changes. Only one background refit runs at a time, and the cache is warmed at app startup. Only a cold cache or an explicit ?refit waits. The cold wait is shielded so a caller that gives up doesn't cancel the shared refit, and a refit that finishes late doesn't overwrite a newer one.
  • contrib/backtest_forecast.py: leave-one-out params come from forecast_fits, matching what the deployed forecast learns from.
  • tests/test_wallmonitor.py: two new tests. The first checks that a stale fit is served without waiting and that repeated stale requests start only one refit. The second is a regression test. A flat history with a low stretch at the 40th-newest session must not alarm, and the old 40-session window on the same data must. It also pins the forecast to the newest 40.
  • docs/thermal-model.md: the "What it compares" section now says the watch uses the whole lookback.

Deliberately left alone: the forecast's 40-session window. Lifting it for the forecast too was backtested and rejected (see Verification). The constant's comment records why the cap is still there.

Risk

  • Drift verdicts change. The check now regresses on up to 120 days instead of ~5 weeks. Expect the 10-14 alert on the live install to clear at the next refit. A real slow trend now has more data behind it, and a short-lived blip near the window edge has less leverage.
  • Cost. The refit fits more sessions: 27 s for 72 fits on a 4-core Intel N100 with 8 GB RAM and NVMe storage, against 9–10 s for 30 on main. Slower hardware, such as a Raspberry Pi, or a heavier charging history will take longer. The poller runs it in a worker thread when a session ends. /api/thermal no longer refits inline: on main, the request that hit the 6 h refresh waited for the whole fit, which was already close to the amp controller's 10 s timeout. A stale fit is now served at once while the refit runs in the background.
  • Cold start. Right after a restart there is no fit to serve, so requests in the first ~30 s wait for the warm-up refit. The controller's polls in that window time out and skip their ticks. It handles that safely: it logs skip: cannot reach wallmonitor and leaves the current unchanged. main has the same window at 9–10 s.
  • Staleness. For up to one refit (~30 s) after the 6 h mark or an idle-offset recalibration, callers get the previous fit. The live prediction itself is still computed on every request.
  • API. session_fits in /api/thermal now lists the whole lookback, not the newest 40 sessions, and each fit gains a forecast boolean. The dashboard's rise-vs-date chart shows more history. Nothing else consumes the list.
  • Unchanged: forecast parameters, the live forecast, and the amp controller's inputs.

Verification

  • uv run pytest -q: 182 passed. The new test fails on main (assert 40 == 52: only 40 of 52 sessions fitted).
  • contrib/backtest_forecast.py was run against a snapshot of the live database (114 sessions), with --truth fit and --truth last:
    • Cap lifted for the forecast too (first attempt): median tau fell 11.5 → 11.25 min, n 1.53 → 1.47, and the in-run trajectory forecast at cold starts moved 0.4–0.5 °C optimistic (bias +0.12 → −0.31 fit truth, −0.28 → −0.79 last truth; optimistic share 18 → 24%, 20 → 31%). The sensor-ambient paths improved slightly and the cap path (implied/n) got slightly more optimistic. That's a net mixed result, so it was not shipped.
    • This branch: output is byte-identical to main in both truth modes.
  • Live timing: I started a scratch instance of this branch on the same N100 machine, using a snapshot of a live database, a spare port, and no charger. Results for /api/thermal: the first request took 33.4 s (cold, waiting on the warm-up), and the next two took 0.005 s each. The stale-cache path is covered by the new test.
  • On the same snapshot the drift check now returns Δ −0.12 °C over 79.7 days (n=45, CI −5.9..5.7), not drifting and not a lead.
  • ruff reports 3 errors that already exist on main in test files; none are from this change.

Deploy: update the package (or git pull a checkout) on the host running wallmonitor, then restart the service, e.g. sudo systemctl restart wallmonitor.service. The stale drift alert clears at the next refit (the next session end, or /api/thermal?refit=1 for the dashboard's view).

zebraengine and others added 3 commits October 19, 2026 12:51
…t 40 sessions

fit_sessions kept only the newest 40 sessions, so the drift regression's
baseline was whatever happened ~40 charges ago. On the live install a
single week of low fits (Sep 14-20) at that edge read as +8.1 C of
statistically confirmed drift and raised the alert on 10-14; over the
80 days since the verified anchor the same regression is flat
(-0.04 C, 95% CI -5.9..5.8).

fit_sessions now fits every session in the lookback and tags fits from
the newest FORECAST_SESSIONS (40) with forecast=True. fit_history takes
tau, rise and the current exponent from those only, so the forecast is
unchanged: lifting the cap for it too was backtested and moved the
in-run cold-start forecast 0.4-0.5 C optimistic. With the split,
contrib/backtest_forecast.py output is byte-identical to main under
both --truth fit and --truth last on the live history.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
The history fit now covers the whole drift lookback and takes ~27 s on
the live install (72 fits; it was 9-10 s for 30), while the amp
controller polls /api/thermal with a 10 s timeout. The endpoint used to
refit inline every 6 h, so that poll would time out and the controller
would skip the tick.

A stale fit (past THERMAL_REFIT_S, or a changed idle-offset model) is
now served at once while a single background refit runs. The cache is
warmed at app startup; only a cold cache or an explicit ?refit waits,
and the cold wait is shielded so a caller giving up doesn't cancel the
refit. A refit that finishes after a newer one doesn't overwrite it.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@zebraengine
zebraengine merged commit 5d3a392 into main Oct 19, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant