diff --git a/docs/SCOPE.md b/docs/SCOPE.md index e46fb47..84f6c6b 100644 --- a/docs/SCOPE.md +++ b/docs/SCOPE.md @@ -365,11 +365,11 @@ times its busiest shower) rejected right readings after a 35-minute update. when the size is back, or when the user says the region is right (`POST /sensors/{id}/accept-picture`) or draws a new one (both clear the size). Held rather than paused, so a camera that comes back at its old size needs nothing. A camera that moved without - changing size is not caught by this: that is C3 in `docs/WHEEL_READER_PLAN.md`. + changing size is not caught by this: that is *camera drift* in `docs/WHEEL_READER_PLAN.md` §8. **Wheel reader** (default for mechanical counters, `counter_reader: "wheels"`, registry -`wheel_readers` in `readers.json`, model `wheels-v1`, bundled, 2.2 MB, ~2 ms for 8 wheels on one -CPU thread): the crop is split into `digits` equal cells (whole cells, dividers included); each +`wheel_readers` in `readers.json`; bundled: `wheels-v1`, 2.2 MB, ~2 ms for 8 wheels on one CPU +thread, and `wheels-v2`, the default for new counters, 6.1 MB, ~7 ms): the crop is split into `digits` equal cells (whole cells, dividers included); each cell, grey by its darkest colour channel, 32×48, full contrast, goes through a small CNN that gives a distribution over 100 positions around the wheel (0.0–9.9). `readers.wheel_value` then finds the most likely *consistent* value by dynamic programming from right to left: the last @@ -377,7 +377,10 @@ wheel turns freely, every other wheel stands on its digit plus how far its right past 9. Wheels at rest may sit one bin off (`READING["wheel_slack_bins"]`), and all wheels may appear shifted together by up to `wheel_max_shift_bins` (a region drawn a little above or below the digits) at a cost of `wheel_shift_penalty` per bin. The last wheel is rounded to the nearest -digit; the confidence is the mean probability per wheel. It always returns `digits` digits, so +digit; the confidence is the mean probability per wheel (`"confidence": "mean"`). wheels-v2 also +reads every cell with half a cell of the frame on each side (`"context"`, +`readers.wheel_context_cells`) and averages both, and its confidence is the weakest wheel's +(`"confidence": "weakest"`). It always returns `digits` digits, so the digit count check does not apply. Counters made before it (no stored `counter_reader`) keep the text reader: schema v12 stores `"ocr"` for them, and so does importing an older export (`db.keep_text_reader`). The model, its training data and its @@ -387,6 +390,12 @@ results: [`tools/wheelreader/README.md`](../tools/wheelreader/README.md); how it - Why a wheel reader (2026-10-09): on every meter tested it read more readings right than PP-OCRv6 small and gave no value too high, which the text reader did (numbers in its README). The remaining misses are ±1 on a half-turned last wheel. +- Why wheels-v2 (2026-10-24): leaving each of two meters out in turn, five training runs + published no wrong value where wheels-v1 published six, read significantly more frames right + on one meter and the same on the other, and stays right when the region does not fit. It + publishes less often on a meter seen from the side whose last wheel turns; that was accepted. + Counters keep the reader they were set up with (schema v16, `db.keep_wheel_reader`). Numbers: + [`WHEEL_READER_PLAN.md`](WHEEL_READER_PLAN.md) §4.1. - Evaluated (spike on Commons photos): PP-OCR read LCD, LED, dot-matrix and flip-segment displays correctly (7/7 with a tight region); it fails on small blurry LCDs, and on rolling counter wheels when the whole counter is read as one line. DINOv2 per digit (4/23) and a CNN @@ -687,14 +696,15 @@ automatically. checked, accepted or rejected (newest first, `"answer": "unchecked"`; never used as labels without a check, see `docs/WHEEL_READER_PLAN.md`), whose frame is kept — each only the region it was read in plus `export_margin` (the region is kept with every stored reading since 0.7.1b2, - `probs["roi"]`; older ones are cut with today's region, `"region": "current"`). One ZIP of at + `probs["roi"]`; older ones are cut with today's region, `"region": "current"`; `"box"`: the + region inside the image in pixels, as the margin is cut off at the frame's edges). One ZIP of at most `READING["export_max_mb"]` (GitHub takes 25 MB per file; about 1,000 readings); the - readings that did not fit are counted (`left_out`). `readings.json` (`"format": 3`; 2 left out + readings that did not fit are counted (`left_out`). `readings.json` (`"format": 5`; 4 had no `"confidence_rule"`, 3 no `"box"`, 2 left out the rejected readings nobody answered, so a sensor's checks could not be replayed: read text, value, rejection reason, confidence, answer, right value, day only and seconds since the oldest reading; per reading the reader id and app version (kept with every stored reading since 0.7.1b5, null before), whether the picture was greyscale (IR), whether - the sensor's light was on and the camera picture's size; for the wheel reader each wheel's position, probability and most - likely positions, the common shift and the value with the last wheel's fraction), the reading + the sensor's light was on and the camera picture's size; for the wheel reader each wheel's position, probability (the one the confidence is made + of, `"confidence_rule"`) and most likely positions, the common shift and the value with the last wheel's fraction), the reading settings without anything that tells where the sensor is (no source, name or region), the user's optional "What meter is this?" text, a README and a CC0 LICENSE — shared images may then be used in tests and evaluations. Images are crops of the stored frames at their resolution. @@ -801,7 +811,8 @@ is part of backups and needs no restart. | **Counter limits** ✅ | A counter's limit grows with the time it was away (learned, a fixed rate or per reading), Review moves a counter on, readings held when the camera's picture changes size, training images in the app's storage (backups hold them) | 0.9.1 (betas 0.9.1b1–b6) | | **Reviewing readings** ✅ | A counter's value sent at its first reading, decimals for its rate, a rejected reading that repeats waits in Review once, Review shows a reading's region large and one sensor at a time | 0.9.2 (betas 0.9.2b1–b4) | | **Backup of everything** ✅ | One file with everything (parts to leave out, camera passwords only when ticked), a restore that replaces it and keeps the entities, backups of every database version kept in the tests; uploads over 16 MB through Ingress; much less memory (no ONNX Runtime arena, classifier without scikit-learn) | 0.10.0 (betas 0.10.0b1–b7) | -| **Better counter readings** | Benchmark, the last value as an expectation, calibrated confidence, extra decimal, self-calibrating regions, wheels-v2, learning per sensor — status per phase in [`WHEEL_READER_PLAN.md`](WHEEL_READER_PLAN.md) | planned | +| **Wheel reader v2** ✅ | Benchmark, `wheels-v2` (cells with context, weakest-wheel confidence) the default for new counters, counters keep the wheel reader they read with, exports say where the region is in each image | 0.10.1 (betas 0.10.1b1–b4) | +| **Better counter readings** | More shared meters, labels, loss and calibration for the next model, later the last value as an expectation — status per step in [`WHEEL_READER_PLAN.md`](WHEEL_READER_PLAN.md) | planned | **Open ideas** (not scheduled): more on the History page (export of a filtered selection, a summary per hour or day, "seen in the frame" as a filter, saved filters, bulk actions); merge diff --git a/docs/WHEEL_READER_PLAN.md b/docs/WHEEL_READER_PLAN.md index a1cc093..99a5afd 100644 --- a/docs/WHEEL_READER_PLAN.md +++ b/docs/WHEEL_READER_PLAN.md @@ -1,196 +1,341 @@ # Wheel reader: plan for better counter readings -The design for improving how VisionState reads **mechanical counters** (rolling digit wheels on -water and gas meters); what is done and what is next is in the table in §9. It applies to reading sensors with the *counter* display and the wheel reader -(`counter_reader: "wheels"`). The general design is in [`SCOPE.md`](SCOPE.md), and the model and its -training are in [`tools/wheelreader`](../tools/wheelreader/README.md). - -## 1. Where we are (wheels-v1) - -- **How it reads.** The region is split into one cell per wheel. A small CNN (565 k parameters, - 2.2 MB, about 2 ms for 8 wheels on one CPU thread) gives each wheel's position as a distribution - over 100 bins (0.0–9.9). `readers.wheel_value` then finds the most likely value whose wheels fit - together, from right to left, with one common shift of up to ±2 bins. The last wheel is rounded, - and the confidence is the mean probability per wheel. -- **Training data and results:** [`tools/wheelreader/README.md`](../tools/wheelreader/README.md). -- **Known weaknesses, with evidence:** - - *±1 on the last digit* when the last wheel is about half way. The rounding moves with the - per-frame shift (pedro32: 81 exact + 12 at +1). - - *Only three real meter types tested.* Gas meters, IR or night images and angled cameras are untested. - - *Confidence is only moderately informative.* A threshold that rejects 4 % of the reads catches - about half of the wrong ones (Dryad test). - - *Wheels at rest sit off their digit, each its own way*, but the decoder allows only one common - shift plus ±`wheel_slack_bins`. On oleost1 two black wheels sat at 0.3–0.4 past their digit all - day while the others sat at 0.1–0.2. The decoder put them outside the slack, where their - probability dropped to 0.02–0.08 although the digit was plain. That one wheel pulled the mean - under 0.70, so 22 of the 29 rejected readings were right. A low threshold is not the fix: at 0.56, - 3–4 of that day's 7 misreads would have passed, one of them too high (585 instead of 565). It can - also flip the digit. In one sharp, well-lit frame the black 7 had 0.49 of its mass at 7.2–7.3 - and 0.21 at 0.2. Only 7.2 fell inside the slack, so the decoder read 032.776 instead of 732.776 - at confidence 0.84. Only "went down" caught it. See A2. - - *Red wheels half way while the water stands still.* All of oleost1's misreads came from the last - wheel stopped exactly half way (6½). The neighbouring red 9 or 6 was then read as 8, and the last - wheel as 2, 8 or 9. The same hard frame then comes back for as long as nobody draws water. - - *Every frame is read on its own.* The last value is only used afterwards, by hard checks - (`went down`, `max_step`, `readers.settling`). - - *Geometry is assumed:* equal-width cells and a fronto-parallel view. - - *A wrong label can hide in the training data* (pedro32/0120 is labelled 628596; the wheels show 7–8). +How VisionState is to read **mechanical counters** (rolling digit wheels on water and gas meters) +better: what the shared data shows, what was tried, and what comes next, with its status in §9. +It applies to reading sensors with the *counter* display and the wheel reader +(`counter_reader: "wheels"`). The general design is in [`SCOPE.md`](SCOPE.md); the model and how +to train it are in [`tools/wheelreader`](../tools/wheelreader/README.md). + +## 1. What the data shows + +**How wheels-v1 reads.** The region is split into one equal cell per wheel. A small CNN (565 k +parameters, 2.2 MB, about 2 ms for 8 wheels on one CPU thread) gives each wheel's position as a +distribution over 100 bins (0.0–9.9). `readers.wheel_value` finds the most likely value whose +wheels fit together, with one common shift of up to ±2 bins; the confidence is the geometric mean +of the decoded bins. + +**The benchmark** (§3) holds about 6,000 readings of four real meters and the Dryad test photos: +the maintainer's water meter (8 wheels, 3 red, ESP32-CAM; 8 days), pedromfa's (7 wheels, ESP32; +8 days, shared in issue #40), two bolausson meters and Dryad's 5- and 6-digit meters. Results of +wheels-v1, region as drawn: + +| Set | Right (±1 on the last wheel) | Too high | Note | +|---|---|---|---| +| pedromfa, 2 periods | 99.0–99.7 % | 4 + 1 | the replay (§3) of the first period: 3 too high published, 5.3 h blocked | +| maintainer, 19–22 Oct | 98.0 % | 1 | | +| maintainer, 14–18 Oct | 86.6 % | 9 | 6.7 h blocked over the 8 days (5 stretches) | +| bolausson cold / warm | 100 % / 100 % | 0 | | +| Dryad 5 digits / 6 digits | 99.4 % / 90.2 % | 6 / 26 | right way up | + +What goes wrong, in order of harm: + +1. **A region that does not fit the wheels.** v1 reads the middle of equal cells, so a region a + little wider, narrower or off makes the outer cells show half a wheel. On pedromfa's frames + (99–100 % as drawn), with the region's right edge half a cell short v1 read 43–69 % right; + with half a cell more on each side, 0–94 % depending on the training run; 10 % taller added up + to 9 too-high readings. The maintainer's region cut the last wheel, which caused most of the + misreads of 14–18 Oct. +2. **The last (red) wheel half way, on a meter the model has not seen.** On the maintainer's meter + a red wheel between 5 and 6 was read as 7.8–8.0 (two steps too high) or as 2.0; the + second-to-last wheel half way between 1 and 2 was read as 5 (734.156 published for about + 734.120). When the water then stands still, the same hard frame comes back for hours, and a + too-high value blocks the sensor until the meter catches up. Three of the five blocked stretches + came from a too-high value. In the other two the value was right and the readings after it + misread lower. +3. **A confidence that hides one bad wheel.** The mean over wheels stays high when one wheel is + unclear. The weakest wheel's mass within ±0.4 of where the value puts it ("weakest wheel") is a + much better signal. Replayed through the counter checks at 0.70, it published 2 wrong instead + of 5 on the maintainer's meter (436 instead of 419 published), and 0 instead of 3 on + pedromfa's. With 1 % of the accepted readings wrong, 82 % instead of 42 % of the maintainer's + readings were accepted; risk–coverage (AURC) was better on Dryad and pedromfa #32 too. +4. **Dark, striped or noisy frames** (camera glitches) were read as noise; the checks stopped them + all, two above the threshold. +5. **Dryad 6-digit meters** (dirt, blur, fog) are read 83 % exactly. + +**Trained candidates** (same recipe, 25 epochs, three training runs each where it mattered; the +benchmark differs by 3–5 points on the maintainer's hard period and 2 on Dryad 6 between runs of +the *same* recipe, so smaller differences mean nothing): + +| Candidate | Result | +|---|---| +| **Cell with context**: each cell plus half a cell on each side, 64×48 input, drawn rows of wheels with the region off by up to 38 % of a cell | **robust to a region that does not fit**: 99–100 % on pedromfa's frames with the region half a cell short or wide, or 10 % taller (v1: 0–94 %); as drawn the same as v1 (within run-to-run spread); 958 k parameters, 3.7 MB, 4.7 ms for 8 wheels on one thread | +| The same, model twice as wide | no better | +| One cell, drawn cells with the wheel off-centre and neighbours' digits | no better than v1 | +| **One-cell net + context net, leaving one meter out** | better reading and geometry, more wrong published values in some runs: §4.1 | +| Shared exports as training data (labels: §5) | on the meter's own other period: small gain at most (14–18 Oct 83–89 % → 91 % with 19–22 Oct in training; the other way round no gain) | + +**Tried without training, and dropped:** reading 3–8 slightly shifted crops and averaging (a +small gain on one meter, none on the others); letting the app tune the region by what the model +finds most likely (§7: it made things worse). ## 2. Principles -1. **The base model stays general and stable.** Per-sensor tuning is a fallback on top of it, never - instead of it, and it can always be switched off. -2. **Every change is measured** on a fixed benchmark (§3) before it ships. Nothing that adds a - reading that is *too high* ships: a too-high value passes the counter checks and blocks the - sensor until the meter catches up. -3. **Only human answers are labels.** The model never trains on its own guesses (confirmation bias: - self-training on its own pseudo-labels makes wrong predictions stronger, especially when the - images differ from the training data). +1. **The base model stays general and stable.** Anything per sensor is a fallback on top of it, + never instead of it, and it can always be switched off. +2. **Every change is measured** on the benchmark (§3) before it ships, with at least three training + runs. Nothing that adds a reading that is *too high* ships: a too-high value passes the counter + checks and blocks the sensor until the meter catches up. +3. **Labels come from people or from other frames, never from the reading itself.** A reading's + label is a person's answer, or the interval between the readings next to it in time that the + sequence confirms (§5). A model never trains on its own value for a frame (confirmation bias: + self-training on its own pseudo-labels makes wrong predictions stronger). 4. **Licences:** training data must allow any use (CC0, or CC BY with attribution in the model card and a verified source). No NC/ND sets, no sets without a licence. 5. **No telemetry.** Data only arrives when a user exports it and shares it on purpose. -6. **The registry rules stay.** A changed model, or changed calibration of a model, gets a new +6. **The registry rules stay.** A new model, or a changed way to compute its confidence, gets a new registry id. Existing sensors keep their behaviour until the user picks the new one, as in 0.8.0 (counters made before the wheel reader kept the text reader). -## 3. Phase 0 — benchmark and metrics (the base for everything else) +## 3. The benchmark -**Benchmark** (in `VisionStateLocal/`, never in the repository; any image that can go into the -repository is CC0 and becomes a test asset): +The scripts are in [`tools/wheelreader`](../tools/wheelreader/README.md) (`bench.py`, +`exports.py`); the sets live in `VisionStateLocal/wheel/` and never go into the repository (a +set's images may only become test assets when they are CC0). The sets of the new exports are +`ole_a`, `ole_b`, `ped_a` and `ped_b`, and their twins with labels read by eye (`_eye`, §4.1). | Set | Source | Use | |---|---|---| -| `pedro40` 0001–0017 | issue #40 (CC0) | held out (wheels-v1 was trained on it in the end: hold out the *next* export instead) | +| `ole2_a`, `ole2_b` | the maintainer's export of 2026-10-22 (CC0), split by date: 14–18 Oct / 19–22 Oct | test; `a` may train a candidate that is then tested on `b` only | +| `ped18_a`, `ped18_b` | pedromfa's export of 2026-10-18 (#40, CC0), up to / after 15 Oct | the same | +| `pedro40`, `pedro32`, `pedro_esp` | pedromfa's earlier exports (#32, #40) | wheels-v1 was trained on them | | `bol_cold`, `bol_warm` | bolausson (GPL-3.0) | test only, never training | -| `dryad_test` | Dryad (CC0) | test, right way up | -| `oleost1`, … | the maintainer's water meter (CC0 exports, one per few days from 2026-10-14; ESP32-CAM 1600×1200 with flash, 8 wheels, 3 red) | held out: test only, a meter with wheels sitting off their digits at rest | -| new shared exports | §5 | **split by meter**: one meter is either training or test, never both | - -**Metrics**, reported per set by `tools/wheelreader/evaluate.py`: - -- **exact** (last digit rounded), **±1**, **too high** (above +1), **too low**. -- **Risk–coverage:** sort the readings by confidence and plot the error rate of the accepted - readings against the share accepted. The default threshold is then picked for a target such as - "at most 1 % of accepted readings wrong", and its coverage (how many are accepted) is reported. - The area under the curve (AURC) gives one number to compare models and decoders. -- **Calibration:** expected calibration error (ECE). Readings are grouped by confidence, and in - each group the share that is right is compared with the mean confidence. -- **Sequence metrics** for exports that are a time series: published value right, number of - rejections, longest stretch blocked by a too-high value. - -**Gate for a new model or decoder:** no set gets more too-high readings, the exact rate on unseen -meters does not drop, and AURC on the benchmark is lower. - -**Label hygiene:** a reading whose label the model strongly disagrees with (very likely value ≠ -label, high confidence) is listed for a person to look at again. That is how pedro32/0120 was -found. The label is corrected or the image is dropped, never silently. - -## 4. Phase A — safer decoding (no new training) - -### A1. The last value as a soft expectation - -**Why in the decoder and not in the network:** the network sees one wheel. Teaching it time would -need sequences we do not have, and it could learn to repeat the last value instead of looking. -In the decoder the expectation can be tested, explained, set per sensor and used with every model. - -**Design:** - -- `wheel_value` gets an optional *prior* over the value V (in last-wheel units, with the fraction): - `score(V) = Σ wheels log p(position_i(V)) + log prior(V)`. -- The prior comes from the last **published** (debounced) value L at time t₀, with elapsed time Δt: - - V < L − settling tolerance: −∞ in practice (a very large penalty, so it can still be inspected); - - L ≤ V ≤ L + R·Δt + slack: flat, with a mild preference for small steps (an exponential - in V − L); - - above that, a steep penalty. -- **R (largest plausible rate):** the sensor's `max_step` per interval when it is set, otherwise - learned from its own history (for example 3× the 99th percentile of the accepted rate over 30 - days, at least a default in `settings.READING`). It is never learned from rejected readings. -- **Search:** the window [L, L + R·Δt + slack] is usually small (a few hundred last-wheel steps). - Every V in it is scored directly (wheels × candidates, microseconds), and also the free - optimum without the prior (today's DP). Result: - - the free optimum is inside the window: same as today, with a better confidence; - - outside the window, but the window's best explains the image almost as well (log-likelihood - ratio below a threshold): take the window's best (an unclear wheel settled by the expectation); - - outside and clearly better: **do not publish**. Mark the reading as "disagrees with the last - value", send it to review and count it. -- **Protection against lock-in** (a wrong value used as the expectation): - - only debounced, published values become L; - - N readings in a row that clearly disagree with L, *and agree with each other*, release the - prior. They are treated as a new start, which also covers a replaced or reset meter. This is - shown in the History as an event and on the Live tab. The counter limit decided otherwise for - now (readings that agree never override it, `docs/SCOPE.md`): a release would wait for Review; - - after a restart, L is the last value from the history (as today), but its age counts: an old L - gives a wide window. -- **UI and history:** a new reason, `disagrees_with_last` (lower-case key for the "problem" entity), - and the Live tab shows "read as … (expected … – …)". -- **Tests:** a pure unit test of the prior and the window decision (drawn positions); integration - tests that replay sequences (one blurry wheel settled by the prior; a meter that is replaced - releases the prior after N readings; a too-high value is never published). -- **Measure:** the sequence metrics on the exports, before and after. - -### A2. A confidence that means what it says - -- **Problem:** the mean per-wheel probability is not the chance that the whole reading is right. - Networks are also often over-confident (Guo et al. 2017). -- **Temperature scaling:** divide the logits by one temperature T, fitted on the benchmark's - held-out real images. This changes the confidence, not the reading. -- **Reading-level confidence:** use the decoder's alternatives: P(best V) / Σ P(top-k V) (a - normalised margin between the best value and its nearest rivals, typically ±1 on a wheel), not the - mean per wheel. The k-best come from the DP (keep the top few per state) or from the A1 window. -- **A wheel at rest counts by its whole digit**, not by the one bin the common shift puts it at: its - probability is the mass within about ±0.45 of its digit. The **weakest wheel** also counts on its - own, not only through the mean. On oleost1 (§1), with only the exported top three per wheel, the - right readings came out at 0.94–0.99 with their weakest wheel mostly ≥ 0.8. The misreads had a - weakest wheel of 0.14–0.43. "Weakest wheel under about 0.5 → unsure" would have passed nearly all - right readings and caught every misread, including the one that was published (last wheel read - as 2 at 0.44). This is to be measured with the full 100-bin output on the whole benchmark, set - against the value margin above. -- **Pick the default threshold** from the risk–coverage curve (§3). The UI text stays "at least - 70 % sure", but 70 % then really means 70 %. -- **Registry:** T belongs to the model, so a calibrated wheels-v1 is a new entry - (`wheels-v1c`, same file and SHA-256, plus `"temperature"`). Sensors keep their old behaviour until chosen - (registry rule; `confidence: "wheel-mean"` versus `"value-margin"`). - -### A3. The last wheel's fraction - -- The decoder already knows the last wheel is at, say, 9.8. A sensor option "**one more decimal from the last - wheel**" (wheel reader only) publishes V with one decimal more, for a smoother rate entity and - earlier leak detection. -- The settling rule (`readers.settling`) and the "went down" check work on V with a tolerance of a - few tenths, not one whole step. -- Off by default (entity precision and the Energy dashboard change), with a hint in the reading editor. - -## 5. Phase B — better data in - -### B1. The export ("Export readings") — done - -Built in 0.8.0; what it contains is in [`SCOPE.md`](SCOPE.md) §12. Decisions that differ from the -first plan (2026-10-10): -- the light is recorded per reading automatically (greyscale picture, and whether VisionState had - the sensor's light on) instead of a flag in the dialog; -- frames are **not** deduplicated in the app — a reading of the same frame is still a reading; the - training tools remove duplicate images (§B4); -- the unchecked readings, accepted and rejected, are **always** included (no option, no count - limit): the images are what is scarce, and with the rejected ones a sensor's checks can be - replayed in order (`seconds`). Their value is the reader's guess, so the tools use them as labels only after - a check (§B4); -- no spot-check button; `spot_rate` is the way to get accepted readings checked; -- one ZIP of at most 24 MB (GitHub's limit is 25 MB per file), checked readings first, then the - newest unchecked ones; what does not fit is counted in `left_out`. Splitting into parts was built - and dropped as more than we need now. Images are not scaled down. - -Privacy: only the region with a small margin, no camera, name or position, and the day only. The -user looks at the images before sharing (README.txt in the ZIP). - -### B2. Collecting - -- A pinned GitHub Discussion, "Share your meter", that explains why exports help, how they are - used (split by meter; credited in the model card) and that CC0 applies. -- A Discussions form (or an issue template) with the meter type, camera and light. -- Most wanted: gas meters (often white on black), electricity meters with a rolling counter, IR or - night frames, angled cameras and fogged glass. - -### B3. External data (checked 2026-10) +| `dryad5`, `dryad6` | Dryad test set (CC0); a third of the photos is upside down, the right way up is kept | test | +| new shared exports | §6, from VisionState 0.8.0 on only (§4.2) | **split by meter** (or by date for one meter); a candidate is tested leaving its meter out | + +**Labels:** + +- A person's answer (checked readings) gives the value. +- Every other reading gets the interval between the trusted readings before and after it in time. + Trusted readings are the checked ones, plus the accepted readings on the longest non-decreasing + chain of checked and accepted values (checked ones weigh most). +- Where a stretch was blocked (a run of "went down"), the frames were looked at and labelled by + eye. + +The export now says where the region is in each image (`"box"`). Before, a margin cut off at the +frame's edge put the cells in the wrong place in 94 % of the maintainer's images. + +**Metrics** per set: + +- right (within ±1 on the last wheel), exact, too high; +- risk–coverage: AURC, and the share accepted at 1 % wrong — for the mean and for the + weakest-wheel confidence; +- for the time series: a **replay** through *went down* and the change limit at threshold 0.70. + It counts published values, wrong ones and the hours a too-high value stood; +- **geometry**: the same frames read with the region's edges moved (half a cell short or wide, a + quarter cell off, 10 % up, down, taller, shorter). + +**Gate for a model or decoder:** over five training runs (§4.2), no set gets more too-high readings and +none drops by more than the run-to-run spread. The geometry cases must not get worse. AURC must +not rise. + +## 4. The next wheel model + +### 4.1 Where wheels-v2 stands + +The first candidate: a one-cell net and a context net (each cell plus half a cell on each side, 64×48), +their log probabilities averaged, the weakest-wheel confidence. Measured **leaving one meter out** +(trained without the meter it is tested on, three training runs), against wheels-v1: + +| | wheels-v1 | v2, three runs | +|---|---|---| +| maintainer 14–18 Oct, right / too high | 86.6 % / 9 | 92.3–93.5 % / 6–10 | +| maintainer, replay: published / wrong / hours too high | 192 / 2 / 3.2 h | 198–218 / 3–5 / 2.4 h | +| maintainer, region as drawn, too high | 2 | 3–5 | +| Dryad 6 digits, right / too high | 90.2 % / 26 | 91.5–94.0 % / 9–16 | +| region's right edge half a cell in: maintainer / pedromfa | 43 % / 69 % | 64–66 % / 93–99 % | +| pedromfa #32: share accepted at 1 % wrong | 0.95 (trained on it) | 0.11–0.23 | +| bolausson cold | 35 of 35 | 34 of 35 in two runs | + +Clearly better at reading and at regions that do not fit, but worse on wrong published values and, +in some runs, on too-high readings: **not released** (the maintainer's rule: nothing ships that is +only a little better and worse on any point). What was learnt on the way: + +- **A model memorises a meter's framing.** Trained on pedromfa's 716 frames, all cut the same way, + a region moved a quarter cell read 0 % right (239 too high). Training on the same frames also + cut with the region moved at random (8 cuts per frame, up to 38 % of a cell) fixed it (99.6 %). +- **The context net alone is unstable** between training runs (maintainer 14–18 Oct: 70–91 %); + averaging weights (EMA) did not help; the pair with a one-cell net is stable. +- **The hard case on the maintainer's meter** is a wheel showing 7 with the next digit in view + below it, read as 2 or 4. Crossed sevens are now in the drawn rows. +- **Run-to-run spread is as large as most differences** (3–5 points on the hard period), and the + counts that decide (wrong published values) are small (2 against 3–5): three runs cannot tell + them apart well (§4.2, step 1). + +**On the new exports** (the maintainer's and pedromfa's of 2026-10-23, from 0.8.0 on, each split +in two periods `a` and `b`; leaving one meter out): + +- **Labels read by eye.** The chain labels of §5 were too wide to judge a model by (median 8 + steps, 3–18 % exact). All 1,305 test frames were read by eye instead: one value, or two + neighbours when the last wheel stands between digits; 4 frames nobody can read are left out. + Five of these readings were wrong at first, all caught where the readers were sure and the + frame, zoomed in, agreed with them. +- **One network pair is still not steady.** Five training runs trained on chain labels published + 16, 3, 2, 0 and 0 wrong values on the maintainer's first period (wheels-v1: 3); trained on the + labels read by eye, three runs published 1, 3 and 0, and one run was worse than wheels-v1 on two + of the four sets. +- **An ensemble is.** The average of five runs (or three) published **no wrong value on any of the + four sets** (wheels-v1: 3, 2, 1 and 0), read more frames right (maintainer `a`: 97.8 % of the + runs of frames against 93.0 %; the paired interval +1.5 to +8.5 points) and published more. + It costs one pair per run, hence distillation (§4.2, step 4). +- **Distilled into one pair: wheels-v2.** One pair trained towards the ensemble's probabilities + (`train.py --teacher`; teacher of eight pairs, sharpened with `--teacher-t 0.3`, 70 % weight on + the labels with `--alpha 0.7`, not restricted to the interval). In five runs: **no wrong value + published** in 19 of the 20 runs and sets, one run 1 on pedromfa's `a` (2 litres, a last wheel + half way; wheels-v1 there: 1); wheels-v1 published 3, 2, 1 and 0. The paired gain over + wheels-v1 is significant on the maintainer's meter (`a` +4.4 points, `b` +1.4) and none on + pedromfa's (+0.6, +0.1); the region moved or the wrong size hurts far less (right edge half a + cell in: 96 % against 69 % on pedromfa's meter). With half the weight on the teacher, + unsharpened, one run of three published a wrong value standing for 6 hours (a wheel half way + from 1 to 2 read as 7.6), with a teacher of three pairs as of eight. A temperature fitted on the + Dryad test photos does not help: there it is already right (99 % of the wheels). +- **Where it is worse:** it is less sure on the maintainer's `a`, a meter seen from the side whose + last wheel turns and is partly hidden (128–149 frames published at the threshold, wheels-v1 + 168 with 3 of them wrong, the ensemble 202). Of the frames it holds back there, 97 of 101 it + would have read right; counting the last wheel's confidence within ±1 instead of ±0.4 digit did + not change that. The maintainer accepted this (2026-10-24): most meters can be faced square on, + and the cost is a later value, not a wrong one. +- **Released as `wheels-v2`:** the same recipe on all frames of both meters, the teacher all 16 + pairs (each meter seen by half of them). The 33 free photos of §3 (seen by no model): 28 right + (wheels-v1 26), none wrong published; Dryad six digits 94.0 % (wheels-v1 90.2 %, too high 13 + against 26); the two CC0 test photos and the drawn counters right. 6.1 MB, about 7 ms for 8 + wheels on one desktop thread (wheels-v1 2.3 ms); not timed on a Raspberry Pi (none at hand). + +### 4.2 How the next model is trained + +New data first: the maintainer and pedromfa export again with 0.10.1b1 or later, and training and +tests use **readings from VisionState 0.8.0 on only** (`"region": "as read"`, with `"box"`), so +every cell is cut where it was read. Then, in this order, each step measured on its own (§3): + +1. **Measurement that can decide.** Five training runs instead of three; a per-frame paired + comparison (the same frames read by both models) with a bootstrap interval, instead of + comparing totals; a still meter's repeats counted once (a frame shown for hours otherwise + counts hundreds of times). Without this, a gain and the noise look alike. +2. **Decoding the time series** (app side, no new model). A counter never goes down and rises at + a bounded rate, so the value is decoded over the last frames together (a Viterbi pass over + candidate values with monotone steps), not frame by frame. It targets exactly what held v2 + back: a hard frame that comes back for hours. It helps v1 too, comes as a new registry entry + or setting so existing counters keep their behaviour, and is measured with the replay. +3. **Labels and loss.** The interval loss only asks that the mass lies inside the label's + interval. Add a soft circular target inside the interval (most mass near the middle for a wheel + at rest), and fit a temperature on held-out meters (Guo et al.), so that "70 % sure" means about + that. Weight the frames that matter: a wheel half way, the last wheel, red wheels. +4. **Teachers into one student (distillation).** Train five networks, then one network of the + app's size on their averaged output, restricted to each frame's label interval and + renormalised. The labels still come from other frames (principle 3); what the ensemble knows + about *where inside* the interval comes along, and the run-to-run spread of a single network + shrinks. Frames with a wide label interval then teach too. +5. **Better drawn wheels.** Today's rows are flat strips. Draw the wheels as cylinders (digits + squeezed at top and bottom, curvature, the next digit half in view), with the flash hot spot + and glass reflections of an ESP32-CAM, red wheels, IR and fogged glass, and the fonts of real + meters. This aims at what no amount of one meter's frames fixes: a meter the model has not seen. +6. **A pretrained start, last.** Self-supervised pretraining on all unlabelled cells (every export + frame and Dryad), then training as now. It helps most when labels are scarce, but fine-tuning + can drift towards the few labelled meters when the domains differ (Wang et al.); only worth it + if steps 1–5 leave a gap. A large backbone (DINOv2) is too slow per wheel on small hardware. + +Not planned: self-training on the model's own readings (principle 3), and a new network design +before the steps above are measured. + +### 4.3 Model and app + +The context design stays the base: neighbours in view let the model find the wheel nearest the +middle when a cell is off, and see a wheel the region cuts. A second output gives each cell's +embedding (the layer before the bins), so *learning per sensor* (§8) can be built in the app +without a new model. The app is ready for a second wheel reader: each counter names its own +(`readers.wheel_reader_of`; counters set up before keep wheels-v1), a model with a second input +is fed the cells with their neighbours from the frame (`readers.wheel_context_cells`), the +confidence kind is a registry field, and a wheel reader is loaded on first use and let go when no +counter reads with it. Both models are bundled; new counters read with wheels-v2. + +**Before a release** of a wheel reader: five training runs pass the gate (§3) on the shared +exports, with no case of §4.1 worse than the reader before, or the case named and accepted; the +model checked against real photos in the tests (the drawn e2e counter and the CC0 test images); +a model card with the data and its licences on Hugging Face. + +### 4.4 Learnt making wheels-v2, for the next model + +**Data: check everything, assume nothing.** + +- **A shared export is not ground truth.** Chain labels (§5) are too wide to judge by, and a + checked or accepted value can be wrong. Read the test frames by eye (`frames.json`, `null` for a + frame nobody can read). Then check every frame where a model is sure and disagrees with the + label: zoomed in, five of the first readings by eye were wrong. Frames that stay unclear go to + the meter's owner (the maintainer confirmed two that way). +- **Before using an export:** it has `"region": "as read"` with `"box"` (0.8.0 on; before, the + cells were in the wrong place); draw the box on a few images; digits and decimals match the + meter; its LICENSE is CC0. After any label change, build the sets again (`exports.py` writes + both the bench and the training sets). +- **A meter's frames are also cut with the region moved** (`exports.py ... 8`); without it a + model learns one meter's framing and fails when the region moves a quarter cell (§4.1). + +**Measuring.** + +- **Five training runs, not three.** One run in three kept publishing the same wrong value (6 + hours too high) while the others did not. An ensemble of the runs shows what one network can + reach. +- **The replay through the app's rules decides**, not accuracy alone. The test frames are hours + apart, so the change limit hardly acts there; a real camera sends far more frames. +- **The user's threshold (0.70) is shared by all models**, so a model must be sure where it is + right: `bench.py eval --sweep` shows how much it would publish at other thresholds. The Dryad + test photos are too easy to fit a temperature on (99 % of wheels right). +- **Look at the frames behind a number** (cells side by side). The gap that was left was one + side-on meter whose last wheel turns and is partly hidden; counting the last wheel within ±1 + digit instead of ±0.4 did not close it. +- **Meters no model has seen:** the 33 free Openverse photos (`ov4`–`ov8`) and Dryad's six-digit + photos; the shared meters are both water meters with red decimals. + +**Training.** The recipe is in [`tools/wheelreader`](../tools/wheelreader/README.md). A teacher +of three pairs did as well as one of eight. A soft teacher left the student unsure; sharpening it +(`--teacher-t 0.3`) with more weight on the labels (`--alpha 0.7`) removed the last wrong value. +Not tried yet: + +- the teacher only inside the label's interval (`--restrict`); +- cells skewed and with one edge hidden (a meter seen from the side), and more weight on a last + wheel that turns; +- cylinder-drawn wheels (§4.2, step 5); +- grey from the darkest colour channel makes orange digits on black wheels dark (a UGI gas meter + photo); check preprocessing for coloured digits on dark wheels. + +**Releasing.** + +- Check the ONNX file against PyTorch (`export_v2.py` prints the largest difference) and on the + free photos before uploading it. +- The tests that assumed one wheel reader had to change: the default in the wizard, the name on + Settings, and the drawn e2e counter at x.5, where either neighbour is right. +- Text that only shows with two wheel readers had never been seen: look at every notice a new + reader brings up, on a phone too. + +## 5. Labels from shared exports + +- An export holds the checked readings and all unchecked ones, accepted or rejected, with + `seconds` (`SCOPE.md` §12). From those, every reading gets an interval of values from its + neighbours (§3). Every wheel then gets the interval of positions it passes through, widened a + little for a wheel at rest. A wheel that may have turned more than half way gets no label (zero + loss). +- The intervals come from other frames, so a reading's own misread never teaches the model + (principle 3). The chain can still hold an accepted misread of a step or two. Such a misread + only widens the intervals around it. +- Label check after every training run: a reading whose label the model strongly disagrees with + (very likely value ≠ label, high confidence) is listed for a person to look at. That is how + pedro32/0120 was found (labelled 628596; the wheels show 7–8). The label is corrected or the + image dropped, never silently. +- Deduplicate a still meter's repeats when they dominate a set, and split by meter (or by date for + one meter). + +## 6. Data: the biggest limit + +Four real meters are not enough to measure "a meter the model has not seen" well. On the +maintainer's meter every candidate reads the hard period at 81–89 %, whatever its design. Wanted +most: + +- more meters, each exported after a week or two: gas meters (often white on black), electricity + meters with a rolling counter, red decimal wheels, IR or night frames, angled cameras, fogged + glass; +- sharing already works (*Quality* → *Export readings*, then a GitHub Discussion; `visionstate/DOCS.md` + says how). What is missing is asking for it: a pinned Discussion, "Share your meter", saying + which meters are wanted most and how exports are used (split by meter, credited in the model + card); +- external sets checked so far (2026-10): | Source | Licence | Verdict | |---|---|---| @@ -200,170 +345,106 @@ user looks at the images before sharing (README.txt in the ZIP). | UniData / ud-smart-city water meter sets | CC BY-NC-ND 4.0 / commercial | not usable | | UFPR-AMR, Copel-AMR (Laroca et al.) | no open licence found (given out on request for research) | not usable | | AI-on-the-edge data and models | no licence (data); non-commercial (project) | ideas only, no data or code | - -### B4. Data hygiene - -- Deduplicate by frame. Split by meter (or by date for one meter, as for #40). -- **Unchecked readings** (`"answer": "unchecked"`) are the reader's own value, so they are never a - label as they are (principle 3). They become one after a check: a person looks at them, or the - sequence confirms them (a counter only goes up: a value between two confirmed neighbours that - fits the rate). Until the tools do that check, they skip them. -- A label check after every training run (§3). -- A last wheel stopped half way has two right answers. On oleost1 the same frame was marked right - as 596 on some readings and as 597 on others. The tools count either neighbour as right there and - keep such cells as interval labels, as the per-sensor labels in §8 do. -- Collect the half-way red-wheel cells from exports as their own case for wheels-v2 (§7). -- `VisionStateLocal/README.md` lists the source and licence of every set, as it does now. - -## 6. Phase C — geometry: let the app correct itself - -The goal: a region that is drawn a little off, or a camera that moves a little, does not hurt. - -### C1. A more robust model (training, in wheels-v2) - -Drawn cells with perspective, rotation of ±5°, cells of unequal width, a neighbouring wheel -showing at the side and wheels that sit a little off each other at rest. - -### C2. Calibration per sensor (no training) - -- In the background, every so often (for example after 50 readings, and after the region is - changed), the app takes the last K frames of the sensor and tries small changes to how the region - is read: - - vertical offset ±10 %; - - height scale ±10 %; - - rotation ±3°; - - cell borders moved ±10 % one by one, or as a linear stretch (cells of unequal width); - - a rest offset per wheel: where each wheel that is not turning sits relative to its digit, found - over the K frames (on oleost1 it was +0.3 for two wheels and +0.1 for the rest, the same all day). -- It keeps the change with the best total decode score over the K frames. Cost: K × changes × - about 2 ms; a coarse-to-fine search keeps it to a few seconds on a Pi. -- **The result** is stored with the sensor (`reading.calibration`, reset when the region or the - number of digits changes). - - A small change is used straight away and shown on the Live tab. - - A large change is offered: "The digits sit a little lower than the region. Adjust?", with a - before/after image. -- It **replaces the per-frame shift** in time: a fixed offset per sensor gives a stable last digit - and removes most ±1 readings (§1). -- Tests: drawn counters with a known offset, rotation and cell stretch, where the calibration - must find them again; e2e for the suggestion on all three devices. - -### C3. Camera drift - -For a camera that moves a little over days (a bracket that sags, a door being closed): estimate -the shift of the region against a reference frame by phase correlation (numpy FFT, sub-pixel, no -OpenCV) and move the region along. AI-on-the-edge does this with two reference marks the user -picks. Doing it automatically, on the area around the region, needs no user step. It must check -the correlation peak and skip frames where it is weak (light changes, IR switching). -A first real case was oleost1: an ESP32-CAM mount moved on its first day, the region was redrawn -twice, and the digits still sat at different heights in crops of the same size. -A camera that comes back with another picture size is already caught without this (readings held, -"camera changed", [`SCOPE.md`](SCOPE.md) §5); C3 is for a picture that moved at the same size. - -### C4. Perspective correction - -The manual 4-point correction stays planned for strongly angled cameras (see the earlier -decision: measure the read rate at real angles first, for example with the user's ESP32 camera, or -with `fake_camera` given a perspective parameter). C2 can later suggest the four points. Open -questions: reuse the 4-corner polygon region or a new corner tool (existing sensors must keep -working), the corner order and orientation, and dragging corners on a phone. - -## 7. Phase D — wheels-v2 - -- **Drawn cells:** whole counter rows drawn in perspective and then cut like the app does; - monochrome IR; flash reflections on glass; red wheels stopped half way; fog and drops; DIN-like narrow fonts (OFL); the failure - types from phase B. -- **Context:** the cell plus a strip of each neighbour (or a model that reads the whole row and - gives one position per wheel), so carries are seen together. Measure against the per-wheel model. -- **Test-time augmentation:** read 2–3 slightly shifted crops and average them (a few ms). A small - ensemble only if the benchmark shows a clear gain. -- **Dryad position labels:** fit positions for the Dryad crops with v1 (inside each crop's - nearest-digit interval, offline) and train v2 on the sharper labels. This is only for that public - training set, anchored by the exactly known drawn wheels, never for user data. -- **An embedding output** (the layer before the last) as a second ONNX output, for phase E. -- **Limits:** under 5 MB and under 10 ms per reading on a Raspberry Pi 4. Measure on a real Pi - (CI's QEMU check only shows that it runs). -- **Release:** new id `wheels-v2`, the benchmark gate (§3), and a model card with the data and its licences. - -## 8. Phase E — learning per sensor - -- **What already exists:** the review queue asks "read correctly / misread, what was it?", and the - answer is stored with the frame and the region it was read in (`prediction.read_ok`, - `correct_value`, `probs.roi`). The reader does not learn from it today. -- **Design (like object sensors' taught boxes):** - - From every reading checked by hand, the wheels at rest give exact labels (digit d at d.0), and - the last wheel gives a ±0.5 interval. - - Their embeddings (v2's second output) are kept per sensor. - - A new wheel's position probabilities are mixed with a nearest-neighbour vote among that - sensor's own wheels. The weight grows with the number of examples and with how well it did on - the sensor's own checked readings. - - The base model is never changed. The adaptation can be dropped and rebuilt at any time, and - it is part of the sensor's data (backup and export). -- **Safety net:** a leave-one-out check on the sensor's checked readings. The adaptation is used - only when it beats the base model there. Otherwise it is off, and the Quality tab says so - ("adapted from 23 checked readings · 21 → 23 right"). -- **Asking at the right moment:** a reading where the base model and the adaptation disagree, or - the confidence is low (A2), goes to review as "Read as 632.59, not sure — right?". That is the - same queue and the same answers, used only for that sensor. -- **No self-training** on the sensor's own unchecked readings (principle 3). - -## 9. Order, size and dependencies - -| Phase | Builds on | Size | Gives | Status | +| MR-AMR (Mendeley, 140,000 digits of Moroccan water and electricity meters, whole and half-turned) | CC BY-NC 3.0 | not usable, though the closest match | +| Dig21000 (IEEE DataPort, 21,000 rotary meter digits, partly from Roboflow) | no licence stated; behind a subscription | not usable | +| NRC-GAMMA (28,883 gas meter images, Ottawa) | CC BY 4.0 | not usable: pointer dials of one meter model, not wheels | +| Photos on Openverse (Flickr, Wikimedia Commons) of water, gas and electricity meters | CC0, public domain or CC BY per photo | **possible as a test set of meters the model has not seen**: hundreds of photos per search term, of which only some show a legible counter; each picked and labelled by eye, CC BY credited in the model card, only CC0 into the repository | + +## 7. Tried and dropped + +- **Tuning the region by the model's own likelihood** (the region's edges moved until the last K + frames decode most likely). On the maintainer's meter it raised the too-high readings from 9 to + 30. A model is often most sure on a crop it reads wrong, so its likelihood cannot steer the + geometry. The context model makes the geometry matter less instead. +- **Fitting each export stretch's region to its labels** (by the model's likelihood of them): it + made the maintainer's 14 Oct worse. Exports now say where the region was instead. +- **Training on a meter's frames only as they were cut**: the model learns the framing (§4.1); + frames are always also cut with the region moved. +- **Weight averaging (EMA)** for the context net: no steadier. +- **Averaging shifted crops** at read time: no gain on most sets, 3–8× the work. +- **A wider model, and drawn off-centre cells without context**: no gain. +- **Decoding over the time series** (§4.2, step 2; `temporal.py`): no gain for the pair; the gain + it showed for wheels-v1 came from tuning its cost on the same frames it was measured on. +- **Weakest-wheel confidence for wheels-v1 on its own:** it works (§1), but it comes with + the next model instead of as a separate setting for v1 (the maintainer's call, 2026-10-23). + +## 8. Later, when a case asks for it + +- **The last value as an expectation** is now step 2 of §4.2. What stays here: a reading far + outside it that the image explains clearly better is not published but sent to Review. It would + have stopped 734.156 (+36 steps in four minutes). Risks for both: a wrong last value that locks + the prior in, and a meter that was replaced; both need tests that replay sequences. +- **One more decimal from the last wheel's fraction**: a smoother rate, earlier leak detection; + off by default. +- **Camera drift:** a picture that moved at the same size (phase correlation against a reference + frame, numpy FFT). A picture of another size is already caught (readings held, "camera + changed", `SCOPE.md` §5). +- **Four-point perspective correction** for strongly angled cameras: waits for a real angled meter. +- **Learning per sensor** from checked readings (nearest neighbours on an embedding output): the + data so far shows a meter's own frames from another period help little. It waits until shared + data says otherwise. +- **Refuse a frame that is too dark or flat** before reading it, instead of stretching noise into + digits. +- **Publish the last wheel's fraction** (one decimal more, e.g. 6.5 litres): the readers already + give it (`details["wheels"]["value"]`, within 0.1 of a wheel on drawn counters) but the value is + rounded. Measure it on real meters first (tenths read by eye on a sample of frames); an app + setting, no new model. + +## 9. Order and status + +| Step | Builds on | Size | Gives | Status | |---|---|---|---|---| -| 0 Benchmark | — | small | a fair measure for everything after | planned | -| A1 Expectation | 0 | medium | far fewer too-high and unclear readings | planned | -| A2 Calibration | 0 | small | a threshold that means what it says | planned | -| A3 Extra decimal | — | small | a smoother rate, earlier leak detection | planned | -| B1 Export v2 | — | small–medium | the data we need, labelled properly | **done, 0.8.0** | -| B2 Collecting | B1 | ongoing | more meter types | next (issue forms point to Discussions) | -| C2 Calibration per sensor | 0 | medium | a stable last digit, forgiving regions | planned | -| C3 Camera drift | C2 | small–medium | sensors survive small camera moves | when a real case asks for it | -| C4 4-point perspective | real angle tests | medium | angled cameras | waits for a test with a real angled meter | -| D wheels-v2 | 0, B (data), C1 | large | a better base model | planned | -| E Learning per sensor | D (embedding), A2 | medium | the last step for an unusual meter | planned | - -Update the status column in the same change that builds a phase. - -Suggested order: **0 → A1 + A2 (B2 alongside) → A3 → C2 → D → E**, -with C3 and C4 when real cases ask for them. A and C2 need no new training and help every existing -counter sensor right away. +| Export: where the region is in the image (`"box"`) | — | small | correct cells from every export | **done** | +| Benchmark, labels, replay, geometry cases (§3) | — | small | a fair measure | **done** (`tools/wheelreader`) | +| Counters name their wheel reader; context and confidence kinds in the app (§4.3) | — | medium | a new model without changing existing counters | **done** (0.10.1b1) | +| wheels-v2 pair, leaving one meter out (§4.1) | benchmark | medium | robust to regions that do not fit | **done**; its distilled successor is `wheels-v2` | +| New exports from 0.8.0 on, maintainer and pedromfa (§4.2) | 0.10.1b1 | small | correct cells, fresh frames to test on | **done**; test labels read by eye (§4.1) | +| Measurement: five runs, paired comparison, repeats once (§4.2, 1) | benchmark | small | decisions that are not noise | **done** | +| Decoder over the time series (§4.2, 2) | measurement | medium | fewer wrong published values, for v1 too | **dropped** (§7) | +| Ensemble of training runs (§4.1) | measurement | small | no wrong published value on any set | **done**, too slow to ship as it is | +| Distillation of the ensemble into one pair (§4.2, 4) | ensemble | medium | the ensemble's results at the app's cost | **done**: `wheels-v2`, the default for new counters (§4.1) | +| Labels, loss, calibration; cylinder wheels (§4.2, 3, 5) | measurement | medium | the next model | planned | +| Ask for shared meters (§6) | export | ongoing | the data everything else is limited by | next | +| The rest of §8 | real cases | — | — | when a case asks for it | + +Update the status column in the same change that builds a step. ## 10. Risks -- **Prior lock-in (A1):** mitigated by publishing only debounced values, releasing the prior after - N agreeing disagreements, and showing it. It must be tested on sequences with a deliberately wrong start. -- **Calibration on little data (A2):** T fitted on a few hundred readings may not carry over to - other meters. Refit for each new benchmark and report the ECE per set. -- **Overfitting to the few meters we have (D):** split by meter, keep unseen meters in the gate, - and prefer drawn variety over more images of the same meter. -- **Self-calibration fooled by a frame without digits (C2/C3)** (night, a hand in front): use only - frames with a confident reading, and require an improvement over many frames. -- **Per-sensor adaptation that learns a mistake (E):** human labels only, the leave-one-out gate - and an off switch. +- **Overfitting to the few meters we have:** split by meter, keep unseen meters in the gate, + prefer drawn variety over more images of the same meter, and three training runs per decision. +- **Labels from sequences:** an accepted misread on the chain widens intervals around it. Wrong + labels are found by the label check (§5). +- **A slower model on small hardware:** the v2 pair does about three times v1's work; measure on + a real Pi 4 before release. +- **Context from outside the region** can bring in something that moves (a hand, a shadow): drawn + rows and real exports cover it; watch for it in the label check. ## 11. References - Guo, Pleiss, Sun, Weinberger: *On Calibration of Modern Neural Networks*, ICML 2017 — [arXiv:1706.04599](https://arxiv.org/abs/1706.04599). Temperature scaling and ECE. -- Minderer et al.: *Revisiting the Calibration of Modern Neural Networks*, 2021 — - [arXiv:2106.07998](https://arxiv.org/pdf/2106.07998). Calibration under distribution shift. - Geifman, El-Yaniv: *SelectiveNet: A Deep Neural Network with an Integrated Reject Option*, ICML 2019 — [arXiv:1901.09192](https://arxiv.org/abs/1901.09192). Risk–coverage and reject options. +- Hinton, Vinyals, Dean: *Distilling the Knowledge in a Neural Network*, 2015 — + [arXiv:1503.02531](https://arxiv.org/abs/1503.02531). Teachers into one student. +- Kim, Chung, Jang: *Calibration of Ordinal Regression Networks*, 2024 — + [arXiv:2410.15658](https://arxiv.org/abs/2410.15658). Soft ordinal targets and calibration. +- Wang et al.: *Self-Tuning for Data-Efficient Deep Learning*, ICML 2021 — + [arXiv:2102.12903](https://arxiv.org/abs/2102.12903). Pretraining and fine-tuning drift with + few labels across domains. - Self-training under distribution shift and confirmation bias: [arXiv:2411.00586](https://arxiv.org/html/2411.00586v1), [arXiv:2303.10856](https://arxiv.org/pdf/2303.10856). - Zhao et al.: *A Comprehensive Dataset for Word-Wheel Water Meter Reading Under Challenging Conditions*, Scientific Data 2026 — [PMC13031774](https://pmc.ncbi.nlm.nih.gov/articles/PMC13031774/), - data [doi:10.5061/dryad.7d7wm3860](https://doi.org/10.5061/dryad.7d7wm3860) (CC0). It labels - half-turned wheels but reports no separate results for them (DenseNet 98.15 % overall). + data [doi:10.5061/dryad.7d7wm3860](https://doi.org/10.5061/dryad.7d7wm3860) (CC0). - AI-on-the-edge-device documentation (ideas only): - [parameters](https://jomjol.github.io/AI-on-the-edge-device-docs/Parameters/) (PreValue, - MaxRateValue, ChangeRateThreshold, ExtendedResolution), + [parameters](https://jomjol.github.io/AI-on-the-edge-device-docs/Parameters/), [correction algorithm](https://jomjol.github.io/AI-on-the-edge-device-docs/Correction%20Algorithm/), [alignment with two reference marks](https://jomjol.github.io/AI-on-the-edge-device-docs/Alignment/). - Phase correlation for image shifts: - [scikit-image registration example](https://scikit-image.org/docs/0.25.x/auto_examples/registration/plot_register_translation.html) - (the method; we would write it with numpy FFT). + [scikit-image registration example](https://scikit-image.org/docs/0.25.x/auto_examples/registration/plot_register_translation.html). - Dataset licences: [Dataset Ninja: Water Meters](https://datasetninja.com/water-meters) (CC BY-NC-ND 4.0), [UniDataPro/water-meters](https://huggingface.co/datasets/UniDataPro/water-meters), [Roboflow: water meter digits reading](https://universe.roboflow.com/ai-8rgrs/water-meter-digits-reading) (CC BY 4.0). diff --git a/tools/wheelreader/README.md b/tools/wheelreader/README.md index ed72861..656bec6 100644 --- a/tools/wheelreader/README.md +++ b/tools/wheelreader/README.md @@ -1,6 +1,6 @@ # Wheel reader -The model VisionState reads mechanical counters (water and gas meters) with: a small CNN that +The models VisionState reads mechanical counters (water and gas meters) with: a small CNN that looks at one wheel and tells how far it has turned — a distribution over 100 positions around the wheel (0.0, 0.1, … 9.9). The app (`visionstate/backend/visionstate/readers.py`, `WheelReader`) splits the region into one cell per wheel and finds the most likely value whose @@ -9,14 +9,17 @@ wheel positions fit together (a wheel only turns while the wheel to its right go This folder holds everything needed to make the model again. Nothing here is part of the app image; the data never goes into the repository (it lives in `VisionStateLocal/`, ignored by git). -## The released model +## The released models -| Id | File | Size | Input | Output | +| Id | File | Size | Inputs | Outputs | |---|---|---|---|---| | `wheels-v1` | `wheels-v1.onnx` | 2.2 MB, 565 k parameters | `cells`: N × 1 × 48 × 32, grey 0…1 | `logits`: N × 100 | +| `wheels-v2` | `wheels-v2.onnx` | 6.1 MB, 1.5 M parameters | `cells` as v1; `context`: N × 1 × 48 × 64, each cell with half a cell of the frame on each side | `logits`: N × 100 (log probabilities of both nets, averaged); `embedding`: N × 512 | -Cell preprocessing (must match `readers.wheel_cells`): grey from each pixel's darkest colour -channel, resized to 32 × 48 (bilinear), autocontrast with 1 % cut-off, divided by 255. +Cell preprocessing (must match `readers.wheel_cells`, and `readers.wheel_context_cells` for the +context): grey from each pixel's darkest colour channel, resized to 32 × 48 (64 × 48 with +context, bilinear), autocontrast with 1 % cut-off, divided by 255. wheels-v2 is a one-cell net +and a context net in one file (`export_v2.py`); its confidence is the weakest wheel's. Licence: Apache-2.0. Trained on data that may be used for anything: @@ -25,6 +28,7 @@ Licence: Apache-2.0. Trained on data that may be used for anything: | Drawn wheels (`synth.py`, 600 000 cells): fonts from Google Fonts (`fonts.py`, pinned) | SIL OFL 1.1 (the fonts; drawn images are not covered) | training | | [Word-Wheel Water Meter Dataset](https://doi.org/10.5061/dryad.7d7wm3860) (Sci Data 2026), recognition crops | CC0 | training (labels: the nearest digit per wheel) | | Checked readings exported from VisionState and shared in issues [#32](https://github.com/oleost/VisionState/issues/32) and [#40](https://github.com/oleost/VisionState/issues/40) | CC0 (the export's LICENSE.txt) | training; `wheels-v1` was evaluated with #40 0001–0017 held out, then trained on all of it | +| wheels-v2 also: drawn rows of wheels (`rows.py`), and the exports of two water meters of 2026-10-23 (the maintainer's and pedromfa's, #40), 0.8.0 on | OFL fonts; CC0 | training; evaluated leaving each meter out, then trained on both | Results of `wheels-v1` (2026-10-09; whole reading exact, last digit rounded): @@ -70,6 +74,53 @@ $PY evaluate.py ../../VisionStateLocal/wheel/runs/wheels.pt --width 16 --shift 2 `items.py` also knows local test-only sets that are not redistributable (GPL), which are skipped when they are missing. +Results of `wheels-v2` (2026-10-24): leaving each meter out, five training runs against +`wheels-v1` on the same frames, about 1,300 frames read by eye, through the app's rules (70 %, +never down): wrong values published 0 in 19 of 20 runs and sets (one: 2 litres), wheels-v1 3, 2, +1 and 0; frames right +4.4 and +1.4 points on the maintainer's meter (significant), +0.6 and +0.1 +on pedromfa's. Trained on all of it: 28 of 33 free Openverse photos (wheels-v1 26), Dryad six +digits 94.0 % (90.2 %). Weaker: it publishes less often on a meter seen from the side whose last +wheel turns. Details: the plan, §4.1. + +### Making wheels-v2 again + +After the steps above, and `$PY rows.py build 300000`: + +``` +$PY exports.py olee 2026-10-18 8 # labels read by eye (frames.json in the export) +$PY exports.py pede 2026-10-20 8 +CELL="synth:1 rows:0.5 dryad_train:1 pede_a:0.03 pede_b:0.03 pede_a_moved:0.08 pede_b_moved:0.08 olee_a:0.03 olee_b:0.03 olee_a_moved:0.08 olee_b_moved:0.08" +CTX="rows_ctx:2 dryad_train_ctx:1 pede_a_ctx:0.03 pede_b_ctx:0.03 pede_a_moved_ctx:0.08 pede_b_moved_ctx:0.08 olee_a_ctx:0.03 olee_b_ctx:0.03 olee_a_moved_ctx:0.08 olee_b_moved_ctx:0.08" +# teachers: pairs trained as below without --teacher, each on one meter only, several --seed +$PY distill.py t16 synth rows dryad_train pede_a pede_b pede_a_moved pede_b_moved olee_a olee_b olee_a_moved olee_b_moved +O="--epochs 25 --width 16 --dryad-orient ../../VisionStateLocal/wheel/runs/s1_synth.pt --teacher t16 --teacher-t 0.3 --alpha 0.7" +$PY train.py final_cell $CELL $O +$PY train.py final_ctx $CTX $O --input 64 --ema 0.999 +$PY export_v2.py runs/final_cell.pt runs/final_ctx.pt wheels-v2.onnx +``` + +The released teacher was 16 pairs (5 seeds per meter trained before the frames were read by eye, +3 after). + +## Candidates and the benchmark + +The next model (a one-cell net and a context net read together, `context.py`) is built and +measured with these; each script's docstring says how to run it: + +| Script | Does | +|---|---| +| `rows.py` | drawn rows of wheels, so a cell can sit off-centre with its neighbour in view | +| `exports.py` | labels and cells from a shared export (only readings with `"box"`); `eye.json` and `frames.json` in the export folder hold labels read by eye | +| `bench.py` | the benchmark: `eval` (with a replay through *went down* and the change limit), `geometry` (the region's edges moved) and `compare` (paired, with a bootstrap interval); a model spec of several `.pt` files joined with `+` is an ensemble | +| `distill.py` | an ensemble's probabilities on the training sets, for `train.py --teacher` | +| `temporal.py` | decoding over the time series instead of frame by frame | +| `fit_boxes.py` | regions fitted to the labels (dropped: the export says where the region was) | +| `export_v2.py` | the pair as one ONNX file with a second output, the embedding | + +A candidate is measured **leaving one meter out**: trained without the meter it is tested on, +several training runs (`--seed`), each compared with wheels-v1 on the same frames. Results and +what was tried are in the plan. + What is planned next (benchmark, wheels-v2 and more): [`docs/WHEEL_READER_PLAN.md`](../../docs/WHEEL_READER_PLAN.md). A new model is a new registry entry (`readers.json`, `wheel_readers`) with a new id, hosted on diff --git a/tools/wheelreader/bench.py b/tools/wheelreader/bench.py new file mode 100644 index 0000000..5821b1e --- /dev/null +++ b/tools/wheelreader/bench.py @@ -0,0 +1,355 @@ +"""The wheel reader benchmark (`docs/WHEEL_READER_PLAN.md` §3). + +python bench.py build bench/ sets of pedromfa #32/#40, bolausson, Dryad test +python bench.py eval [--sets a,b] [--tag name] [--sweep] +python bench.py geometry the same frames with the region's edges moved +python bench.py compare [ ...] [--sets a,b] + frame by frame against a base (results of eval --tag) + +Export sets come from exports.py (bench/_a.npz, _b.npz, each with a _ctx twin). : +runs/.pt (train.py; --input 64 reads the _ctx twins), ".pt+.pt" (both, as +wheels-v2), or an .onnx file (wheels-v1: one input; wheels-v2: "cells" and "context"). + +Decoding is the app's (readers.WheelReader.read_counter): slack 1 bin, common shift +-2 bins +(penalty 0.3 per bin). Confidence: "mean" (geometric mean of the decoded bins, wheels-v1) and +"wmin" (the weakest wheel's mass within +-0.4 of where the value puts it, wheels-v2). +Metrics: right (+-1 on the last wheel), exact, too high; risk-coverage (AURC, share accepted at +1 % wrong); for export sets a replay through "went down" and the change limit at 0.70 (--sweep: also +at other limits, for a model whose confidence is spread differently), and "ok" +and "too high" with every run of frames with the same label (a still meter) counted once. + +compare: the tags after the base are training runs of one candidate. Per set, the difference in +"ok" and "too high" per run of frames (each run counted once), averaged over the candidate's +training runs, with a 95 % interval from resampling the runs of frames (bootstrap). An interval +that holds 0 is no difference. +""" + +import argparse +import json +import math +from pathlib import Path + +import numpy as np +from PIL import Image + +from common import BINS, cell_input, positions, split_cells +from context import ctx_cells +from paths import WORK + +BENCH = WORK / "bench" +TENTH = BINS // 10 + + +def wheel_decode(logp): + """readers.wheel_decode.""" + wheels, bins = logp.shape + best = logp[-1].copy() + back = [] + for k in range(wheels - 2, -1, -1): + rest = int(best[: 9 * TENTH + 1].argmax()) + lead = np.concatenate([[best[rest]], best[9 * TENTH + 1 :]]) + lead_bin = np.concatenate([[rest], np.arange(9 * TENTH + 1, bins)]) + best = (logp[k].reshape(10, TENTH) + lead[None, :]).reshape(bins) + back.append(np.tile(lead_bin, 10)) + b = int(best.argmax()) + score = float(best[b]) + at = [b] + for arg in reversed(back): + b = int(arg[b]) + at.append(b) + value = 0.0 + for a in at[:-1]: + value = value * 10 + a // TENTH + return value * 10 + at[-1] / TENTH, score, at + + +def read(probs, slack=1, max_shift=2, penalty=0.3): + """Value (last wheel rounded), mean and weakest-wheel confidence, log probability.""" + near = sum(np.roll(probs, s, axis=1) for s in range(-slack, slack + 1)) + logp = np.log(np.maximum(near, 1e-12)) + best = None + for shift in range(-max_shift, max_shift + 1): + value, score, at = wheel_decode(np.roll(logp, -shift, axis=1)) + score -= penalty * abs(shift) + if best is None or score > best[1]: + best = (value, score, at, shift) + value, score, at, shift = best + digits = probs.shape[0] + seen = [(b + shift) % BINS for b in at] + mass = [sum(probs[k, (b + d) % BINS] for d in range(-4, 5)) for k, b in enumerate(seen)] + return math.floor(value + 0.5) % 10**digits, math.exp(score / digits), float(min(mass)), score + + +def softmax(logits): + logits = logits - logits.max(axis=1, keepdims=True) + p = np.exp(logits) + return p / p.sum(axis=1, keepdims=True) + + +class Model: + """probs(cells, context) -> N x BINS for a .pt, a "one-cell.pt+context.pt" pair or an .onnx. + + "@" divides the log probabilities by a temperature t (below 1: surer). + """ + + def __init__(self, spec: str): + self.parts = [] + spec, _, t = spec.partition("@") + self.t = float(t or 1) + if spec.endswith(".onnx"): + import onnxruntime as ort + + self.onnx = ort.InferenceSession(spec, providers=["CPUExecutionProvider"]) + self.inputs = [i.name for i in self.onnx.get_inputs()] + else: + from train import load_model + + self.onnx = None + self.parts = [load_model(p, 16) for p in spec.split("+")] + + def probs(self, cells, context): + if self.onnx is not None: + feeds = [cells, context][: len(self.inputs)] + out = [] + for i in range(0, len(cells), 2048): + feed = {n: x[i : i + 2048].astype(np.float32)[:, None] / 255 for n, x in zip(self.inputs, feeds)} + out.append(self.onnx.run(None, feed)[0]) + return softmax(np.concatenate(out).astype(np.float64) / self.t) + from train import predict + + logp = sum(predict(m, context if _is_ctx(m) else cells) for m in self.parts) + return softmax(logp.astype(np.float64) / len(self.parts) / self.t) + + +def _is_ctx(model) -> bool: + """Whether a train.py model reads cells with context (its head is twice as wide).""" + width = model.features[0][0].out_channels + return model.head[2].in_features == 4 * width * 6 * 8 + + +def simulate(rows, conf_i, t): + """Replay through "went down" + the change limit while it learns (40 steps a minute + 2).""" + rows = sorted(rows, key=lambda r: r["seconds"]) + last, last_t = rows[0]["lo"], rows[0]["seconds"] + pub = wrong = high = 0 + stale = 0.0 + prev = rows[0]["seconds"] + for r in rows[1:]: + if last > r["hi"] + 1: + stale += r["seconds"] - prev + prev = r["seconds"] + v, conf = r["v"], r["conf"][conf_i] + if conf < t or v < last or v - last > 2 + 40 * (r["seconds"] - last_t) / 60: + continue + pub += 1 + if not (r["lo"] - 1 <= v <= r["hi"] + 1): + wrong += 1 + high += v > r["hi"] + 1 + last, last_t = v, r["seconds"] + return pub, wrong, high, stale / 3600 + + +def evaluate(model: Model, sets: list[str], sweep: bool = False) -> tuple[dict, dict]: + """Per set: the metrics, and per frame ok / too high / weakest-wheel confidence / run.""" + results, frames = {}, {} + for name in sets: + d, c = np.load(BENCH / f"{name}.npz"), np.load(BENCH / f"{name}_ctx.npz") + n, digits = d["X"].shape[:2] + P = model.probs(d["X"].reshape(-1, *d["X"].shape[2:]), c["X"].reshape(-1, *c["X"].shape[2:])).reshape(n, digits, BINS) + rows = [] + for i in range(n): + v, mean, wmin, score = read(P[i]) + rows.append({"v": v, "conf": (mean, wmin), "lo": int(d["lo"][i]), "hi": int(d["hi"][i]), + "seconds": int(d["seconds"][i]), "group": int(d["group"][i])}) + if name.startswith("dryad"): # both orientations of every photo: keep the one the label fits + for i, r in enumerate(rows): + pos = positions(float(r["lo"]), digits) + r["fit"] = sum(math.log(max(1e-12, sum(P[i, k, (round(p * 10) + o) % BINS] for o in range(-5, 6)))) for k, p in enumerate(pos)) + best = {} + for r in rows: + if r["group"] not in best or r["fit"] > best[r["group"]]["fit"]: + best[r["group"]] = r + rows = list(best.values()) + frames[name] = { + "ok": np.array([r["lo"] - 1 <= r["v"] <= r["hi"] + 1 for r in rows]), + "high": np.array([r["v"] > r["hi"] + 1 for r in rows]), + "wmin": np.array([r["conf"][1] for r in rows]), + "group": np.array([r["group"] for r in rows]), + } + res = {"n": len(rows)} + res["exact"] = sum(r["lo"] <= r["v"] <= r["hi"] for r in rows) + res["ok"] = sum(r["lo"] - 1 <= r["v"] <= r["hi"] + 1 for r in rows) + res["high"] = sum(r["v"] > r["hi"] + 1 for r in rows) + sequence = name.endswith(("_a", "_b", "_a_eye", "_b_eye")) + if sequence: # every run of frames with the same label once + f = frames[name] + runs, inv = np.unique(f["group"], return_inverse=True) + w = 1 / np.bincount(inv)[inv] + res["runs"] = len(runs) + res["ok_runs"] = float((f["ok"] * w).sum()) + res["high_runs"] = float((f["high"] * w).sum()) + for ci, cname in ((0, "mean"), (1, "wmin")): + srt = sorted(rows, key=lambda r: -r["conf"][ci]) + risk = np.cumsum([not (r["lo"] - 1 <= r["v"] <= r["hi"] + 1) for r in srt]) / np.arange(1, len(srt) + 1) + res[f"aurc_{cname}"] = float(risk.mean() * 100) + res[f"cov1_{cname}"] = max([k + 1 for k in range(len(srt)) if risk[k] <= 0.01] or [0]) / len(srt) + if sequence: + res[f"replay_{cname}"] = simulate(rows, ci, 0.7) + results[name] = res + line = (f"{name:10s} n={res['n']:5d} exact {res['exact'] / res['n']:6.1%} ok {res['ok'] / res['n']:6.1%} " + f"high {res['high']:4d} AURC mean {res['aurc_mean']:5.2f} wmin {res['aurc_wmin']:5.2f} " + f"cov@1% wmin {res['cov1_wmin']:.2f}") + if sequence: + p, w, _h, s = res["replay_wmin"] + line += (f" | runs {res['runs']}: ok {res['ok_runs'] / res['runs']:6.1%} high {res['high_runs']:5.1f}" + f" | replay@0.7 wmin: {p} published, {w} wrong, {s:.1f} h too high") + print(line, flush=True) + if sequence and sweep: + print(" " + " ".join(f"@{t:.2f}: {p} pub {w} wrong" for t in (0.4, 0.5, 0.6, 0.8) + for p, w, _h, _s in [simulate(rows, 1, t)]), flush=True) + return results, frames + + +def compare(base: str, tags: list[str], sets: list[str] | None, rounds: int = 2000) -> None: + load = lambda tag: np.load(WORK / "results" / f"{tag}.npz") # noqa: E731 + b, cands = load(base), [load(t) for t in tags] + names = sorted({k.split("|")[0] for k in b.files}) + rng = np.random.default_rng(0) + print(f"{', '.join(tags)} against {base}: difference per run of frames (95 % interval)") + for name in sets or names: + if f"{name}|ok" not in b.files or any(f"{name}|ok" not in c.files for c in cands): + continue + group = b[f"{name}|group"] + runs, inv = np.unique(group, return_inverse=True) + w = 1 / np.bincount(inv)[inv] + out = [] + for metric in ("ok", "high"): + diff = np.mean([c[f"{name}|{metric}"].astype(float) for c in cands], axis=0) - b[f"{name}|{metric}"] + per_run = np.bincount(inv, weights=diff * w) # each run's share, summing to its difference + total = per_run.sum() + boot = per_run[rng.integers(0, len(runs), (rounds, len(runs)))].sum(axis=1) + lo, hi = np.percentile(boot, [2.5, 97.5]) + scale = 100 / len(runs) if metric == "ok" else 1 + unit = " pts" if metric == "ok" else "" + out.append(f"{metric} {total * scale:+6.2f}{unit} [{lo * scale:+.2f}, {hi * scale:+.2f}]") + print(f" {name:10s} {len(runs):4d} runs " + " ".join(out)) + + +def save(name, cells, context, lo, hi, digits): + for suffix, X in (("", cells), ("_ctx", context)): + np.savez_compressed(BENCH / f"{name}{suffix}.npz", X=np.stack(X), lo=np.array(lo, np.int64), hi=np.array(hi, np.int64), + seconds=np.arange(len(X)), group=np.array(_groups(name, len(X))), digits=digits) + print(name, len(cells)) + + +def _groups(name, n): + return [k // 2 for k in range(n)] if name.startswith("dryad") else list(range(n)) + + +def one_cell(region: Image.Image, digits: int) -> np.ndarray: + return np.stack([(cell_input(c) * 255).astype(np.uint8) for c in split_cells(region, digits)]) + + +def build() -> None: + from items import SETS, pedromfa + + BENCH.mkdir(exist_ok=True) + pedro = {"pedro40": ("pedromfa-watermeter-issue40", set(range(1, 18))), "pedro_esp": ("pedromfa-watermeter-esp", None), + "pedro32": ("pedromfa-watermeter", None)} + for name, (folder, only) in pedro.items(): + cells, context, lo = [], [], [] + for _id, nd, truth, (image, box) in pedromfa(folder, only, whole=True): + cells.append(one_cell(image.crop(tuple(round(v) for v in box)), nd)) + context.append(ctx_cells(image, box, nd)) + lo.append(round(truth)) + save(name, cells, context, lo, lo, 7) + for name in ("bol_cold", "bol_warm"): # GPL-3.0: test only, skipped when missing + try: + items = list(SETS[name]()) + except FileNotFoundError: + print(name, "skipped") + continue + cells, context, lo, hi = [], [], [], [] + for _id, nd, truth, region in items: + cells.append(one_cell(region, nd)) + context.append(ctx_cells(region, (0, 0, region.width, region.height), nd)) + frac = truth - math.floor(truth) # the tenths read by eye: near half way either neighbour is fair + lo.append(round(truth) if abs(frac - 0.5) > 0.2 else math.floor(truth)) + hi.append(round(truth) if abs(frac - 0.5) > 0.2 else math.floor(truth) + 1) + save(name, cells, context, lo, hi, 7) + for nd in (5, 6): # both orientations (a third of the photos is upside down) + cells, context, lo = [], [], [] + for _id, d, truth, region in (r for r in SETS["dryad_test"]() if r[1] == nd): + for flip in (0, 1): + im = region.rotate(180) if flip else region + cells.append(one_cell(im, nd)) + context.append(ctx_cells(im, (0, 0, im.width, im.height), nd)) + lo.append(round(truth)) + save(f"dryad{nd}", cells, context, lo, lo, nd) + + +GEOMETRY = { # the region's left, right (in cells), top, bottom (share of its height) moved by + "as drawn": (0, 0, 0, 0), + "shift right 0.25 cell": (0.25, 0.25, 0, 0), + "shift left 0.25 cell": (-0.25, -0.25, 0, 0), + "wider 0.5 cell each side": (-0.5, 0.5, 0, 0), + "narrower 0.3 cell each side": (0.3, -0.3, 0, 0), + "right edge in 0.5 cell": (0, -0.5, 0, 0), + "up 10 %": (0, 0, -0.1, -0.1), + "down 10 %": (0, 0, 0.1, 0.1), + "taller 10 % each side": (0, 0, -0.1, 0.1), + "shorter 10 % each side": (0, 0, 0.1, -0.1), +} + + +def geometry(model: Model, folder: Path, every: int = 3) -> None: + import exports + + data = json.loads((folder / "readings.json").read_text(encoding="utf-8")) + readings, digits = data["readings"], int(data["settings"]["digits"]) + labels = exports.labels(readings, int(data["settings"]["decimals"])) + frames = [] + for r, lab in list(zip(readings, labels, strict=True))[::every]: + if lab["lo"] is not None and r.get("region") == "as read" and r.get("box"): + frames.append((Image.open(folder / r["file"]).convert("RGB"), r["box"], lab)) + print(f"{folder.name}: {len(frames)} frames") + for case, (left, right, top, bottom) in GEOMETRY.items(): + cells, context = [], [] + for im, (x0, y0, x1, y1), _lab in frames: + cw, h = (x1 - x0) / digits, y1 - y0 + box = (x0 + left * cw, max(0, y0 + top * h), x1 + right * cw, min(im.height, y1 + bottom * h)) + region = im.crop((round(max(0, box[0])), round(box[1]), round(min(im.width, box[2])), round(box[3]))) + cells.append(one_cell(region, digits)) + context.append(ctx_cells(im, box, digits)) + X, C = np.stack(cells), np.stack(context) + P = model.probs(X.reshape(-1, *X.shape[2:]), C.reshape(-1, *C.shape[2:])).reshape(len(X), digits, BINS) + ok = high = 0 + for (_im, _box, lab), p in zip(frames, P, strict=True): + v = read(p)[0] + ok += lab["lo"] - 1 <= v <= lab["hi"] + 1 + high += v > lab["hi"] + 1 + print(f" {case:28s} ok {ok / len(frames):6.1%} too high {high}", flush=True) + + +if __name__ == "__main__": + ap = argparse.ArgumentParser() + ap.add_argument("what", choices=["build", "eval", "geometry", "compare"]) + ap.add_argument("model", nargs="?") + ap.add_argument("folder", nargs="*") + ap.add_argument("--sets") + ap.add_argument("--tag") + ap.add_argument("--sweep", action="store_true") + a = ap.parse_args() + if a.what == "build": + build() + elif a.what == "geometry": + geometry(Model(a.model), Path(a.folder[0])) + elif a.what == "compare": + compare(a.model, a.folder, a.sets.split(",") if a.sets else None) + else: + names = a.sets.split(",") if a.sets else sorted(p.stem for p in BENCH.glob("*.npz") if not p.stem.endswith("_ctx")) + res, frames = evaluate(Model(a.model), names, a.sweep) + if a.tag: + (WORK / "results").mkdir(exist_ok=True) + (WORK / "results" / f"{a.tag}.json").write_text(json.dumps(res, indent=1)) + np.savez_compressed(WORK / "results" / f"{a.tag}.npz", **{f"{n}|{k}": v for n, f in frames.items() for k, v in f.items()}) diff --git a/tools/wheelreader/common.py b/tools/wheelreader/common.py index 670c973..f5eeac9 100644 --- a/tools/wheelreader/common.py +++ b/tools/wheelreader/common.py @@ -28,7 +28,11 @@ def split_cells(region: Image.Image, digits: int) -> list[Image.Image]: MARGIN = 0.15 # export margin around the region (settings.READING["export_margin"]) -def export_region(image: Image.Image) -> Image.Image: +def export_region(image: Image.Image, box: list[int] | None = None) -> Image.Image: + """The region read, out of an exported image: ``box`` from readings.json (format 4), or the + middle without the margin (older exports; wrong where the margin was cut off at the frame's edge).""" + if box: + return image.crop(tuple(box)) w = image.width / (1 + 2 * MARGIN) h = image.height / (1 + 2 * MARGIN) x = (image.width - w) / 2 diff --git a/tools/wheelreader/context.py b/tools/wheelreader/context.py new file mode 100644 index 0000000..7b199f0 --- /dev/null +++ b/tools/wheelreader/context.py @@ -0,0 +1,112 @@ +"""Cells with context (wheels-v2): each wheel's cell plus CTX of a cell on each side, taken from the +picture around the region where there is some (else the edge repeated), as a CTX_W x CELL_H input. +Must match ``readers.wheel_context_cells`` in the app. + +python context.py dryad data/dryad_train_ctx.npz (both orientations, as data.py dryad) +python context.py pedromfa data/pedro_train_ctx.npz, data/pedro_test_ctx.npz (as data.py pedromfa) +""" + +import math +import sys +from pathlib import Path + +import numpy as np +from PIL import Image, ImageOps + +from common import CELL_H, MARGIN, darkest +from paths import LOCAL, WORK + +CTX = 0.5 +CTX_W = 64 + + +def ctx_input(part: Image.Image) -> np.ndarray: + """A cell with its context (any size) -> uint8 CELL_H x CTX_W.""" + g = part if part.mode == "L" else darkest(part) + g = ImageOps.autocontrast(g.resize((CTX_W, CELL_H), Image.Resampling.BILINEAR), cutoff=1) + return np.asarray(g, dtype=np.uint8) + + +def ctx_cells(image: Image.Image, box, digits: int) -> np.ndarray: + """``box`` (x0, y0, x1, y1) is the region inside ``image``; the cells reach CTX beyond it.""" + x0, y0, x1, y1 = box + cell = (x1 - x0) / digits + pad = int(math.ceil(cell * CTX)) + 2 + g = np.pad(np.asarray(darkest(image)), ((0, 0), (pad, pad)), mode="edge") + wide = Image.fromarray(g) + out = [] + for k in range(digits): + a = x0 + k * cell - CTX * cell + pad + b = x0 + (k + 1) * cell + CTX * cell + pad + out.append(ctx_input(wide.crop((round(a), round(y0), round(b), round(y1))))) + return np.stack(out) + + +def centre_box(image: Image.Image): + """Where the region is in an exported image without a "box" (older exports: the middle).""" + w, h = image.width / (1 + 2 * MARGIN), image.height / (1 + 2 * MARGIN) + x, y = (image.width - w) / 2, (image.height - h) / 2 + return (x, y, x + w, y + h) + + +def save(name: str, xs, mids, halves, **extra) -> None: + np.savez_compressed( + WORK / "data" / f"{name}.npz", + X=np.stack(xs), + mid=np.array(mids, np.float32), + half=np.array(halves, np.float32), + **extra, + ) + print(name, len(xs), "cells") + + +def build_dryad() -> None: + from data import dryad_rows, intervals + + xs, mids, halves, img_id, rot = [], [], [], [], [] + for k, (path, digits, _flags) in enumerate(dryad_rows("train")): + im = Image.open(path).convert("RGB") + for flip in (0, 1): + src = im.rotate(180) if flip else im + for cell, (m, h) in zip(ctx_cells(src, (0, 0, src.width, src.height), len(digits)), intervals(digits), strict=True): + xs.append(cell) + mids.append(m) + halves.append(h) + img_id.append(k) + rot.append(flip) + save("dryad_train_ctx", xs, mids, halves, img=np.array(img_id), rot=np.array(rot)) + + +def build_pedromfa() -> None: + import json + + from data import intervals + + sets = { + "pedro_train_ctx": [(LOCAL / "pedromfa-watermeter", None), (LOCAL / "pedromfa-watermeter-esp", None)], + "pedro_test_ctx": [(LOCAL / "pedromfa-watermeter-issue40", set(range(1, 18)))], + } + for name, sources in sets.items(): + xs, mids, halves = [], [], [] + for folder, only in sources: + data = json.loads((Path(folder) / "readings.json").read_text(encoding="utf-8")) + for r in data["readings"]: + i = int(r["file"][-8:-4]) + if only and i not in only: + continue + if r["answer"] == "unchecked" or (r["answer"] == "misread" and not r["right_value"]): + continue + v = float(r["right_value"] or r["value"]) + if v > 10000: + v /= 1000 # an old export stored "629558" as 629558.000 + im = Image.open(Path(folder) / r["file"]).convert("RGB") + cells = ctx_cells(im, r.get("box") or centre_box(im), 7) + for cell, (m, h) in zip(cells, intervals(f"{round(v * 1000):07d}"), strict=True): + xs.append(cell) + mids.append(m) + halves.append(h) + save(name, xs, mids, halves) + + +if __name__ == "__main__": + {"dryad": build_dryad, "pedromfa": build_pedromfa}[sys.argv[1]]() diff --git a/tools/wheelreader/distill.py b/tools/wheelreader/distill.py new file mode 100644 index 0000000..0159607 --- /dev/null +++ b/tools/wheelreader/distill.py @@ -0,0 +1,38 @@ +"""Teacher targets for distillation (`docs/WHEEL_READER_PLAN.md` §4.2, step 4). + +python distill.py [ ...] + + is an ensemble as bench.Model takes it ("a.pt+b.pt+..."). For every training set, writes +data/_t_.npz {"T": N x BINS float16}: the ensemble's probabilities, the mean log +probability of all its parts that can read the set (one-cell parts on a set without a _ctx twin, +such as synth). A _ctx set uses its one-cell twin's targets, so name the one-cell sets only. +train.py --teacher then trains towards them. +""" + +import argparse + +import numpy as np + +import bench +from train import HERE, load, predict + + +def main() -> None: + ap = argparse.ArgumentParser() + ap.add_argument("tag") + ap.add_argument("model") + ap.add_argument("sets", nargs="+") + a = ap.parse_args() + parts = bench.Model(a.model).parts + for name in a.sets: + cells = load(name)["X"] + ctx_path = HERE / "data" / f"{name}_ctx.npz" + context = np.load(ctx_path)["X"] if ctx_path.exists() else None + use = [m for m in parts if context is not None or not bench._is_ctx(m)] + logp = sum(predict(m, context if bench._is_ctx(m) else cells) for m in use) / len(use) + np.savez(HERE / "data" / f"{name}_t_{a.tag}.npz", T=bench.softmax(logp.astype(np.float64)).astype(np.float16)) + print(name, len(cells), "from", len(use), "parts", flush=True) + + +if __name__ == "__main__": + main() diff --git a/tools/wheelreader/evaluate.py b/tools/wheelreader/evaluate.py index 136c7bd..f074414 100644 --- a/tools/wheelreader/evaluate.py +++ b/tools/wheelreader/evaluate.py @@ -9,12 +9,11 @@ from collections import Counter import numpy as np -import torch from common import cell_input, decode, decode_shifted, split_cells from items import SETS from paths import LOCAL, WORK -from train import DEV, WheelNet, predict +from train import load_model, predict HERE = WORK DEFAULT = ["pedro40", "pedro_esp", "pedro32", "bol_cold", "bol_warm"] @@ -59,8 +58,7 @@ def main(): args = ap.parse_args() global SHIFT SHIFT = args.shift - model = WheelNet(args.width).to(DEV) - model.load_state_dict(torch.load(args.model, map_location=DEV)) + model = load_model(args.model, args.width) model.eval() base = {} for rid in ("ppocrv6-small", "ppocrv6-tiny"): diff --git a/tools/wheelreader/export.py b/tools/wheelreader/export.py index 9947c40..bf117a5 100644 --- a/tools/wheelreader/export.py +++ b/tools/wheelreader/export.py @@ -9,12 +9,10 @@ import torch from common import CELL_H, CELL_W -from train import WheelNet +from train import load_model pt, width = Path(sys.argv[1]), int(sys.argv[2]) -m = WheelNet(width) -m.load_state_dict(torch.load(pt, map_location="cpu")) -m.eval() +m = load_model(pt, width, "cpu") out = pt.with_suffix(".onnx") torch.onnx.export( m, diff --git a/tools/wheelreader/export_v2.py b/tools/wheelreader/export_v2.py new file mode 100644 index 0000000..c26bcde --- /dev/null +++ b/tools/wheelreader/export_v2.py @@ -0,0 +1,72 @@ +"""wheels-v2 as one ONNX file: a one-cell model and a context model read together. + +python export_v2.py runs/.pt runs/.pt (paths under VisionStateLocal/wheel) +Inputs: "cells" (N x 1 x 48 x 32, as wheels-v1) and "context" (N x 1 x 48 x 64: the same cells with +half a cell on each side). Outputs: "logits" (N x 100, the mean of the two models' log +probabilities, so a softmax gives their geometric mean) and "embedding" (N x 512: the layer before +the positions of both, for learning per sensor later). +""" + +import sys +import time +from pathlib import Path + +import numpy as np +import onnxruntime as ort +import torch +import torch.nn as nn +import torch.nn.functional as F + +from paths import WORK +from train import load_model + + +def logits_and_embedding(net, x): + f = net.features(x * 2 - 1) + head = net.head # Flatten, Dropout, Linear, ReLU, Linear + e = head[3](head[2](head[0](f))) + return head[4](e), e + + +class WheelsV2(nn.Module): + def __init__(self, cell, ctx): + super().__init__() + self.cell, self.ctx = cell, ctx + + def forward(self, cells, context): + a, ea = logits_and_embedding(self.cell, cells) + b, eb = logits_and_embedding(self.ctx, context) + return (F.log_softmax(a, dim=1) + F.log_softmax(b, dim=1)) / 2, torch.cat([ea, eb], dim=1) + + +if __name__ == "__main__": + cell, ctx = load_model(WORK / sys.argv[1], device="cpu"), load_model(WORK / sys.argv[2], device="cpu") + out = Path(sys.argv[3]) + m = WheelsV2(cell, ctx).eval() + torch.onnx.export( + m, + (torch.zeros(1, 1, 48, 32), torch.zeros(1, 1, 48, 64)), + str(out), + input_names=["cells", "context"], + output_names=["logits", "embedding"], + dynamic_axes={"cells": {0: "n"}, "context": {0: "n"}, "logits": {0: "n"}, "embedding": {0: "n"}}, + opset_version=17, + dynamo=False, + ) + so = ort.SessionOptions() + so.intra_op_num_threads = 1 + s = ort.InferenceSession(str(out), so, providers=["CPUExecutionProvider"]) + x32, x64 = np.random.rand(8, 1, 48, 32).astype(np.float32), np.random.rand(8, 1, 48, 64).astype(np.float32) + with torch.no_grad(): + ref = m(torch.from_numpy(x32), torch.from_numpy(x64))[0].numpy() + got, emb = s.run(None, {"cells": x32, "context": x64}) + print(out.name, out.stat().st_size, "bytes; max diff", float(np.abs(ref - got).max()), "; embedding", emb.shape) + print("parameters", sum(p.numel() for p in m.parameters())) + feed = {"cells": x32, "context": x64} + for _ in range(5): + s.run(["logits"], feed) + t = time.perf_counter() + n = 300 + for _ in range(n): + s.run(["logits"], feed) + print(f"8 wheels, 1 thread: {(time.perf_counter() - t) / n * 1000:.2f} ms") diff --git a/tools/wheelreader/exports.py b/tools/wheelreader/exports.py new file mode 100644 index 0000000..bc9f933 --- /dev/null +++ b/tools/wheelreader/exports.py @@ -0,0 +1,200 @@ +"""Labels and cells from a shared readings export (wheels-v2; `docs/WHEEL_READER_PLAN.md` §3, §5). + +python exports.py [moved] + +Every reading gets an interval of values [lo, hi] in steps of the last wheel, never from its own +read: a person's answer, else the interval between the trusted readings before and after it in +time (trusted: checked, or accepted on the longest non-decreasing chain of checked and accepted +values; checked ones weigh most). ``eye.json`` in the export folder overrides stretches looked at +by eye: [[first image number, last, lo, hi], ...]; ``frames.json`` there overrides single frames +read by eye: {"": [lo, hi], or null for an image nobody can read}. Every reading helps the labels; only those read +with the region they were cut with and whose export says where it is (``"region": "as read"`` +with ``"box"``, VisionState 0.10.1b1 on) become frames: a guessed region teaches the wrong cells. + +Writes, for part a (dates up to the split date) and part b (after it): + bench/_.npz, bench/__ctx.npz readings x wheels cells, lo, hi, seconds, + group (a run of frames in a row with the + same label: a still meter, counted once) + data/_.npz, data/__ctx.npz training cells with an interval per wheel + data/__moved.npz, ..._moved_ctx.npz with [moved] > 0: every frame cut that many + times more with its region moved at random, + as rows.py cuts drawn rows +""" + +import json +import random +import sys +from bisect import bisect_left +from pathlib import Path + +import numpy as np +from PIL import Image + +from common import cell_input, positions, split_cells +from context import ctx_cells +from paths import WORK + +REST = 0.3 # a wheel at rest may sit this far off its digit +PAD = 0.15 # added to the interval of a wheel that turned + + +def steps(text: str | None, decimals: int) -> int | None: + return None if text in (None, "") else round(float(text) * 10**decimals) + + +def labels(readings: list[dict], decimals: int) -> list[dict]: + order = sorted(range(len(readings)), key=lambda i: readings[i]["seconds"]) + known = {} + for i, r in enumerate(readings): + if r["answer"] == "read correctly": + known[i] = steps(r["value"], decimals) + elif r["answer"] == "misread" and r["right_value"]: + known[i] = steps(r["right_value"], decimals) + cand = [i for i in order if i in known or (r := readings[i])["answer"] == "unchecked" and not r["rejected"] and r["value"]] + val = {i: known.get(i, steps(readings[i]["value"], decimals)) for i in cand} + weight = {i: (1000 if i in known else 1) for i in cand} + best, prev = [], [] + for j, i in enumerate(cand): # the heaviest non-decreasing chain in time + b, p = weight[i], -1 + for k in range(j): + if val[cand[k]] <= val[i] and best[k] + weight[i] > b: + b, p = best[k] + weight[i], k + best.append(b) + prev.append(p) + chain = [] + j = max(range(len(cand)), key=lambda k: best[k]) if cand else -1 + while j >= 0: + chain.append(cand[j]) + j = prev[j] + chain.reverse() + t = [readings[i]["seconds"] for i in chain] + out = [] + for i, r in enumerate(readings): + if i in known: + out.append({"lo": known[i], "hi": known[i], "how": "checked"}) + continue + s = r["seconds"] + k = bisect_left(t, s) + before = [c for c in chain[max(0, k - 2) : k + 1] if c != i and readings[c]["seconds"] <= s] + after = [c for c in chain[k : k + 2] if c != i and readings[c]["seconds"] >= s] + if not before or not after: + out.append({"lo": None, "hi": None, "how": "edge"}) + else: + out.append({"lo": val[before[-1]], "hi": val[after[0]], "how": "chain" if i in chain else "between"}) + return out + + +def wheel_intervals(lo: int, hi: int, digits: int) -> list[tuple[float, float]]: + """Every wheel's interval of positions (middle, half width) over the values [lo - 0.5, hi + 0.5]; + half 5 for a wheel that went more than half way round (no information).""" + vs = np.linspace(lo - 0.5, hi + 0.5, min(2000, max(21, (hi - lo + 1) * 20))) + pos = np.array([positions(float(v) % 10**digits, digits) for v in vs]) + out = [] + for k in range(digits): + p = np.unwrap(pos[:, k] * 2 * np.pi / 10) * 10 / (2 * np.pi) + a, b = p.min(), p.max() + if b - a > 5: + out.append((0.0, 5.0)) + elif b - a < 0.05: + out.append(((a + b) / 2 % 10, REST)) + else: + out.append(((a + b) / 2 % 10, (b - a) / 2 + PAD)) + return out + + +MAX_OFFSET = 0.38 # of a cell, as rows.MAX_OFFSET + + +def moved_box(box, digits: int, height: int): + """The region moved as a user might draw it: shifted by up to MAX_OFFSET of a cell, a little + narrower or wider, higher or lower (rows.row_cell's cut, on a real frame).""" + x0, y0, x1, y1 = box + cw, h = (x1 - x0) / digits, y1 - y0 + off = random.uniform(-MAX_OFFSET, MAX_OFFSET) if random.random() < 0.8 else random.uniform(-0.1, 0.1) + wide = random.uniform(-0.15, 0.15) * digits / 2 # cells, each side + top, bottom = random.uniform(-0.12, 0.12), random.uniform(-0.12, 0.12) + return (x0 + (off - wide) * cw, max(0, y0 + top * h), x1 + (off + wide) * cw, min(height, y1 + bottom * h)) + + +def cut(im: Image.Image, box, digits: int): + """The one-cell input and the context input of a region in a frame.""" + x0, y0, x1, y1 = box + region = im.crop((round(max(0, x0)), round(y0), round(min(im.width, x1)), round(y1))) + return np.stack([(cell_input(c) * 255).astype(np.uint8) for c in split_cells(region, digits)]), ctx_cells(im, box, digits) + + +def training_cells(X, lo, hi, digits): + xs, mids, halves = [], [], [] + for i in range(len(X)): + for k, (m, h) in enumerate(wheel_intervals(int(lo[i]), int(hi[i]), digits)): + if h < 5: + xs.append(X[i, k]) + mids.append(m) + halves.append(h) + return {"X": np.stack(xs), "mid": np.array(mids, np.float32), "half": np.array(halves, np.float32)} + + +def build(name: str, folder: Path, split: str, moved: int = 0) -> None: + random.seed(0) + data = json.loads((folder / "readings.json").read_text(encoding="utf-8")) + readings, digits = data["readings"], int(data["settings"]["digits"]) + labs = labels(readings, int(data["settings"]["decimals"])) + eye = folder / "eye.json" + if eye.exists(): + number = {r["file"][-8:-4]: i for i, r in enumerate(readings)} + for first, last, lo, hi in json.loads(eye.read_text()): + for n in range(first, last + 1): + if (i := number.get(f"{n:04d}")) is not None: + labs[i] = {"lo": lo, "hi": hi, "how": "eye"} + frames = folder / "frames.json" + if frames.exists(): + number = {r["file"][-8:-4]: i for i, r in enumerate(readings)} + for n, v in json.loads(frames.read_text()).items(): + if (i := number.get(n)) is not None: + labs[i] = {"lo": None, "hi": None, "how": "unreadable"} if v is None else {"lo": v[0], "hi": v[1], "how": "eye"} + parts = {p: {"X": [], "C": [], "lo": [], "hi": [], "seconds": [], "MX": [], "MC": [], "Mlo": [], "Mhi": []} for p in "ab"} + skipped = 0 + for r, lab in sorted(zip(readings, labs, strict=True), key=lambda x: x[0]["seconds"]): + if lab["lo"] is None or not (folder / r["file"]).exists(): + continue + if r.get("region") != "as read" or not r.get("box"): + skipped += 1 + continue + im = Image.open(folder / r["file"]).convert("RGB") + box = r["box"] + part = parts["a" if r["date"] <= split else "b"] + cells, context = cut(im, box, digits) + part["X"].append(cells) + part["C"].append(context) + for _ in range(moved): + cells, context = cut(im, moved_box(box, digits, im.height), digits) + part["MX"].append(cells) + part["MC"].append(context) + part["Mlo"].append(lab["lo"]) + part["Mhi"].append(lab["hi"]) + part["lo"].append(lab["lo"]) + part["hi"].append(lab["hi"]) + part["seconds"].append(r["seconds"]) + (WORK / "bench").mkdir(exist_ok=True) + for p, d in parts.items(): + if not d["X"]: + continue + lo, hi = np.array(d["lo"], np.int64), np.array(d["hi"], np.int64) + group = np.cumsum(np.r_[True, (lo[1:] != lo[:-1]) | (hi[1:] != hi[:-1])]) - 1 # in time order + for suffix, key in (("", "X"), ("_ctx", "C")): + X = np.stack(d[key]) + np.savez_compressed( + WORK / "bench" / f"{name}_{p}{suffix}.npz", + X=X, lo=lo, hi=hi, seconds=np.array(d["seconds"], np.int64), group=group, digits=digits, + ) + np.savez_compressed(WORK / "data" / f"{name}_{p}{suffix}.npz", **training_cells(X, lo, hi, digits)) + if moved: + mlo, mhi = np.array(d["Mlo"], np.int64), np.array(d["Mhi"], np.int64) + MX = np.stack(d["M" + key]) + np.savez_compressed(WORK / "data" / f"{name}_{p}_moved{suffix}.npz", **training_cells(MX, mlo, mhi, digits)) + print(f"{name}_{p}", len(lo), "readings in", group[-1] + 1, "runs", f"(+ {len(d['Mlo'])} moved)" if moved else "") + print(skipped, "readings without a region as read left out (still used for the labels)") + + +if __name__ == "__main__": + build(sys.argv[1], Path(sys.argv[2]), sys.argv[3], int(sys.argv[4]) if len(sys.argv) > 4 else 0) diff --git a/tools/wheelreader/fit_boxes.py b/tools/wheelreader/fit_boxes.py new file mode 100644 index 0000000..585a009 --- /dev/null +++ b/tools/wheelreader/fit_boxes.py @@ -0,0 +1,127 @@ +"""Boxes that fit the wheels, for training on a shared export (`docs/WHEEL_READER_PLAN.md` §5). + +python fit_boxes.py writes fitted.json ({file: box}) in the export folder + +An export says where the user's region was, and a region can be off (cut a wheel, sit a little +wide): trained on as it is, a model learns to read the neighbouring wheel. For every stretch of +frames read with the same region, this finds the move of its edges (left/right in cells, top/bottom +in shares of its height) under which the frames' labels are most likely: the labels come from +other frames (exports.labels), never from what the model reads. as in bench.py. +""" + +import json +import sys +from pathlib import Path + +import numpy as np +from PIL import Image + +import bench +import exports +from context import ctx_cells + +SHIFTS = np.round(np.arange(-0.5, 0.501, 0.05), 2) # cells, the whole box +WIDTHS = np.round(np.arange(-0.3, 0.301, 0.05), 2) # cells, each side +HEIGHTS = (-0.1, -0.05, 0.0, 0.05, 0.1) # shares of the height, top and bottom +MIN_GAIN = 0.25 # log probability per frame a move must gain, else the region stays as drawn +COST = 0.1 # per frame and cell (or share of the height) moved: of two moves that explain the labels alike, the smaller + + +def stretches(items, tolerance=0.03): + """Runs of frames (in time) whose box differs by less than ``tolerance`` of its width.""" + runs, current = [], [] + for item in sorted(items, key=lambda it: it["seconds"]): + if current: + first = current[0]["box"] + if max(abs(a - b) for a, b in zip(item["box"], first)) > tolerance * (first[2] - first[0]): + runs.append(current) + current = [] + current.append(item) + return runs + [current] if current else runs + + +def moved(box, digits, left, right, top, bottom): + x0, y0, x1, y1 = box + cw, h = (x1 - x0) / digits, y1 - y0 + return (x0 + left * cw, y0 + top * h, x1 + right * cw, y1 + bottom * h) + + +def score(model, frames, digits, move): + """Sum over frames of the log probability of every wheel's label interval, less COST for the move.""" + cells, context, masks = [], [], [] + for f in frames: + x0, y0, x1, y1 = moved(f["box"], digits, *move) + im = f["image"] + y0, y1 = max(0, y0), min(im.height, y1) + region = im.crop((round(max(0, x0)), round(y0), round(min(im.width, x1)), round(y1))) + cells.append(bench.one_cell(region, digits)) + context.append(ctx_cells(im, (x0, y0, x1, y1), digits)) + masks.append(f["mask"]) + X, C = np.stack(cells), np.stack(context) + P = model.probs(X.reshape(-1, *X.shape[2:]), C.reshape(-1, *C.shape[2:])) + M = np.concatenate(masks) + fit = float(np.log(np.maximum((P * M).sum(axis=1), 1e-9)).sum()) + return fit - COST * len(frames) * sum(abs(v) for v in move) + + +def wheel_masks(lo, hi, digits): + centres = np.arange(bench.BINS) / 10 + out = [] + for mid, half in exports.wheel_intervals(lo, hi, digits): + dist = np.abs((centres - mid + 5) % 10 - 5) + out.append(dist <= half + 1e-4) + return np.array(out, np.float64) + + +def fit(folder: Path, model) -> None: + data = json.loads((folder / "readings.json").read_text(encoding="utf-8")) + readings, digits = data["readings"], int(data["settings"]["digits"]) + labels = exports.labels(readings, int(data["settings"]["decimals"])) + boxes = json.loads((folder / "boxes.json").read_text()) if (folder / "boxes.json").exists() else {} + items = [] + for r, lab in zip(readings, labels, strict=True): + box = r.get("box") or boxes.get(r["file"]) + if box is not None: + items.append({"file": r["file"], "box": box, "seconds": r.get("seconds", 0), "date": r.get("date"), "label": lab}) + out = {} + for run in stretches(items): + # a wide interval (or none) says little about where the wheels are: such frames only take + # the box their stretch gets + known = [f for f in run if f["label"]["lo"] is not None and f["label"]["hi"] - f["label"]["lo"] <= 10] + sample = known[:: max(1, len(known) // 40)] # enough frames to fit a box, not every one + if not sample: + for f in run: + out[f["file"]] = f["box"] + continue + for f in sample: + f["image"] = Image.open(folder / f["file"]).convert("RGB") + f["mask"] = wheel_masks(f["label"]["lo"], f["label"]["hi"], digits) + best = (0.0, 0.0, 0.0, 0.0) + best_score = score(model, sample, digits, best) + as_drawn = best_score + for shift in SHIFTS: + for width in WIDTHS: + move = (shift - width, shift + width, 0.0, 0.0) + s = score(model, sample, digits, move) + if s > best_score: + best, best_score = move, s + for top in HEIGHTS: + for bottom in HEIGHTS: + move = (best[0], best[1], top, bottom) + s = score(model, sample, digits, move) + if s > best_score: + best, best_score = move, s + per_frame = (best_score - as_drawn) / len(sample) + if per_frame < MIN_GAIN: + best = (0.0, 0.0, 0.0, 0.0) + print(f"{run[0]['date']}..{run[-1]['date']} ({len(run)} frames, {len(sample)} fitted on): left {best[0]:+.2f} right {best[1]:+.2f} cells, " + f"top {best[2]:+.2f} bottom {best[3]:+.2f}; log p per frame {per_frame:+.2f}", flush=True) + for f in run: + out[f["file"]] = [round(v) for v in moved(f["box"], digits, *best)] + f.pop("image", None) + (folder / "fitted.json").write_text(json.dumps(out)) + print(len(out), "boxes") + + +if __name__ == "__main__": + fit(Path(sys.argv[1]), bench.Model(sys.argv[2])) diff --git a/tools/wheelreader/items.py b/tools/wheelreader/items.py index 3495ecc..a2dfdcb 100644 --- a/tools/wheelreader/items.py +++ b/tools/wheelreader/items.py @@ -28,7 +28,8 @@ } # fmt: skip -def pedromfa(folder: str, only: set[int] | None = None): +def pedromfa(folder: str, only: set[int] | None = None, whole: bool = False): + """``whole``: the exported image and where the region is in it, instead of the region.""" data = json.loads((LOCAL / folder / "readings.json").read_text(encoding="utf-8")) for r in data["readings"]: i = int(r["file"][-8:-4]) @@ -39,12 +40,14 @@ def pedromfa(folder: str, only: set[int] | None = None): v = float(r["right_value"] or r["value"]) if v > 10000: v /= 1000 - yield ( - f"{folder}/{i:04d}", - 7, - float(round(v * 1000)), - export_region(Image.open(LOCAL / folder / r["file"]).convert("RGB")), - ) + image = Image.open(LOCAL / folder / r["file"]).convert("RGB") + if whole: + from context import centre_box + + region = (image, r.get("box") or centre_box(image)) + else: + region = export_region(image, r.get("box")) + yield f"{folder}/{i:04d}", 7, float(round(v * 1000)), region def bolausson(meter: str, first_only: bool = True): diff --git a/tools/wheelreader/rows.py b/tools/wheelreader/rows.py new file mode 100644 index 0000000..f76c226 --- /dev/null +++ b/tools/wheelreader/rows.py @@ -0,0 +1,190 @@ +"""Drawn rows of wheels (wheels-v2): a cell is cut from a drawn row of five wheels, so a region +drawn a little off shows the wheel off-centre with a part of its neighbour (digits and all) at the +side, and printed text above or below the windows. Photometric steps as ``synth.render``. + +python rows.py sheet out.png [seed] a sheet to look at +python rows.py build N data/rows.npz (one cell, 32 x 48) and data/rows_ctx.npz + (the cell with context, 64 x 48), N cells each +""" + +import io +import random +import string +import sys +from multiprocessing import Pool + +import numpy as np +from PIL import Image, ImageDraw, ImageEnhance, ImageFilter + +from synth import FONTS, SCHEMES, font, jitter, wheel_strip + +MAX_OFFSET = 0.38 # of the cell width + + +def row_cell(position: float, context: float = 0.0) -> Image.Image: + """One cell (``context``: plus that share of a cell on each side) cut from a drawn row of five wheels.""" + cw = random.randint(40, 90) + ch = int(cw * random.uniform(1.2, 2.4)) + scheme = random.choice(SCHEMES) + scheme = (jitter(scheme[0], 18), jitter(scheme[1], 18)) + # neighbours often share the scheme; a red/black boundary sometimes + other = scheme if random.random() < 0.7 else (jitter(random.choice(SCHEMES)[0], 18), jitter(random.choice(SCHEMES)[1], 18)) + wheel_w = int(cw * random.uniform(0.5, 0.85)) + pitch_x = cw # wheels are one cell apart when the region fits + window_top = int(ch * random.uniform(-0.05, 0.2)) + window_bot = int(ch * random.uniform(0.8, 1.05)) + win_h = max(8, window_bot - window_top) + digit_h = win_h * random.uniform(0.42, 0.78) + pitch = digit_h * random.uniform(1.15, 1.7) + fnt = font(random.choice(FONTS), max(8, int(digit_h * 1.32))) + curve = random.choice([0, random.uniform(0.3, 1.1)]) + crossed = random.random() < 0.3 # the whole row has crossed sevens or none + frame_c = jitter( + random.choice([(30, 30, 34), (200, 200, 205), (60, 60, 66), (120, 120, 125), (230, 230, 230), scheme[0]]), 25 + ) + W = 5 * pitch_x + canvas = Image.new("RGB", (W, ch), frame_c) + d = ImageDraw.Draw(canvas) + # neighbours: at rest (any digit), the right one maybe turning freely + def at_rest(): + return random.randint(0, 9) + (random.gauss(0, 0.06) if random.random() < 0.8 else 0) + + right_p = random.uniform(0, 10) if random.random() < 0.4 else at_rest() + right_s = other if random.random() < 0.5 else scheme + wheels = ((at_rest(), other), (at_rest(), other), (position, scheme), (right_p, right_s), (random.uniform(0, 10), right_s)) + for j, (p, sch) in enumerate(wheels): + strip = wheel_strip(p % 10, wheel_w, win_h, sch, fnt, digit_h, pitch, curve, crossed).resize((wheel_w, win_h)) + x0 = j * pitch_x + (pitch_x - wheel_w) // 2 + canvas.paste(strip, (x0, window_top)) + if random.random() < 0.5: + for xx in (x0 - 1, x0 + wheel_w): + d.line((xx, 0, xx, ch), fill=jitter(frame_c, 40), width=random.randint(1, 3)) + # printed text below or above the windows (type plates, serial numbers) + if random.random() < 0.35: + size = max(6, int(ch * random.uniform(0.1, 0.2))) + tf = font(random.choice(FONTS), size) + text = "".join(random.choice(string.ascii_uppercase + string.digits + "-/,") for _ in range(random.randint(6, 16))) + y = random.choice([random.uniform(0.82, 0.98), random.uniform(-0.05, 0.08)]) * ch + d.text((random.uniform(-W / 3, W / 3), y), text, font=tf, fill=jitter((30, 30, 30), 30), anchor="lm") + # the cell as the app cuts it: one cell wide, off by up to MAX_OFFSET, sometimes a little narrower or wider + off = random.uniform(-MAX_OFFSET, MAX_OFFSET) * cw if random.random() < 0.8 else random.uniform(-0.1, 0.1) * cw + width = cw * random.uniform(0.85, 1.15) + left = 2 * pitch_x + (pitch_x - width) / 2 + off - context * width + img = canvas.crop((round(left), 0, round(left + width * (1 + 2 * context)), ch)) + return finish(img, frame_c) + + +def finish(img: Image.Image, frame_c) -> Image.Image: + """synth.render's geometry and photometric steps, unchanged.""" + cw, ch = img.size + if random.random() < 0.7: + img = img.rotate(random.uniform(-5, 5), resample=Image.Resampling.BILINEAR, expand=False, fillcolor=frame_c) + if random.random() < 0.4: + sh = random.uniform(-0.12, 0.12) + img = img.transform(img.size, Image.Transform.AFFINE, (1, sh, -sh * ch / 2, 0, 1, 0), Image.Resampling.BILINEAR, fillcolor=frame_c) + if random.random() < 0.5: + t = int(ch * random.uniform(-0.1, 0.15)) + b = int(ch * random.uniform(0.85, 1.1)) + img = img.crop((0, t, cw, b)) + arr = np.asarray(img).astype(np.float32) + H, W = arr.shape[:2] + yy, xx = np.mgrid[0:H, 0:W] + light = np.ones((H, W), np.float32) + if random.random() < 0.7: + gx, gy = random.uniform(-0.5, 0.5), random.uniform(-0.6, 0.6) + light *= 1 + gx * (xx / W - 0.5) + gy * (yy / H - 0.5) + if random.random() < 0.3: + cx, cy, r = random.uniform(0, W), random.uniform(0, H), random.uniform(0.2, 0.8) * max(W, H) + light += random.uniform(0.2, 0.9) * np.exp(-(((xx - cx) ** 2 + (yy - cy) ** 2) / (r * r))) + if random.random() < 0.3: + top_shadow = random.uniform(0.1, 0.35) * H + light *= 1 - random.uniform(0.2, 0.6) * np.clip(1 - yy / top_shadow, 0, 1) + if random.random() < 0.2: # shadow at the bottom of the window too + bot = random.uniform(0.1, 0.3) * H + light *= 1 - random.uniform(0.2, 0.6) * np.clip((yy - (H - bot)) / bot, 0, 1) + arr *= light[:, :, None] + if random.random() < 0.15: + cx, cy = random.uniform(0, W), random.uniform(0, H) + rx, ry = random.uniform(2, W / 2), random.uniform(2, H / 3) + g = np.exp(-(((xx - cx) / rx) ** 2 + ((yy - cy) / ry) ** 2)) + arr = arr * (1 - g[:, :, None]) + 255 * g[:, :, None] + if random.random() < 0.3: + k = random.uniform(0.3, 0.8) + arr = arr * k + arr.mean() * (1 - k) + if random.random() < 0.2: + arr *= np.array([random.uniform(0.8, 1.2) for _ in range(3)], np.float32) + if random.random() < 0.2: + for _ in range(random.randint(1, 8)): + px, py = random.randint(0, W - 1), random.randint(0, H - 1) + r = random.randint(1, 3) + arr[max(0, py - r) : py + r, max(0, px - r) : px + r] *= random.uniform(0.3, 0.8) + img = Image.fromarray(np.clip(arr, 0, 255).astype(np.uint8)) + if random.random() < 0.6: + img = img.filter(ImageFilter.GaussianBlur(random.uniform(0.3, 2.2) * W / 40)) + if random.random() < 0.15: + k = random.randint(2, 6) + a = np.asarray(img).astype(np.float32) + a = sum(np.roll(a, s, axis=0) for s in range(k)) / k + img = Image.fromarray(a.astype(np.uint8)) + if random.random() < 0.6: + f = random.uniform(0.25, 0.8) + img = img.resize((max(6, int(img.width * f)), max(8, int(img.height * f))), Image.Resampling.BILINEAR) + if random.random() < 0.6: + a = np.asarray(img).astype(np.float32) + a += np.random.normal(0, random.uniform(2, 14), a.shape) + img = Image.fromarray(np.clip(a, 0, 255).astype(np.uint8)) + if random.random() < 0.7: + buf = io.BytesIO() + img.save(buf, "JPEG", quality=random.randint(15, 90)) + img = Image.open(io.BytesIO(buf.getvalue())).convert("RGB") + if random.random() < 0.3: + img = ImageEnhance.Contrast(img).enhance(random.uniform(0.5, 1.4)) + return img + + +def _chunk(args): + seed, n, context = args + from common import cell_input + from context import CTX, ctx_input + + random.seed(seed) + np.random.seed(seed % 2**32) + xs, ps = [], [] + for _ in range(n): + p = random.uniform(0, 10) + if random.random() < 0.4: + p = (round(p) + random.gauss(0, 0.06)) % 10 + if context: + xs.append(ctx_input(row_cell(p, context=CTX))) + else: + xs.append((cell_input(row_cell(p)) * 255).astype(np.uint8)) + ps.append(p) + return np.stack(xs), np.array(ps, np.float32) + + +def build(n: int) -> None: + from paths import WORK + + for name, context, seed0 in (("rows", False, 10_000), ("rows_ctx", True, 20_000)): + with Pool() as pool: + parts = pool.map(_chunk, [(seed0 + i, 2000, context) for i in range(n // 2000)]) + x = np.concatenate([p[0] for p in parts]) + p = np.concatenate([p[1] for p in parts]) + np.savez_compressed(WORK / "data" / f"{name}.npz", X=x, mid=p, half=np.full(len(p), 0.05, np.float32)) + print(name, x.shape) + + +if __name__ == "__main__": + if sys.argv[1] == "build": + build(int(sys.argv[2])) + else: + random.seed(int(sys.argv[3]) if len(sys.argv) > 3 else 0) + cols, nrows = 16, 8 + sheet = Image.new("RGB", (cols * 70, nrows * 110), "white") + dr = ImageDraw.Draw(sheet) + for k in range(cols * nrows): + p = random.uniform(0, 10) + x, y = (k % cols) * 70, (k // cols) * 110 + sheet.paste(row_cell(p).resize((60, 90)), (x, y)) + dr.text((x, y + 92), f"{p:.1f}", fill="black") + sheet.save(sys.argv[2]) diff --git a/tools/wheelreader/synth.py b/tools/wheelreader/synth.py index a121478..e6c6e7f 100644 --- a/tools/wheelreader/synth.py +++ b/tools/wheelreader/synth.py @@ -56,9 +56,10 @@ def jitter(c: tuple[int, int, int], amount: int) -> tuple[int, int, int]: def wheel_strip( - position: float, w: int, h: int, scheme, fnt, digit_h: float, pitch: float, curve: float + position: float, w: int, h: int, scheme, fnt, digit_h: float, pitch: float, curve: float, crossed: bool = False ) -> Image.Image: - """The visible part of one wheel, ``h`` tall, digits rolling by ``position``.""" + """The visible part of one wheel, ``h`` tall, digits rolling by ``position``. ``crossed``: sevens + with a bar across the stem, as on many European meters.""" bg, fg = scheme big_h = int(h * 2.2) strip = Image.new("RGB", (w, big_h), bg) @@ -69,6 +70,9 @@ def wheel_strip( digit = (base + step) % 10 y = cy + (base + step - position) * pitch d.text((w / 2, y), str(digit), font=fnt, fill=fg, anchor="mm") + if crossed and digit == 7: + bar = max(1, round(digit_h * 0.08)) + d.line((w / 2 - digit_h * 0.2, y, w / 2 + digit_h * 0.12, y), fill=fg, width=bar) # cylinder: rows far from the centre are compressed (sample with arcsin) arr = np.asarray(strip) out_h = h diff --git a/tools/wheelreader/temporal.py b/tools/wheelreader/temporal.py new file mode 100644 index 0000000..78e9ae0 --- /dev/null +++ b/tools/wheelreader/temporal.py @@ -0,0 +1,125 @@ +"""Decoding a counter over time, not frame by frame (`docs/WHEEL_READER_PLAN.md` §4.2, step 2). + +python temporal.py [--sets ole_a,ole_b,ped_a,ped_b] [--cost 0.5,1,2] [--reset 8] + +A counter never goes down and mostly moves slowly, so every frame's value is chosen together with +the frames before it: a forward Viterbi pass, frame by frame as the app would run it. Per frame the +candidates are the frame's own best value and its neighbours (+-3 steps) and the values carried +from the frame before (a meter at rest). A candidate's score is how well the frame's wheels fit it +(the app's slack and common shift); a step from u to v costs ``cost`` x (v - u) / (expected + 1), +expected being the steps the rate learnt so far (``learnt_rate``) gives over the time between the frames, and a +step down is only possible through a reset (``reset``, in log probability), so one wrong frame +cannot lock the path in. The chosen value then goes through the same checks as the single-frame +replay (bench.simulate: went down, change limit, threshold 0.70). Its confidence is the smaller +of the weakest wheel's mass at it and the share of the probability among the candidates within +one step of it (a wheel half way is not doubt): a frame +that fits two values about as well is not sure, whichever the path prefers. +""" + +import argparse +import math + +import numpy as np + +import bench +from common import BINS, positions + +NEAR = 3 # steps around the frame's own best value +KEEP = 12 # candidates carried to the next frame + + +def wheel_logp(probs, slack=1): + near = sum(np.roll(probs, s, axis=1) for s in range(-slack, slack + 1)) + return np.log(np.maximum(near, 1e-12)) + + +def fit(logp, probs, value, digits, max_shift=2, penalty=0.3): + """Log probability of a value under one frame and the weakest wheel's mass at it. The value is + a step of the last wheel: that wheel may stand anywhere in [value - 0.5, value + 0.5) (the + wheels to its left follow it), and the region may sit a little high or low (common shift).""" + best, best_bins, best_shift = -1e18, None, 0 + for f in np.arange(-0.5, 0.5, 0.1): + bins = [round(p * BINS / 10) % BINS for p in positions(float(value) + f, digits)] + for shift in range(-max_shift, max_shift + 1): + s = sum(logp[k, (b + shift) % BINS] for k, b in enumerate(bins)) - penalty * abs(shift) + if s > best: + best, best_bins, best_shift = s, bins, shift + mass = min(sum(probs[k, (b + best_shift + d) % BINS] for d in range(-4, 5)) for k, b in enumerate(best_bins)) + return best, float(mass) + + +RATE_WINDOW_S = 86_400 # the rate is learnt from the values chosen over the last day + + +def learnt_rate(history, now): + """Steps per second of the values chosen before ``now`` over the last RATE_WINDOW_S: only + what the app would know then (never the labels, never later frames).""" + recent = [(t, v) for t, v in history if now - t <= RATE_WINDOW_S] + if len(recent) < 2 or recent[-1][0] == recent[0][0]: + return 0.0 + return max(0.0, (recent[-1][1] - recent[0][1]) / (recent[-1][0] - recent[0][0])) + + +def decode(rows, P, digits, cost, reset): + """Each frame's value as chosen by the forward pass, with the weakest-wheel confidence at it.""" + prev, prev_t, out, history = {}, None, [], [] + for r, probs in zip(rows, P, strict=True): + logp = wheel_logp(probs) + own = r["v"] + cand = {(own + d) % 10**digits for d in range(-NEAR, NEAR + 1)} | set(prev) + scores = {v: fit(logp, probs, v, digits) for v in cand} + cur = {} + if not prev: + cur = {v: s for v, (s, _m) in scores.items()} + else: + expected = learnt_rate(history, r["seconds"]) * (r["seconds"] - prev_t) + floor = max(prev.values()) - reset + for v, (s, _m) in scores.items(): + best = floor + for u, su in prev.items(): + if u <= v: + best = max(best, su - cost * (v - u) / (expected + 1)) + cur[v] = s + best + top = max(cur.values()) + prev = dict(sorted(((v, s - top) for v, s in cur.items()), key=lambda x: -x[1])[:KEEP]) + prev_t = r["seconds"] + v = max(cur, key=cur.get) + history.append((r["seconds"], v)) + # how sure: the weakest wheel at the value, and the value's share among the candidates + total = sum(math.exp(s - cur[v]) for s in cur.values()) + share = sum(math.exp(s - cur[v]) for u, s in cur.items() if abs(u - v) <= 1) / total + out.append((v, min(scores[v][1], share))) + return out + + +def main() -> None: + ap = argparse.ArgumentParser() + ap.add_argument("model") + ap.add_argument("--sets", default="ole_a,ole_b,ped_a,ped_b") + ap.add_argument("--cost", default="0.5,1,2,4") + ap.add_argument("--reset", type=float, default=8.0) + a = ap.parse_args() + model = bench.Model(a.model) + for name in a.sets.split(","): + d, c = np.load(bench.BENCH / f"{name}.npz"), np.load(bench.BENCH / f"{name}_ctx.npz") + n, digits = d["X"].shape[:2] + P = model.probs(d["X"].reshape(-1, *d["X"].shape[2:]), c["X"].reshape(-1, *c["X"].shape[2:])).reshape(n, digits, BINS) + order = np.argsort(d["seconds"], kind="stable") + rows, probs = [], [] + for i in order: + v, mean, wmin, _score = bench.read(P[i]) + rows.append({"v": v, "conf": (mean, wmin), "lo": int(d["lo"][i]), "hi": int(d["hi"][i]), "seconds": int(d["seconds"][i])}) + probs.append(P[i]) + ok = lambda rs: sum(r["lo"] - 1 <= r["v"] <= r["hi"] + 1 for r in rs) # noqa: E731 + high = lambda rs: sum(r["v"] > r["hi"] + 1 for r in rs) # noqa: E731 + p, w, _h, s = bench.simulate(rows, 1, 0.7) + print(f"{name:6s} frame by frame ok {ok(rows):4d}/{n} high {high(rows):3d} | replay: {p} published, {w} wrong, {s:.1f} h too high") + for cost in (float(x) for x in a.cost.split(",")): + chosen = decode(rows, probs, digits, cost, a.reset) + tr = [{**r, "v": v, "conf": (m, m)} for r, (v, m) in zip(rows, chosen, strict=True)] + p, w, _h, s = bench.simulate(tr, 1, 0.7) + print(f"{'':6s} over time {cost:4g} ok {ok(tr):4d}/{n} high {high(tr):3d} | replay: {p} published, {w} wrong, {s:.1f} h too high") + + +if __name__ == "__main__": + main() diff --git a/tools/wheelreader/train.py b/tools/wheelreader/train.py index 92961cd..fbb1ea5 100644 --- a/tools/wheelreader/train.py +++ b/tools/wheelreader/train.py @@ -1,9 +1,18 @@ """Train the wheel position model. python train.py : [: ...] [--epochs N] [--dryad-orient model.pt] + [--input 32|64] [--width 16] [--ema 0.999] [--seed 0] [--teacher tag --alpha 0.5 --restrict --teacher-t 1] Sets are data/.npz. A weight is the share of each batch drawn from that set. -"dryad_train" needs --dryad-orient (a model that picks each crop's orientation) or uses upright. +Dryad sets need --dryad-orient (a model that picks each crop's orientation) or use upright; the +orientation is picked on the one-cell set dryad_train and kept for dryad_train_ctx too. +--input 64: cells with context (context.py); --ema keeps an exponential average of the weights. +--teacher: distillation (distill.py wrote data/_t_.npz; a _ctx set reads its one-cell +twin's); the loss is alpha x the label's interval loss + (1 - alpha) x cross entropy with the +teacher's probabilities (--restrict: only inside the label's interval, renormalised, so the +teacher only says where inside the interval; --teacher-t: the teacher's probabilities to the power +1/t and renormalised, below 1 sharper). A set without targets trains on its labels only. +Saves runs/.pt: {"state", "width", "input"}. """ import argparse @@ -21,7 +30,7 @@ class WheelNet(nn.Module): - def __init__(self, width: int = 32): + def __init__(self, width: int = 32, input_w: int = 32): super().__init__() c = width @@ -45,7 +54,7 @@ def block(i, o): block(4 * c, 4 * c), ) self.head = nn.Sequential( - nn.Flatten(), nn.Dropout(0.2), nn.Linear(4 * c * 6 * 4, 256), nn.ReLU(inplace=True), nn.Linear(256, BINS) + nn.Flatten(), nn.Dropout(0.2), nn.Linear(4 * c * 6 * (input_w // 8), 256), nn.ReLU(inplace=True), nn.Linear(256, BINS) ) def forward(self, x): # x: N x 1 x H x W in 0..1 @@ -106,16 +115,28 @@ def interval_logp(logp: np.ndarray, mid: np.ndarray, half: np.ndarray) -> np.nda return np.log(np.maximum(1e-12, (np.exp(logp) * mask).sum(axis=1))) -def orient_dryad(d: dict, model) -> dict: - """Keep the orientation of each crop the model finds more likely.""" +def load_model(path, width: int | None = None, device: str = DEV) -> "WheelNet": + """A model saved by train.py (or a plain state dict of the given width, as wheels-v1's).""" + state = torch.load(path, map_location=device) + if "state" in state: + model = WheelNet(state["width"], state["input"]) + state = state["state"] + else: + model = WheelNet(width or 32) + model.load_state_dict(state) + return model.to(device).eval() + + +def dryad_turned(model) -> np.ndarray: + """Per Dryad training photo, whether the model finds it more likely turned (1) than upright.""" + d = load("dryad_train") lp = interval_logp(predict(model, d["X"]), d["mid"], d["half"]) keys = d["img"] * 2 + d["rot"] score = np.bincount(keys, weights=lp, minlength=keys.max() + 1) n_img = d["img"].max() + 1 best_rot = (score[1::2][:n_img] > score[0::2][:n_img]).astype(int) - keep = d["rot"] == best_rot[d["img"]] print(f"dryad: {best_rot.mean():.1%} of crops turned") - return {k: v[keep] for k, v in d.items()} + return best_rot def main(): @@ -127,28 +148,48 @@ def main(): ap.add_argument("--batch", type=int, default=512) ap.add_argument("--width", type=int, default=32) ap.add_argument("--dryad-orient") + ap.add_argument("--input", type=int, default=32) + ap.add_argument("--ema", type=float, default=0.0) + ap.add_argument("--seed", type=int, default=0) + ap.add_argument("--teacher") + ap.add_argument("--alpha", type=float, default=0.5) + ap.add_argument("--restrict", action="store_true") + ap.add_argument("--teacher-t", type=float, default=1.0) args = ap.parse_args() - torch.manual_seed(0) - np.random.seed(0) + torch.manual_seed(args.seed) + np.random.seed(args.seed) sets = [] + turned = None for spec in args.sets: name, w = spec.split(":") d = load(name) + if args.teacher and (t := HERE / "data" / f"{name.removesuffix('_ctx')}_t_{args.teacher}.npz").exists(): + d["T"] = np.load(t)["T"] + if args.teacher_t != 1: + sharp = d["T"].astype(np.float32) ** (1 / args.teacher_t) + d["T"] = (sharp / sharp.sum(axis=1, keepdims=True)).astype(np.float16) if "rot" in d: if args.dryad_orient: - m = WheelNet(32).to(DEV) # the orienting model - m.load_state_dict(torch.load(args.dryad_orient, map_location=DEV)) - d = orient_dryad(d, m) + if turned is None: + turned = dryad_turned(load_model(args.dryad_orient, 32)) + d = {k: v[d["rot"] == turned[d["img"]]] for k, v in d.items()} else: d = {k: v[d["rot"] == 0] for k, v in d.items()} + assert d["X"].shape[2] == args.input, (name, d["X"].shape) x = torch.from_numpy(d["X"]).to(DEV) - sets.append((name, float(w), x, torch.from_numpy(d["mid"]).to(DEV), torch.from_numpy(d["half"]).to(DEV))) - print(name, len(x), w) + target = torch.from_numpy(d["T"]).to(DEV) if "T" in d else None + sets.append((name, float(w), x, torch.from_numpy(d["mid"]).to(DEV), torch.from_numpy(d["half"]).to(DEV), target)) + print(name, len(x), w, "teacher" if target is not None else "") total_w = sum(s[1] for s in sets) - model = WheelNet(args.width).to(DEV) + model = WheelNet(args.width, args.input).to(DEV) print("params", sum(p.numel() for p in model.parameters())) + ema = None + if args.ema: + from torch.optim.swa_utils import AveragedModel, get_ema_multi_avg_fn + + ema = AveragedModel(model, multi_avg_fn=get_ema_multi_avg_fn(args.ema), use_buffers=True) opt = torch.optim.AdamW(model.parameters(), lr=2e-3, weight_decay=1e-4) total = args.epochs * args.steps sched = torch.optim.lr_scheduler.OneCycleLR(opt, max_lr=3e-3, total_steps=total, pct_start=0.1) @@ -157,23 +198,43 @@ def main(): model.train() losses = [] for _ in range(args.steps): - xs, ms, hs = [], [], [] - for _name, w, x, m, h in sets: + xs, ms, hs, ts = [], [], [], [] + for _name, w, x, m, h, t in sets: k = max(1, round(args.batch * w / total_w)) idx = torch.randint(0, len(x), (k,), device=DEV) xs.append(x[idx]) ms.append(m[idx]) hs.append(h[idx]) + ts.append(t[idx].float() if t is not None else None) xb = augment(torch.cat(xs).float().div(255)[:, None]) - loss = interval_loss(model(xb), torch.cat(ms), torch.cat(hs)) + logits = model(xb) + if any(t is not None for t in ts): + # rows without targets learn from their labels only + has = torch.cat([torch.full((len(x),), t is not None, device=DEV) for x, t in zip(xs, ts, strict=True)]) + target = torch.cat([t if t is not None else torch.zeros(len(x), BINS, device=DEV) for x, t in zip(xs, ts, strict=True)]) + logp = F.log_softmax(logits, dim=1) + mask = interval_mask(torch.cat(ms), torch.cat(hs)) + own = -torch.logsumexp(logp.masked_fill(~mask, -1e9), dim=1) + if args.restrict: + inside = target * mask + target = torch.where(inside.sum(dim=1, keepdim=True) > 1e-6, inside, mask.float()) + target = target / target.sum(dim=1, keepdim=True).clamp_min(1e-6) + soft = -(target * logp).sum(dim=1) + loss = torch.where(has, args.alpha * own + (1 - args.alpha) * soft, own).mean() + else: + loss = interval_loss(logits, torch.cat(ms), torch.cat(hs)) opt.zero_grad(set_to_none=True) loss.backward() opt.step() sched.step() + if ema is not None: + ema.update_parameters(model) step += 1 losses.append(loss.item()) print(f"epoch {epoch + 1}/{args.epochs} loss {np.mean(losses):.4f}", flush=True) - torch.save(model.state_dict(), HERE / "runs" / f"{args.out}.pt") + if ema is not None: + model = ema.module + torch.save({"state": model.state_dict(), "width": args.width, "input": args.input}, HERE / "runs" / f"{args.out}.pt") if __name__ == "__main__": diff --git a/visionstate/CHANGELOG.md b/visionstate/CHANGELOG.md index 3cf7e70..8c415d6 100644 --- a/visionstate/CHANGELOG.md +++ b/visionstate/CHANGELOG.md @@ -1,5 +1,32 @@ # Changelog +## 0.10.1 + +### New + +- **Wheel reader v2 for mechanical counters.** It reads every wheel twice, on its own and with + the wheels next to it in view, so a region that does not quite fit is still read right, and it + only passes a reading when it is sure of every wheel, not just on average. Tested on two water + meters it was not trained on, it published no wrong value where v1 published six (most of them + 2–3 litres too high, two 20 litres), and it read more frames right. New counters use it. + Counters you have keep v1 until you choose v2 under *Read with* on their Settings tab; their + Live tab suggests it. +- On a meter photographed from the side whose last wheel is turning, v2 waits more often for a + clear picture, so the value can come a little later. It takes about twice as long per reading + as v1 (still a few milliseconds), and the app is 6 MB larger. + +### Better + +- **Exported readings say where the region is in each image** (Export readings on a reading + sensor's Quality tab), also for readings made before this version (since 0.8.0). The margin + around the region is cut off at the picture's edge, so the region was not always in the middle. + For the wheel reader they also say how sure it was of each wheel the way it decides. + +### Fixed + +- The suggestions on a counter's Live tab (to try the wheel reader) no longer break into columns + on narrow screens. + ## 0.10.0 ### New diff --git a/visionstate/DOCS.md b/visionstate/DOCS.md index db3353e..c63eef2 100644 --- a/visionstate/DOCS.md +++ b/visionstate/DOCS.md @@ -290,7 +290,8 @@ First choose **what it looks like**: - *Wheel reader* (default) — a small model made for VisionState that reads how far each wheel has turned, so a wheel half way between two digits is read as such. It reads the wheels together, the way they turn: a wheel only moves on while the wheel to its right goes from 9 - to 0, so a reading where they do not fit together is never chosen. + to 0, so a reading where they do not fit together is never chosen. A counter keeps the wheel + reader it was set up with; should a better one come, its Live tab suggests it. - *Text reader* — reads the wheels as text with the number reader (Settings → AI model). Only the middle of each field is read, so the dividers between the wheels are never mistaken for digits, and a reading with another number of digits than the counter has is rejected. @@ -322,7 +323,7 @@ First choose **what it looks like**: you can also leave it out of the region and count one digit and one decimal less. The wheel reader reads mechanical counters better than the text reader on every meter it was tried on. New counters use it; counters set up before it came keep the text reader until you - choose *Wheel reader* on their Settings tab (their Live tab suggests it). + choose a wheel reader on their Settings tab (their Live tab suggests it). - **Safety net**: a reading is rejected — and the last value kept — when the reader is less sure than the minimum (default 70 %), finds no number, a counter reads lower than before, a mechanical counter is read with the wrong number of digits, a counter goes up faster than its @@ -612,9 +613,12 @@ The models, chosen under **Settings → AI model** for all sensors of a kind (sm sensor recognises faces. - **Reading sensors:** PP-OCRv6 tiny (PaddleOCR) — included, a few milliseconds per reading. PP-OCRv6 small is larger and reads unusual fonts more reliably (downloaded on first use, - 21 MB). Mechanical counters are read by the included **VisionState wheel reader** (2 MB, a few - milliseconds per reading, listed under *Reading sensors: mechanical counters*) unless a sensor - is set to the text reader. There is one wheel reader so far, so there is nothing to choose. + 21 MB). Mechanical counters are read by an included **VisionState wheel reader** (listed under + *Reading sensors: mechanical counters*) unless a sensor is set to the text reader. New + counters use **v2** (6 MB, about 10 milliseconds per reading): it also looks at each wheel's + neighbours, so a region that does not quite fit is still read right, and it holds back a + reading when any one wheel is unclear. **v1** (2 MB) stays for the counters set up with it; + their Live tab suggests v2, and *Read with* on the Settings tab switches. Only loaded while at least one reading sensor exists. **Compare and manage the models** (under the choices) lists every model, from the smallest to diff --git a/visionstate/backend/tests/assets/backups/backup-format1-schema16.zip b/visionstate/backend/tests/assets/backups/backup-format1-schema16.zip new file mode 100644 index 0000000..8d084e1 Binary files /dev/null and b/visionstate/backend/tests/assets/backups/backup-format1-schema16.zip differ diff --git a/visionstate/backend/tests/test_reading.py b/visionstate/backend/tests/test_reading.py index 39ad097..a02a66e 100644 --- a/visionstate/backend/tests/test_reading.py +++ b/visionstate/backend/tests/test_reading.py @@ -1,9 +1,13 @@ """Reading sensors: parsing, plausibility, decoding, the reader on drawn displays, and the API.""" +import dataclasses import io import json +import math +import shutil import sqlite3 import zipfile +from types import SimpleNamespace from unittest.mock import patch import numpy as np @@ -12,8 +16,8 @@ from PIL import Image from sqlalchemy import select -from visionstate import readers -from visionstate.db import Database, Prediction, Sensor, keep_change_step, keep_text_reader +from visionstate import imaging, readers +from visionstate.db import Database, Prediction, Sensor, keep_change_step, keep_text_reader, keep_wheel_reader from visionstate.main import create_app from visionstate.mqtt import SensorDescriptor, discovery_messages from visionstate.readers import counter_line, decode, format_value, implausible, parse, wrong_digit_count @@ -323,14 +327,90 @@ def test_wheel_value_prefers_a_consistent_reading(): ], ) @pytest.mark.parametrize("taller", [0.0, 0.25]) -def test_wheel_reader_reads_drawn_counters(value, expected, taller): - spec = readers.WHEEL_READERS[readers.DEFAULT_WHEEL_READER] +@pytest.mark.parametrize("reader_id", list(readers.WHEEL_READERS)) +def test_wheel_reader_reads_drawn_counters(value, expected, taller, reader_id): + spec = readers.WHEEL_READERS[reader_id] reader = readers.WheelReader(spec, MODEL_DIR / spec.filename) result, used = reader.read_counter(counter_region(render_counter(value), 7, taller), 7) assert result.text == expected and result.score > 0.8 assert used.width > used.height # the cells side by side +class ContextSession: + """A model with the second input of a wheel reader with context, made from one without it: it + reads the cells alone and keeps the shapes it was fed (no such model is in the registry).""" + + def __init__(self, inner): + self.inner, self.fed = inner, [] + + def get_inputs(self): + return [*self.inner.get_inputs(), SimpleNamespace(name="context")] + + def get_outputs(self): + return self.inner.get_outputs() + + def run(self, names, feed): + self.fed.append({k: v.shape for k, v in feed.items()}) + cells = self.inner.get_inputs()[0].name + return self.inner.run(names, {cells: feed[cells]}) + + +@requires_reader +def test_a_wheel_reader_with_context_is_given_the_wheels_around_each_cell(monkeypatch): + v1 = readers.WHEEL_READERS["wheels-v1"] + spec = dataclasses.replace(v1, context=0.5, context_width=64) + with pytest.raises(ValueError, match="takes 1 inputs"): # a model without the second input + readers.WheelReader(spec, MODEL_DIR / v1.filename) + real = readers.cpu_session + monkeypatch.setattr(readers, "cpu_session", lambda path: ContextSession(real(path))) + reader = readers.WheelReader(spec, MODEL_DIR / v1.filename) + frame = render_counter(89932.0) + box = imaging.pixel_box(frame, imaging.region_box(counter_box(7))) + result, used = reader.read_counter(frame, 7, box) + assert result.text == "0089932" and result.score > 0.8 + assert sorted(reader.session.fed[-1].values()) == [(7, 1, 48, 32), (7, 1, 48, 64)] + # What the reader saw: the cells side by side at twice the size, one wheel per field (without + # the neighbours, which would show every wheel twice). + assert used.size == ((7 * (32 + 2) - 2) * 2, 48 * 2) + # The cells are the region's, cut out or not; their neighbours come from the frame around it. + region = frame.crop(box) + (cut, cut_context), (cells, context) = reader.inputs(region, 7), reader.inputs(frame, 7, box) + assert np.array_equal(cut, cells) + assert not np.array_equal(cut_context[-1], context[-1]) and np.array_equal(cut_context[3], context[3]) + # A region at the picture's edge: beyond it, the edge is repeated. + for edge in ((0, box[1], box[2], box[3]), (box[0], box[1], frame.width, box[3])): + assert reader.inputs(frame, 7, edge)[1].shape == (7, 48, 64) + + +def test_a_wheel_reader_registry_mistake_fails_at_once(): + v1 = readers.WHEEL_READERS["wheels-v1"] + with pytest.raises(ValueError, match="confidence"): + dataclasses.replace(v1, confidence="best") + with pytest.raises(ValueError, match="context"): + dataclasses.replace(v1, context=0.5) + + +def test_the_weakest_wheel_is_the_confidence_of_a_wheel_reader_that_says_so(): + """One unclear wheel among clear ones: the mean hides it, the weakest wheel does not.""" + spec = readers.WHEEL_READERS["wheels-v1"] + weakest = dataclasses.replace(spec, confidence="weakest") + probs = np.full((4, 100), 1e-6) + for k, b in enumerate([10, 20, 30]): + probs[k, b] = 1.0 + probs[3, 40], probs[3, 75] = 0.5, 0.5 # the last wheel: 4.0 or 7.5 + probs /= probs.sum(axis=1, keepdims=True) + at, shift = [10, 20, 30, 40], 0 + logp = readers.WheelReader.with_slack(probs) + reader = object.__new__(readers.WheelReader) + reader.spec = weakest + p = reader.wheel_p(probs, logp, at, shift) + assert reader.confidence(0.0, p) == pytest.approx(0.5, abs=0.01) == min(p) + assert p[:3] == pytest.approx([1.0] * 3, abs=0.01) + reader.spec = spec + assert reader.confidence(4 * math.log(0.84), reader.wheel_p(probs, logp, at, shift)) == pytest.approx(0.84) + assert reader.wheel_p(probs, logp, at, shift)[3] == pytest.approx(0.5, abs=0.01) + + @requires_reader @pytest.mark.parametrize( ("asset", "expected"), @@ -340,9 +420,10 @@ def test_wheel_reader_reads_drawn_counters(value, expected, taller): ("counter_turning.jpg", "0632590"), ], ) -def test_wheel_reader_reads_a_real_meter(asset, expected): +@pytest.mark.parametrize("reader_id", list(readers.WHEEL_READERS)) +def test_wheel_reader_reads_a_real_meter(asset, expected, reader_id): """Issue #40: an ESP32 camera with its flash on a water meter with red decimal wheels.""" - spec = readers.WHEEL_READERS[readers.DEFAULT_WHEEL_READER] + spec = readers.WHEEL_READERS[reader_id] reader = readers.WheelReader(spec, MODEL_DIR / spec.filename) result, _ = reader.read_counter(Image.open(ASSETS / asset).convert("RGB"), 7) assert result.text == expected and result.score > 0.8 @@ -350,7 +431,9 @@ def test_wheel_reader_reads_a_real_meter(asset, expected): assert result.details is not None wheels = result.details["wheels"] assert round(wheels["value"]) == int(expected) and wheels["log_p"] < 0 and abs(wheels["shift"]) <= 0.2 - assert len(wheels["per_wheel"]) == 7 + assert len(wheels["per_wheel"]) == 7 and wheels["confidence_rule"] == spec.confidence + if spec.confidence == "weakest": # the export shows the probabilities the confidence was made of + assert result.score == pytest.approx(min(w["p"] for w in wheels["per_wheel"]), abs=1e-4) for wheel in wheels["per_wheel"]: assert 0.0 <= wheel["at"] <= 9.9 and 0.0 < wheel["p"] <= 1.0 assert len(wheel["top"]) == 3 and wheel["top"][0][1] >= wheel["top"][1][1] >= wheel["top"][2][1] @@ -383,6 +466,32 @@ def test_counters_made_before_the_wheel_reader_keep_the_text_reader(tmp_path): assert keep_text_reader(None) is None +def test_counters_made_before_they_named_a_wheel_reader_keep_wheels_v1(tmp_path): + path = tmp_path / "old.db" + Database(path).init() + con = sqlite3.connect(path) + row = ( + "INSERT INTO sensor (slug, name, kind, source_type, source, interval_s, threshold, debounce, enabled, " + "entity_prefix, publish, created_at, reading) VALUES (?, ?, 'reading', 'http', 'x', 30, 0.7, 2, 1, 0, 1, " + "'2026-10-01', ?)" + ) + con.execute(row, ("water", "Water", json.dumps({"display": "counter", "digits": 8, "counter_reader": "wheels"}))) + con.execute(row, ("gas", "Gas", json.dumps({"display": "counter", "digits": 7, "counter_reader": "ocr"}))) + con.execute(row, ("lcd", "Display", json.dumps({"mode": "counter", "display": "auto"}))) + con.execute("PRAGMA user_version = 15") # as a database of 0.10.0 + con.commit() + con.close() + db = Database(path) + db.init() + with db.session() as s: + chosen = {r.slug: readers.wheel_reader_of(merge_reading(r.reading)) for r in s.query(Sensor)} + # A counter read as text keeps wheels-v1 too, should it be switched to the wheel reader; the + # display keeps no stored choice: should it become a counter, it gets the default. + assert chosen == {"water": "wheels-v1", "gas": "wheels-v1", "lcd": readers.DEFAULT_WHEEL_READER} + assert keep_wheel_reader({"display": "counter", "wheel_reader": "x"}) == {"display": "counter", "wheel_reader": "x"} + assert keep_wheel_reader(None) is None + + def test_counters_keep_their_change_limit_as_a_rate(tmp_path): path = tmp_path / "old.db" Database(path).init() @@ -829,6 +938,10 @@ def view(): box = counter_box(7) assert crop.width < frame.width * box["w"] * 1.4 and crop.height < frame.height * box["h"] * 1.4 assert entry["region"] == "as read" + # "box" is the region inside the image: the margin around it on every side. + left, top, right, bottom = entry["box"] + assert 0 < left < right < crop.width and 0 < top < bottom < crop.height + assert abs((right - left) - frame.width * box["w"]) <= 1 and abs((bottom - top) - frame.height * box["h"]) <= 1 # Moving the region later does not change how earlier readings are cut: they keep theirs. assert item["probs"]["roi"] == view()["roi"] @@ -851,6 +964,7 @@ def export_entry(): row.probs = {k: v for k, v in row.probs.items() if k != "roi"} old, size = export_entry() assert old["region"] == "current" and size[0] < crop.size[0] + assert old["box"][0] == 0 # the region starts at the frame's edge: no margin on the left assert client.patch(f"/api/v1/sensors/{sid}", json={"roi": box}).status_code == 200 preview = {"source_type": "http", "source": "http://fake", "roi": counter_box(7)} @@ -981,11 +1095,14 @@ def view(): # Two wheels almost turned over: the wheels together read 89.950. assert wait_for(lambda: view()["reading"]["value"] == "89.950", timeout=60) assert view()["reading"]["counter_reader"] == "wheels" + # A new counter stores the default, so a later default never changes it. + assert view()["reading"]["wheel_reader"] == readers.DEFAULT_WHEEL_READER status = client.get("/api/v1/status").json() - assert status["wheel_reader"] == readers.DEFAULT_WHEEL_READER and status["wheel_reader_error"] == "" + assert status["wheel_readers"] == [readers.DEFAULT_WHEEL_READER] and status["wheel_reader_error"] == "" assert status["reader"] is None # only a counter read with the wheel reader: no text reader loaded models = client.get("/api/v1/settings").json() assert models["wheel_reader"] == readers.DEFAULT_WHEEL_READER + assert models["wheel_readers_in_use"] == [readers.DEFAULT_WHEEL_READER] assert [m["id"] for m in models["wheel_readers"]] == list(readers.WHEEL_READERS) assert models["wheel_readers"][0]["installed"] and models["wheel_readers"][0]["license"] == "Apache-2.0" image = client.get(f"/api/v1/sensors/{sid}/reading/image") @@ -1026,7 +1143,7 @@ def export(meter: str = "") -> dict: return manifest manifest = export() - assert manifest["format"] == 3 and manifest["meter"] is None and manifest["left_out"] == 0 + assert manifest["format"] == 5 and manifest["meter"] is None and manifest["left_out"] == 0 assert manifest["reader"] == readers.DEFAULT_WHEEL_READER assert manifest["settings"]["counter_reader"] == "wheels" # The checked reading first, then the accepted one nobody checked, marked as such. @@ -1094,10 +1211,59 @@ def broken(*_args): client.app.state.runtime.grabber = CounterCamera(1234) sid = client.post("/api/v1/sensors", json=body).json()["id"] assert wait_for(lambda: client.get("/api/v1/status").json()["wheel_reader_error"] == "model file damaged") - assert client.get("/api/v1/status").json()["wheel_reader"] is None + assert client.get("/api/v1/status").json()["wheel_readers"] == [] assert "model file damaged" in client.get(f"/api/v1/sensors/{sid}").json()["live"]["error"] +@requires_reader +def test_each_counter_reads_with_the_wheel_reader_it_names(settings): + """Two wheel readers: each counter loads its own, one no counter reads with is let go, and a + sensor exported with a wheel reader this version does not have is imported with the default.""" + other = dataclasses.replace(readers.WHEEL_READERS["wheels-v1"], id="wheels-test", name="Test wheel reader") + settings.models_dir.mkdir(parents=True, exist_ok=True) + shutil.copy(MODEL_DIR / "wheels-v1.onnx", settings.models_dir / other.filename) + reading = {"mode": "counter", "display": "counter", "digits": 7, "change_limit": "off"} + body = {"kind": "reading", "source_type": "http", "source": "http://fake", "roi": counter_box(7), "interval_s": 1} + + def loaded(): + return client.get("/api/v1/status").json()["wheel_readers"] + + with TestClient(create_app(settings)) as client: + client.app.state.runtime.grabber = CounterCamera(1234) + with patch.dict(readers.WHEEL_READERS, {other.id: other}): + assert [w["id"] for w in client.get("/api/v1/config").json()["wheel_readers"]] == [ + "wheels-v1", + "wheels-v2", + other.id, + ] + unknown = {**body, "name": "Bad", "reading": {**reading, "wheel_reader": "wheels-nope"}} + assert client.post("/api/v1/sensors", json=unknown).status_code == 422 + first = client.post("/api/v1/sensors", json={**body, "name": "Water", "reading": reading}).json()["id"] + second = client.post( + "/api/v1/sensors", json={**body, "name": "Gas", "reading": {**reading, "wheel_reader": other.id}} + ).json()["id"] + default = readers.DEFAULT_WHEEL_READER + assert wait_for(lambda: set(loaded()) == {default, other.id}, timeout=60) + assert set(client.get("/api/v1/settings").json()["wheel_readers_in_use"]) == {default, other.id} + for sid in (first, second): + assert wait_for( + lambda sid=sid: client.get(f"/api/v1/sensors/{sid}").json()["reading"]["value"] == "1234" + ) + exported = client.get(f"/api/v1/sensors/{second}/export").content + # Switched to the other one: the one no counter reads with any more is let go. + switched = {**reading, "wheel_reader": default} + assert client.patch(f"/api/v1/sensors/{second}", json={"reading": switched}).status_code == 200 + assert wait_for(lambda: loaded() == [default]) + # This version has no wheels-test: the import reads with the default. + imported = client.post("/api/v1/import", files={"file": ("b.zip", exported, "application/zip")}) + assert imported.status_code == 201, imported.text + copy = client.get(f"/api/v1/sensors/{imported.json()['id']}").json() + assert copy["reading"]["wheel_reader"] == readers.DEFAULT_WHEEL_READER + for sid in (first, second, imported.json()["id"]): + assert client.delete(f"/api/v1/sensors/{sid}").status_code in (200, 204) + assert wait_for(lambda: loaded() == []) + + @requires_reader def test_spot_checks_of_accepted_readings(settings): camera = DisplayCamera("00500") diff --git a/visionstate/backend/visionstate/api/common.py b/visionstate/backend/visionstate/api/common.py index 01ded36..85077ad 100644 --- a/visionstate/backend/visionstate/api/common.py +++ b/visionstate/backend/visionstate/api/common.py @@ -11,7 +11,7 @@ from sqlalchemy import func, select from sqlalchemy.orm import Session -from .. import change_limit, detectors, imaging, teach +from .. import change_limit, detectors, imaging, readers, teach from ..db import Face, ModelInfo, Sample, SampleLabel, Sensor from ..engine import LiveState, ObjectTrack, Runtime, SensorConfig, with_people from ..mqtt import main_entities @@ -298,6 +298,7 @@ class ReadingIn(BaseModel): display: str = READING_DEFAULTS["display"] digits: int = Field(default=READING_DEFAULTS["digits"], ge=_dlo["digits"], le=_dhi["digits"]) counter_reader: str = READING_DEFAULTS["counter_reader"] + wheel_reader: str = Field(default=READING_DEFAULTS["wheel_reader"], validate_default=True) max_step: float = Field(default=READING_DEFAULTS["max_step"], ge=_dlo["max_step"], le=_dhi["max_step"]) change_limit: str = READING_DEFAULTS["change_limit"] max_rate: float = Field(default=READING_DEFAULTS["max_rate"], ge=_dlo["max_rate"], le=_dhi["max_rate"]) @@ -334,6 +335,15 @@ def _counter_reader(cls, value: str) -> str: raise ValueError(f"Unknown counter reader {value!r}") return value + @field_validator("wheel_reader") + @classmethod + def _wheel_reader(cls, value: str) -> str: + if not value: + return readers.DEFAULT_WHEEL_READER # stored, so a later default never changes it + if value not in readers.WHEEL_READERS: + raise ValueError(f"Unknown wheel reader {value!r}") + return value + @field_validator("device_class") @classmethod def _device_class(cls, value: str) -> str: diff --git a/visionstate/backend/visionstate/api/imports.py b/visionstate/backend/visionstate/api/imports.py index 9a74a16..b6eca1d 100644 --- a/visionstate/backend/visionstate/api/imports.py +++ b/visionstate/backend/visionstate/api/imports.py @@ -12,8 +12,8 @@ from fastapi import APIRouter, File, HTTPException, Request, UploadFile from sqlalchemy import select -from .. import bundle, detectors, faces, imaging -from ..db import Person, keep_change_step, keep_text_reader +from .. import bundle, detectors, faces, imaging, readers +from ..db import Person, keep_change_step, keep_text_reader, keep_wheel_reader from ..engine import Runtime from ..redact import redact from ..settings import ( @@ -72,6 +72,10 @@ def _import_sync(rt: Runtime, path: Path) -> tuple[int, int, bool]: data["reading"] = keep_text_reader(data.get("reading")) # exported before the wheel reader # exported before a counter had a change limit of its own data["reading"] = keep_change_step(data["reading"]) + data["reading"] = keep_wheel_reader(data["reading"]) # exported before a counter named its wheel reader + if isinstance(data["reading"], dict) and data["reading"].get("wheel_reader") not in (None, *readers.WHEEL_READERS): + # exported by a newer version, with a wheel reader this one does not have: the default here + data["reading"] = {**data["reading"], "wheel_reader": ""} try: spec = validated_sensor(data) except ValueError as err: diff --git a/visionstate/backend/visionstate/api/system.py b/visionstate/backend/visionstate/api/system.py index 12c2cc0..4f96d17 100644 --- a/visionstate/backend/visionstate/api/system.py +++ b/visionstate/backend/visionstate/api/system.py @@ -132,12 +132,15 @@ def ui_config() -> dict: "face_unknown_key": FACE_UNKNOWN_KEY, "faces": {k: FACES[k] for k in ("max_people", "max_faces_per_person", "suggest_faces", "stats_window")}, "reading_sensor_defaults": READING_SENSOR_DEFAULTS, - "reading_defaults": READING_DEFAULTS, + "reading_defaults": {**READING_DEFAULTS, "wheel_reader": readers.DEFAULT_WHEEL_READER}, "reading_limits": READING_LIMITS, "counter_standard_steps_per_min": COUNTER_LIMIT["standard_steps_per_min"], "reading_modes": READING_MODES, "reading_displays": READING_DISPLAYS, "counter_readers": COUNTER_READERS, + "wheel_readers": [ + {"id": s.id, "name": s.name, "description": s.description} for s in readers.WHEEL_READERS.values() + ], "reading_device_classes": READING_DEVICE_CLASSES, "reading_counter_cell_share": READING["counter_cell_share"], "reading_export_meter_max_chars": READING["export_meter_max_chars"], @@ -184,8 +187,8 @@ def counts(): "reader": rt.reader.spec.id if rt.reader else None, "reader_name": rt.reader.spec.name if rt.reader else None, "reader_error": rt.reader_error, - "wheel_reader": rt.wheel_reader.spec.id if rt.wheel_reader else None, - "wheel_reader_name": rt.wheel_reader.spec.name if rt.wheel_reader else None, + "wheel_readers": list(rt.wheel_readers), # loaded + "wheel_reader_name": ", ".join(w.spec.name for w in rt.wheel_readers.values()) or None, "wheel_reader_error": rt.wheel_reader_error, # The face model, the people and how many sensors recognise them (the People page). "faces": {**rt.face_status(), "people": people, "sensors": len(face_sensors)}, @@ -262,8 +265,9 @@ def model(spec, kind: str = "") -> dict: "detectors": [model(spec, "detectors") for spec in detectors.DETECTORS.values()], "reader": rt.db.get_text("reader", readers.DEFAULT_READER), "readers": [model(spec, "readers") for spec in readers.READERS.values()], - # Mechanical counters set to the wheel reader; one model for now, so nothing to choose. + # Mechanical counters: each names its wheel reader (new ones get the default). "wheel_reader": readers.DEFAULT_WHEEL_READER, + "wheel_readers_in_use": sorted(rt.wheel_readers_in_use()), "wheel_readers": [model(spec) for spec in readers.WHEEL_READERS.values()], # Faces: the recogniser to choose (one face detector goes with every one); downloaded when # a sensor turns face recognition on. "size" and "installed" count the detector too. diff --git a/visionstate/backend/visionstate/bundle.py b/visionstate/backend/visionstate/bundle.py index 0949342..d5bd928 100644 --- a/visionstate/backend/visionstate/bundle.py +++ b/visionstate/backend/visionstate/bundle.py @@ -107,8 +107,9 @@ def export_sensor(db: Database, storage: Storage, sensor_id: int, target: Path, return f"visionstate-{sensor.slug}.zip" -READINGS_FORMAT = 3 # readings.json; 2 added per-reading reader details, "meter", unchecked readings; -# 3: unchecked rejected readings too, the camera picture's size +READINGS_FORMAT = 5 # readings.json; 2 added per-reading reader details, "meter", unchecked readings; +# 3: unchecked rejected readings too, the camera picture's size; 4: where the region is in the image; +# 5: the wheels' "confidence_rule" ("p" as the confidence counts it; readings without it: the decoder's) READINGS_README = """Readings from a VisionState reading sensor ========================================== @@ -127,8 +128,12 @@ def export_sensor(db: Database, storage: Storage, sensor_id: int, target: Path, the size of the whole camera picture in pixels ("frame_size"; a camera that changes its size after a restart moves what the region shows). For the wheel reader, "wheels" says where it saw each wheel (positions 0.0-9.9, as seen in the -image), how sure it was, its most likely positions, the common shift of all wheels and the -value with the last wheel's fraction. "seconds" is the time since the oldest reading of the +image), how sure it was ("p"), its most likely positions, the common shift of all wheels and the +value with the last wheel's fraction. "confidence_rule" says how the reading's confidence was made +of the wheels' "p": "mean" (their geometric mean) or "weakest" (the smallest; "p" is then the +probability that the wheel stands near that position). Readings made before VisionState 0.10.1 +(beta 0.10.1b3) have no "confidence_rule": their "p" is always the "mean" kind, whichever rule +made the confidence. "seconds" is the time since the oldest reading of the export (no time of day): sorted by it, the readings are in the order they were made. VisionState keeps only the readings that changed the value, the rejected ones and spot checks, so a reading that only repeated the last value is not in it. @@ -144,7 +149,9 @@ def export_sensor(db: Database, storage: Storage, sensor_id: int, target: Path, readings.json ("meter"). Look through the images first; share only what you are happy to make public. By sharing it you release the images and data as public domain (see LICENSE.txt). -Each image is cut with the region the reading was made in ("region": "as read"). Readings +Each image is cut with the region the reading was made in ("region": "as read"), with a margin +that is cut off where the region comes near the edge of the camera picture: "box" is the region +inside the image, in pixels (left, top, right, bottom). Readings from before VisionState 0.8.0 (beta 0.7.1b2) are cut with the region the sensor has now ("region": "current"): if the region was moved since, those images may not show the display at all. """ @@ -176,7 +183,7 @@ def export_readings(db: Database, storage: Storage, sensor_id: int, target: Path if sensor is None: raise KeyError(sensor_id) settings = merge_reading(sensor.reading) - wheels = settings["display"] == "counter" and settings["counter_reader"] == "wheels" + wheels = readers.uses_wheels(settings) mine = (Prediction.sensor_id == sensor_id, Prediction.frame.is_not(None)) newest = Prediction.created_at.desc() rows = [ @@ -201,8 +208,13 @@ def export_readings(db: Database, storage: Storage, sensor_id: int, target: Path frame = imaging.decode(path.read_bytes()) # Cut with the region it was read in; rows from before that was kept use today's region. region_then = "roi" in details - box = imaging.region_box(details["roi"] if region_then else current, READING["export_margin"]) + roi = details["roi"] if region_then else current + box = imaging.region_box(roi, READING["export_margin"]) image = imaging.encode_jpeg(imaging.crop_box(frame, box)) + # Where the region read sits in the image: the margin is cut off at the frame's edges. + width, height = frame.size + read, scale = imaging.region_box(roi), (width, height, width, height) + inside = [int(read[i] * scale[i]) - int(box[i % 2] * scale[i]) for i in range(4)] # Room for readings.json still to come: it compresses to well under a fifth. if items and file.tell() + len(image) + listed // 4 + 100_000 > budget: left_out += 1 @@ -221,6 +233,7 @@ def export_readings(db: Database, storage: Storage, sensor_id: int, target: Path "date": row.created_at.date().isoformat(), # the day only "seconds": round((row.created_at - oldest).total_seconds()) if oldest else 0, "region": "as read" if region_then else "current", + "box": inside, "reader": details.get("reader"), "app": details.get("app"), "greyscale": imaging.is_night(frame), @@ -235,7 +248,7 @@ def export_readings(db: Database, storage: Storage, sensor_id: int, target: Path manifest = { "format": READINGS_FORMAT, "app_version": VERSION, - "reader": readers.DEFAULT_WHEEL_READER if wheels else reader, + "reader": readers.wheel_reader_of(settings) if wheels else reader, "meter": meter.strip()[: READING["export_meter_max_chars"]] or None, # How the sensor reads; nothing about where it is (no camera, name or region position). "settings": { diff --git a/visionstate/backend/visionstate/db.py b/visionstate/backend/visionstate/db.py index a52f432..1c00a55 100644 --- a/visionstate/backend/visionstate/db.py +++ b/visionstate/backend/visionstate/db.py @@ -31,7 +31,7 @@ log = logging.getLogger(__name__) -SCHEMA_VERSION = 15 +SCHEMA_VERSION = 16 # Schema upgrades for existing databases, keyed on the version they upgrade to. # Each step is a list of (table, column, SQL type) columns to add. @@ -81,6 +81,25 @@ def _keep_text_readers(conn: Connection) -> None: ) +def keep_wheel_reader(reading: dict | None) -> dict | None: + """Reading settings of a counter made before a counter named its wheel reader, with the one it + read with kept (wheels-v1; switching is a choice on the Settings tab). Anything else is + returned as it is.""" + if not isinstance(reading, dict) or reading.get("display") != "counter" or "wheel_reader" in reading: + return reading + return {**reading, "wheel_reader": "wheels-v1"} + + +def _keep_wheel_readers(conn: Connection) -> None: + for sensor_id, raw in conn.execute(text("SELECT id, reading FROM sensor WHERE reading IS NOT NULL")).all(): + reading = json.loads(raw) if isinstance(raw, str) else raw + kept = keep_wheel_reader(reading) + if kept is not reading: + conn.execute( + text("UPDATE sensor SET reading = :r WHERE id = :id"), {"r": json.dumps(kept), "id": sensor_id} + ) + + def keep_change_step(reading: dict | None) -> dict | None: """Reading settings of a counter made before change_limit, with the limit it had kept. @@ -111,6 +130,8 @@ def _keep_change_steps(conn: Connection) -> None: 12: _keep_text_readers, # Counters keep the change limit they had (see keep_change_step). 14: _keep_change_steps, + # Counters made before a counter named its wheel reader keep wheels-v1 (see keep_wheel_reader). + 16: _keep_wheel_readers, } diff --git a/visionstate/backend/visionstate/engine/base.py b/visionstate/backend/visionstate/engine/base.py index ed553ee..7dbe829 100644 --- a/visionstate/backend/visionstate/engine/base.py +++ b/visionstate/backend/visionstate/engine/base.py @@ -60,7 +60,8 @@ def __init__(self, settings: Settings, db: Database): self.reader: readers.Reader | None = None # loaded on first use by a reading sensor self.reader_error = "" self._reader_lock = asyncio.Lock() - self.wheel_reader: readers.WheelReader | None = None # loaded on first use by a counter + # Wheel readers by id, each loaded on first use by a counter that reads with it. + self.wheel_readers: dict[str, readers.WheelReader] = {} self.wheel_reader_error = "" self._wheel_reader_lock = asyncio.Lock() # Faces (object sensors): the face model, loaded in the background when a sensor turns face diff --git a/visionstate/backend/visionstate/engine/models.py b/visionstate/backend/visionstate/engine/models.py index 31e27b6..832c8bd 100644 --- a/visionstate/backend/visionstate/engine/models.py +++ b/visionstate/backend/visionstate/engine/models.py @@ -8,10 +8,13 @@ from pathlib import Path from PIL import Image +from sqlalchemy import select from .. import backbones, detectors, imaging, readers +from ..db import Sensor +from ..readers import uses_wheels, wheel_reader_of from ..redact import redact -from ..settings import DETECTION, KIND_OBJECTS, KIND_READING, KIND_STATES +from ..settings import DETECTION, KIND_OBJECTS, KIND_READING, KIND_STATES, merge_reading from .base import RuntimeBase from .logic import filter_detections from .training import TrainingMixin @@ -113,24 +116,38 @@ async def set_reader(self, reader_id: str) -> None: for sensor_id in reading_sensors: self.wake(sensor_id, force=True) - async def ensure_wheel_reader(self) -> readers.WheelReader: - """The wheel reader for mechanical counters, loaded on first use (it is bundled).""" + async def ensure_wheel_reader(self, reader_id: str) -> readers.WheelReader: + """A wheel reader for mechanical counters (each counter names its own), loaded on first use.""" + spec = readers.WHEEL_READERS.get(reader_id) or readers.WHEEL_READERS[readers.DEFAULT_WHEEL_READER] async with self._wheel_reader_lock: - if self.wheel_reader is not None: - return self.wheel_reader - spec = readers.WHEEL_READERS[readers.DEFAULT_WHEEL_READER] + if (loaded := self.wheel_readers.get(spec.id)) is not None: + return loaded try: path = self.model_path(spec) if path is None: path = await asyncio.to_thread(backbones.download, spec, self.settings.models_dir) - self.wheel_reader = await asyncio.to_thread(readers.WheelReader, spec, path) + self.wheel_readers[spec.id] = loaded = await asyncio.to_thread(readers.WheelReader, spec, path) except Exception as err: self.wheel_reader_error = redact(str(err)) log.exception("Could not load wheel reader %s", spec.id) raise self.wheel_reader_error = "" log.info("Wheel reader %s loaded", spec.id) - return self.wheel_reader + return loaded + + def wheel_readers_in_use(self) -> set[str]: + """The wheel readers the counters read with.""" + with self.db.session() as s: + stored = s.scalars(select(Sensor.reading).where(Sensor.kind == KIND_READING)) + return {wheel_reader_of(merge_reading(r)) for r in stored if uses_wheels(merge_reading(r))} + + async def release_wheel_readers(self) -> None: + """Let go of the wheel readers no counter reads with any more (deleted, or switched).""" + in_use = await asyncio.to_thread(self.wheel_readers_in_use) + async with self._wheel_reader_lock: + for reader_id in [k for k in self.wheel_readers if k not in in_use]: + del self.wheel_readers[reader_id] + log.info("Wheel reader %s released", reader_id) async def read_number( self, image: Image.Image, roi: dict | None, reading: dict @@ -138,12 +155,13 @@ async def read_number( """Read the region of ``image``; returns the text and the image the reader used. The text's ``details`` name the reader that read it (``reader``).""" - region = imaging.crop_box(image, imaging.region_box(roi)) - if reading["display"] == "counter" and reading["counter_reader"] == "wheels": - wheels = await self.ensure_wheel_reader() + if uses_wheels(reading): + wheels = await self.ensure_wheel_reader(wheel_reader_of(reading)) + box = imaging.pixel_box(image, imaging.region_box(roi)) async with self._sem: - text, used = await asyncio.to_thread(wheels.read_counter, region, int(reading["digits"])) + text, used = await asyncio.to_thread(wheels.read_counter, image, int(reading["digits"]), box) return replace(text, details={"reader": wheels.spec.id, **(text.details or {})}), used + region = imaging.crop_box(image, imaging.region_box(roi)) reader = await self.ensure_reader() async with self._sem: text, used = await asyncio.to_thread( diff --git a/visionstate/backend/visionstate/engine/runtime.py b/visionstate/backend/visionstate/engine/runtime.py index d8bf7a3..5d628d6 100644 --- a/visionstate/backend/visionstate/engine/runtime.py +++ b/visionstate/backend/visionstate/engine/runtime.py @@ -99,7 +99,8 @@ def release(self) -> None: """After ``stop``, when another runtime takes over in this process (a restored backup): let go of the models, the sensors' heads and the frames kept in memory, and give the memory back, so two sets of models are never loaded at once.""" - self.embedder = self.detector = self.reader = self.wheel_reader = None + self.embedder = self.detector = self.reader = None + self.wheel_readers.clear() self.face_model = self.face_index = None self.heads.clear() self.taught.clear() @@ -153,6 +154,8 @@ async def sensor_updated(self, sensor_id: int, retrain: bool) -> None: await self.refresh_trigger_entities() if cfg.is_objects: await self.faces_settled() + if cfg.kind == KIND_READING: + await self.release_wheel_readers() # a counter may read with another one now self.wake(sensor_id) async def sensor_deleted(self, sensor_id: int, descriptor: SensorDescriptor) -> None: @@ -178,6 +181,8 @@ async def sensor_deleted(self, sensor_id: int, descriptor: SensorDescriptor) -> await self.faces_settled() if descriptor.kind == KIND_READING and not await asyncio.to_thread(self._sensor_ids, KIND_READING): self.reader = None + if descriptor.kind == KIND_READING: + await self.release_wheel_readers() await asyncio.to_thread(backbones.release_memory) def set_global_review(self, rules: dict) -> None: diff --git a/visionstate/backend/visionstate/imaging.py b/visionstate/backend/visionstate/imaging.py index ce23d13..fce8e50 100644 --- a/visionstate/backend/visionstate/imaging.py +++ b/visionstate/backend/visionstate/imaging.py @@ -189,11 +189,16 @@ def region_box(roi: dict | None, margin: float = 0.0) -> Box: ) -def crop_box(image: Image.Image, box: Box) -> Image.Image: - """Crop a normalised box without masking (a detector needs to see whole objects).""" +def pixel_box(image: Image.Image, box: Box) -> tuple[int, int, int, int]: + """A normalised box in the pixels of ``image``.""" x1, y1, x2, y2 = box width, height = image.size - return image.crop((int(x1 * width), int(y1 * height), int(x2 * width), int(y2 * height))) + return int(x1 * width), int(y1 * height), int(x2 * width), int(y2 * height) + + +def crop_box(image: Image.Image, box: Box) -> Image.Image: + """Crop a normalised box without masking (a detector needs to see whole objects).""" + return image.crop(pixel_box(image, box)) def in_region(x: float, y: float, roi: dict | None) -> bool: diff --git a/visionstate/backend/visionstate/readers.json b/visionstate/backend/visionstate/readers.json index ee3363d..7a6a114 100644 --- a/visionstate/backend/visionstate/readers.json +++ b/visionstate/backend/visionstate/readers.json @@ -35,11 +35,11 @@ } }, "wheel_readers": { - "default": "wheels-v1", + "default": "wheels-v2", "readers": { "wheels-v1": { "name": "VisionState wheel reader v1", - "description": "Reads how far each wheel of a mechanical counter has turned; trained for VisionState (tools/wheelreader).", + "description": "Reads how far each wheel has turned, so a wheel between two digits is read right.", "url": "https://huggingface.co/oleost/visionstate-wheel-reader/resolve/42f32402973b3bd3de0258766404232b11e8216f/wheels-v1.onnx", "sha256": "37912f885b061a7b12298872fbdfa1b73ca0664363b293a55617b1d3680a98ac", "size": 2264691, @@ -52,6 +52,25 @@ "ram_mb": 12, "time_ms": 5, "accuracy": 3 + }, + "wheels-v2": { + "name": "VisionState wheel reader v2", + "description": "Reads each wheel alone and with its neighbours, so a region that does not quite fit is still read right; it holds back when any wheel is unclear.", + "url": "https://huggingface.co/oleost/visionstate-wheel-reader/resolve/563f365a5b377c10c7248517ef2576ac5cee3cbf/wheels-v2.onnx", + "sha256": "5644705e1f0e684b30bf3c7a6cb23e7eb1ddda616cd16b5321423f45bf7fd8b6", + "size": 6102872, + "cell_width": 32, + "cell_height": 48, + "bins": 100, + "context": 0.5, + "context_width": 64, + "confidence": "weakest", + "license": "Apache-2.0", + "source": "https://huggingface.co/oleost/visionstate-wheel-reader", + "bundled": true, + "ram_mb": 20, + "time_ms": 12, + "accuracy": 4 } } } diff --git a/visionstate/backend/visionstate/readers.py b/visionstate/backend/visionstate/readers.py index a07f2e1..6db6a1b 100644 --- a/visionstate/backend/visionstate/readers.py +++ b/visionstate/backend/visionstate/readers.py @@ -258,6 +258,9 @@ def decode(probs: np.ndarray, chars: list[str]) -> Text: # --- wheel reader (mechanical counters) ------------------------------------------------------------- +WHEEL_CONFIDENCES = ("mean", "weakest") # see WheelReader.confidence + + @dataclass(frozen=True) class WheelReaderSpec: id: str @@ -273,11 +276,26 @@ class WheelReaderSpec: source: str bundled: bool footprint: Footprint = field(default_factory=lambda: Footprint()) + # A model with a second input: every cell again with this share of a cell on each side (the + # neighbouring wheels, cut from the whole frame: ``wheel_context_cells``), scaled to + # context_width x cell_height. 0 = the cells alone, cut from the region. + context: float = 0.0 + context_width: int = 0 + # The confidence of a read: "mean" (geometric mean of the wheels' decoded probabilities) or + # "weakest" (the weakest wheel's probability near where the value puts it). + confidence: str = "mean" @property def filename(self) -> str: return f"{self.id}.onnx" + def __post_init__(self) -> None: + # A registry mistake fails at start-up (and in the tests), not at a counter's first read. + if self.confidence not in WHEEL_CONFIDENCES: + raise ValueError(f"Wheel reader {self.id}: unknown confidence {self.confidence!r}") + if self.context < 0 or (self.context > 0) != (self.context_width > 0): + raise ValueError(f"Wheel reader {self.id}: context and context_width go together") + def load_wheel_registry() -> tuple[str, dict[str, WheelReaderSpec]]: """The default wheel reader's id and every wheel reader in ``readers.json``.""" @@ -297,15 +315,32 @@ def load_wheel_registry() -> tuple[str, dict[str, WheelReaderSpec]]: source=v.get("source", ""), bundled=bool(v.get("bundled", False)), footprint=Footprint.of(v), + context=float(v.get("context", 0.0)), + context_width=int(v.get("context_width", 0)), + confidence=v.get("confidence", "mean"), ) for key, v in raw["readers"].items() } return raw["default"], specs +# A region in a frame in pixels: left, top, right, bottom. +Pixels = tuple[int, int, int, int] + DEFAULT_WHEEL_READER, WHEEL_READERS = load_wheel_registry() +def uses_wheels(reading: dict) -> bool: + """Whether a reading sensor's settings read a mechanical counter with a wheel reader.""" + return reading.get("display") == "counter" and reading.get("counter_reader") == "wheels" + + +def wheel_reader_of(reading: dict) -> str: + """The wheel reader a counter reads with: its own (``wheel_reader``), else the default.""" + chosen = reading.get("wheel_reader") + return chosen if chosen in WHEEL_READERS else DEFAULT_WHEEL_READER + + def wheel_cells(image: Image.Image, digits: int, width: int, height: int) -> np.ndarray: """The region split into ``digits`` equal cells, each as the wheel model sees it. @@ -322,6 +357,32 @@ def wheel_cells(image: Image.Image, digits: int, width: int, height: int) -> np. return np.stack(out) +def wheel_context_cells( + image: Image.Image, box: Pixels, digits: int, width: int, height: int, context: float +) -> np.ndarray: + """The region ``box`` (pixels in ``image``) split into ``digits`` equal cells, each with + ``context`` of a cell on both sides, as the wheel model sees them. + + The context comes from the picture around the region (beyond the picture's edge, the edge + is repeated), so a wheel the region cuts is still seen whole. Grey, scale and contrast as + ``wheel_cells``. + """ + x0, y0, x1, y1 = box + count = max(1, digits) + cell = (x1 - x0) / count + pad = math.ceil(cell * context) + 2 + grey = np.pad(np.asarray(image.convert("RGB")).min(axis=2), ((0, 0), (pad, pad)), mode="edge") + wide = Image.fromarray(grey) + out = [] + for i in range(count): + a = round(x0 + (i - context) * cell + pad) + b = max(a + 1, round(x0 + (i + 1 + context) * cell + pad)) + part = wide.crop((a, y0, b, max(y0 + 1, y1))) + part = ImageOps.autocontrast(part.resize((width, height), Image.Resampling.BILINEAR), cutoff=1) + out.append(np.asarray(part, dtype=np.float32) / 255.0) + return np.stack(out) + + def wheel_value(logp: np.ndarray) -> tuple[float, float]: """The most likely counter value for per-wheel position log probabilities (wheels x 100 bins). @@ -360,42 +421,70 @@ def wheel_decode(logp: np.ndarray) -> tuple[float, float, list[int]]: return value * 10 + at[-1] / tenth, score, at -def wheel_details(probs: np.ndarray, logp: np.ndarray, value: float, score: float, at: list[int], shift: int) -> dict: +def wheel_details( + probs: np.ndarray, p: list[float], rule: str, value: float, score: float, at: list[int], shift: int +) -> dict: """How a wheel reading was found, in positions (0.0-9.9): per wheel where the value puts it - as seen in the image (``at``, with the common ``shift``), the probability the decoder used - there (``p``, a little slack included) and the model's most likely positions (``top``); - the value in steps of the last wheel with its fraction, and its log probability.""" + as seen in the image (``at``, with the common ``shift``), its probability as the confidence + counts it (``p``, by the spec's ``confidence_rule``: ``WheelReader.wheel_p``) and the model's + most likely positions (``top``); the value in steps of the last wheel with its fraction, and + its log probability.""" bins = probs.shape[1] tenth = bins // 10 top = READING["export_wheel_top"] wheels = [] for k, b in enumerate(at): - seen = (b + shift) % bins order = np.argsort(probs[k])[::-1][:top] wheels.append( { - "at": round(seen / tenth, 1), - "p": round(float(np.exp(logp[k, seen])), 4), + "at": round((b + shift) % bins / tenth, 1), + "p": round(p[k], 4), "top": [[round(int(i) / tenth, 1), round(float(probs[k, i]), 4)] for i in order], } ) - return {"value": round(value, 1), "shift": round(shift / tenth, 1), "log_p": round(score, 3), "per_wheel": wheels} + return { + "value": round(value, 1), + "shift": round(shift / tenth, 1), + "log_p": round(score, 3), + "confidence_rule": rule, + "per_wheel": wheels, + } class WheelReader: - """Reads a mechanical counter wheel by wheel: one small ONNX model gives each wheel's position.""" + """Reads a mechanical counter wheel by wheel: one small ONNX model gives each wheel's position. + + ``image`` and ``box`` below: the frame and the region in it in pixels (no ``box``: the image + is the region). Only a model with context (``WheelReaderSpec.context``) looks beyond the region. + """ def __init__(self, spec: WheelReaderSpec, model_path: Path): self.spec = spec self.session = cpu_session(model_path) - self.input_name = self.session.get_inputs()[0].name + # The cells, then (with context) the cells with their neighbours. + self.input_names = [i.name for i in self.session.get_inputs()] + # The first output is the positions; a later one (such as each cell's embedding) is not used yet. + self.output_name = self.session.get_outputs()[0].name + if len(self.input_names) != (2 if spec.context else 1): + raise ValueError(f"Wheel reader {spec.id}: the model takes {len(self.input_names)} inputs") self._lock = threading.Lock() - def probabilities(self, image: Image.Image, digits: int) -> np.ndarray: + def inputs(self, image: Image.Image, digits: int, box: Pixels | None = None) -> list[np.ndarray]: + """What the model is given: the cells, and with context the cells with their neighbours + (each cells x height x width, 0..1).""" + h = self.spec.cell_height + out = [wheel_cells(image.crop(box) if box else image, digits, self.spec.cell_width, h)] + if self.spec.context: + whole = box or (0, 0, *image.size) + out.append(wheel_context_cells(image, whole, digits, self.spec.context_width, h, self.spec.context)) + return out + + def probabilities(self, image: Image.Image, digits: int, box: Pixels | None = None) -> np.ndarray: """The model's probability of every wheel's position (wheels x bins).""" - cells = wheel_cells(image, digits, self.spec.cell_width, self.spec.cell_height) + feed = {name: x[:, None] for name, x in zip(self.input_names, self.inputs(image, digits, box), strict=True)} with self._lock: - logits = np.asarray(self.session.run(None, {self.input_name: cells[:, None]})[0], dtype=np.float64) + out = self.session.run([self.output_name], feed) + logits = np.asarray(out[0], dtype=np.float64) logits -= logits.max(axis=1, keepdims=True) probs = np.exp(logits) return probs / probs.sum(axis=1, keepdims=True) @@ -407,19 +496,19 @@ def with_slack(probs: np.ndarray) -> np.ndarray: near = sum(np.roll(probs, s, axis=1) for s in range(-slack, slack + 1)) return np.log(np.maximum(near, 1e-12)) - def positions(self, image: Image.Image, digits: int) -> np.ndarray: + def positions(self, image: Image.Image, digits: int, box: Pixels | None = None) -> np.ndarray: """Log probabilities of every wheel's position (wheels x bins), as decoded.""" - return self.with_slack(self.probabilities(image, digits)) + return self.with_slack(self.probabilities(image, digits, box)) - def read_counter(self, image: Image.Image, digits: int) -> tuple[Text, Image.Image]: + def read_counter(self, image: Image.Image, digits: int, box: Pixels | None = None) -> tuple[Text, Image.Image]: """Read ``digits`` wheels; returns the read (all digits, last wheel rounded) and the cells used. A region drawn a little above or below the digits shifts every wheel alike, so a few common shifts are tried (each costs a little) and the most likely reading wins. The - confidence is the mean probability per wheel of the positions read. ``details`` keeps - how the value was found (``wheel_details``), so it can be checked without the model. + confidence is the spec's (``WheelReaderSpec.confidence``). ``details`` keeps how the + value was found (``wheel_details``), so it can be checked without the model. """ - probs = self.probabilities(image, digits) + probs = self.probabilities(image, digits, box) logp = self.with_slack(probs) best: tuple[float, float, list[int], int] | None = None for shift in range(-READING["wheel_max_shift_bins"], READING["wheel_max_shift_bins"] + 1): @@ -430,21 +519,44 @@ def read_counter(self, image: Image.Image, digits: int) -> tuple[Text, Image.Ima assert best is not None value, score, at, shift = best number = math.floor(value + 0.5) % 10**digits + p = self.wheel_p(probs, logp, at, shift) text = Text( f"{number:0{digits}d}", - math.exp(score / max(1, digits)), - {"wheels": wheel_details(probs, logp, value, score, at, shift)}, + self.confidence(score, p), + {"wheels": wheel_details(probs, p, self.spec.confidence, value, score, at, shift)}, ) - return text, self.cells_image(image, digits) - - def cells_image(self, image: Image.Image, digits: int) -> Image.Image: - """The cells side by side as the model sees them (for the "what the reader saw" image).""" - cells = wheel_cells(image, digits, self.spec.cell_width, self.spec.cell_height) + return text, self.cells_image(image, digits, box) + + def wheel_p(self, probs: np.ndarray, logp: np.ndarray, at: list[int], shift: int) -> list[float]: + """Every wheel's probability at where the value puts it, as the confidence counts it: + "mean" the decoder's (a little slack included), "weakest" the mass within + ``wheel_weakest_bins``.""" + bins = probs.shape[1] + if self.spec.confidence != "weakest": + return [float(np.exp(logp[k, (b + shift) % bins])) for k, b in enumerate(at)] + near = np.arange(-READING["wheel_weakest_bins"], READING["wheel_weakest_bins"] + 1) + return [float(probs[k, (b + shift + near) % bins].sum()) for k, b in enumerate(at)] + + def confidence(self, score: float, p: list[float]) -> float: + """How sure the read is, from the decoder's ``score`` and the wheels' ``wheel_p``: "mean" + is the geometric mean of the wheels' decoded probabilities (the score, shift penalty + included); "weakest" the smallest (one unclear wheel is not hidden by the clear ones).""" + if self.spec.confidence != "weakest": + return math.exp(score / max(1, len(p))) + return min(p) + + def cells_image(self, image: Image.Image, digits: int, box: Pixels | None = None) -> Image.Image: + """The cells side by side as the model sees them (for the "what the reader saw" image): + one wheel per field, so a region that cuts the wheels wrong shows. A model with context + also sees the neighbours around each cell; they are left out here, as they would look + like every wheel twice.""" + crop = image.crop(box) if box else image + cells = (wheel_cells(crop, digits, self.spec.cell_width, self.spec.cell_height) * 255).astype(np.uint8) gap = 2 - w, h = self.spec.cell_width, self.spec.cell_height + h, w = cells.shape[1:] line = Image.new("L", (len(cells) * (w + gap) - gap, h), 255) for i, cell in enumerate(cells): - line.paste(Image.fromarray((cell * 255).astype(np.uint8)), (i * (w + gap), 0)) + line.paste(Image.fromarray(cell), (i * (w + gap), 0)) return line.resize((line.width * 2, h * 2), Image.Resampling.NEAREST).convert("RGB") diff --git a/visionstate/backend/visionstate/settings.py b/visionstate/backend/visionstate/settings.py index 08984db..b1e377e 100644 --- a/visionstate/backend/visionstate/settings.py +++ b/visionstate/backend/visionstate/settings.py @@ -94,6 +94,9 @@ "display": "auto", "digits": 6, # "counter" display only: wheels inside the region; other digit counts are rejected "counter_reader": "wheels", # "counter" display only: see COUNTER_READERS + # The wheel reader (an id in readers.json) a counter read by "wheels" uses. A new sensor + # stores the registry's default, so a later default never changes it; "" = that default. + "wheel_reader": "", # Values and times left: the largest plausible change between two readings (0 = no limit). # Counters use change_limit instead (see COUNTER_LIMIT). "max_step": 0.0, @@ -191,6 +194,10 @@ "wheel_slack_bins": 1, "wheel_max_shift_bins": 2, "wheel_shift_penalty": 0.3, + # Wheel readers whose confidence is the weakest wheel ("confidence": "weakest" in the + # registry): a wheel's share is the model's probability within this many bins of where the + # value puts it. + "wheel_weakest_bins": 4, "accepted_window_s": 86_400, # the "accepted" entity: share of the readings in the last 24 h "rate_min_span_s": 30.0, # no rate until two accepted readings are at least this far apart # "Export readings" (to share, e.g. on GitHub): the checked readings, then the accepted ones diff --git a/visionstate/config.yaml b/visionstate/config.yaml index 4a40245..fdc4bae 100644 --- a/visionstate/config.yaml +++ b/visionstate/config.yaml @@ -1,5 +1,5 @@ name: VisionState -version: "0.10.0" +version: "0.10.1" slug: visionstate description: Teach a local AI to recognise states (open, closed, …), find objects (people, cars, pets) or read numbers in your camera images url: https://github.com/oleost/VisionState diff --git a/visionstate/frontend/e2e/seed.setup.ts b/visionstate/frontend/e2e/seed.setup.ts index f8108e9..2e52905 100644 --- a/visionstate/frontend/e2e/seed.setup.ts +++ b/visionstate/frontend/e2e/seed.setup.ts @@ -107,9 +107,10 @@ setup('seed a mechanical counter sensor', async ({ request }) => { }); expect(created.ok()).toBeTruthy(); const sensor = await created.json(); + // The last wheel stands exactly half way from 9 to 0: either neighbour is right. await expect .poll(async () => (await (await request.get(`api/v1/sensors/${sensor.id}`)).json()).reading.value, { timeout: 60_000 }) - .toBe('89.939'); + .toMatch(/^89\.9(39|40)$/); }); setup('seed a sensor that recognises faces', async ({ request }) => { diff --git a/visionstate/frontend/e2e/ui.spec.ts b/visionstate/frontend/e2e/ui.spec.ts index 38aaaee..cd12571 100644 --- a/visionstate/frontend/e2e/ui.spec.ts +++ b/visionstate/frontend/e2e/ui.spec.ts @@ -402,13 +402,13 @@ test('new mechanical counter sensor through the wizard', async ({ page, request // Read by the wheel reader (the default), which reads every field; the text reader instead // reads the digits it sees: told there are eight wheels, the seven digits it reads are not accepted. const readWith = page.getByLabel('Read with'); - await expect(readWith).toHaveValue('wheels'); + await expect(readWith).toHaveValue('wheels:wheels-v2'); await readWith.selectOption('ocr'); await expect(page.getByText(/a turning wheel can be misread/)).toBeVisible(); await page.getByLabel('Number of digits').fill('8'); await expect(page.getByText(/digits, not 8/)).toBeVisible({ timeout: 30_000 }); await page.getByLabel('Number of digits').fill('7'); - await readWith.selectOption('wheels'); + await readWith.selectOption('wheels:wheels-v2'); await expect(page.getByText('45.730 m³')).toBeVisible({ timeout: 30_000 }); await expect(page.getByText('(reading…)')).toHaveCount(0, { timeout: 30_000 }); await expectNoHorizontalOverflow(page); @@ -1145,11 +1145,14 @@ test('a rejected reading keeps the region it was read in', async ({ page, reques } }); -test('the wheel reader shows in Settings and is offered to counters read as text', async ({ page, request }, info) => { +test('the wheel reader shows in Settings and is offered to counters read as text or with an older one', async ({ + page, + request, +}, info) => { const errors = watchErrors(page); await page.goto('#/settings'); const wheels = page.getByTestId('wheel-reader'); - await expect(wheels).toContainText('VisionState wheel reader v1'); + await expect(wheels).toContainText('VisionState wheel reader v2'); await expect(wheels.getByRole('link', { name: 'source' })).toHaveAttribute('href', /huggingface\.co/); await expect(page.getByText('Wheel reader', { exact: true })).toBeVisible(); // its status row await expectNoHorizontalOverflow(page); @@ -1178,6 +1181,33 @@ test('the wheel reader shows in Settings and is offered to counters read as text await page.goto('about:blank'); await request.delete(`api/v1/sensors/${id}`); } + // A counter set up with wheels-v1 keeps it, and its Live tab offers the newer one. + const older = await request.post('api/v1/sensors', { + data: { + name: 'Older water meter', + kind: 'reading', + source_type: 'http', + source: COUNTER_URL(45730), + roi: COUNTER_BOX, + reading: { mode: 'counter', display: 'counter', digits: 7, decimals: 3, unit: 'm³', wheel_reader: 'wheels-v1' }, + interval_s: 3600, + }, + }); + const olderId = (await older.json()).id; + try { + await page.goto(`#/sensors/${olderId}/live`); + const newer = page.getByTestId('try-newer-wheel-reader'); + await expect(newer).toContainText('VisionState wheel reader v2'); + await expectNoHorizontalOverflow(page); + await expectNoClipping(page, '.notice'); + await page.screenshot({ path: path.join('test-results', 'pages', info.project.name, 'live-newer-wheel-reader.png'), fullPage: true }); + await press(newer.getByRole('link'), info); + await expect(page.getByLabel('Read with')).toHaveValue('wheels:wheels-v1'); + errors.expectNone(); + } finally { + await page.goto('about:blank'); + await request.delete(`api/v1/sensors/${olderId}`); + } }); test('a backup of everything is downloaded and restored', async ({ page, request }, info) => { diff --git a/visionstate/frontend/src/lib/components/ModelCompare.svelte b/visionstate/frontend/src/lib/components/ModelCompare.svelte index fa73f09..7ad250e 100644 --- a/visionstate/frontend/src/lib/components/ModelCompare.svelte +++ b/visionstate/frontend/src/lib/components/ModelCompare.svelte @@ -15,14 +15,11 @@ let open = $state(false); type Kind = (typeof MODEL_KINDS)[number]['key']; - const chosen = (kind: Kind) => - ({ - backbones: info.backbone, - detectors: info.detector, - face_models: info.face_model, - readers: info.reader, - wheel_readers: info.wheel_reader, - })[kind]; + // Counters each name their wheel reader, so several can be in use. + const chosen = (kind: Kind, id: string) => + kind === 'wheel_readers' + ? info.wheel_readers_in_use.includes(id) + : id === { backbones: info.backbone, detectors: info.detector, face_models: info.face_model, readers: info.reader }[kind]; const deletable = (kind: Kind): kind is ModelKind => kind !== 'wheel_readers'; async function remove(kind: ModelKind, m: BackboneInfo) { @@ -49,9 +46,9 @@ {kind.title} {kind.text} {#each bySize(info[kind.key] as BackboneInfo[]) as m (m.id)} -
+
- {m.name}{#if m.id === chosen(kind.key)} · in use{/if} + {m.name}{#if chosen(kind.key, m.id)} · in use{/if} ~{ram(m)} RAM diff --git a/visionstate/frontend/src/lib/components/ReadingEditor.svelte b/visionstate/frontend/src/lib/components/ReadingEditor.svelte index b234277..28a806e 100644 --- a/visionstate/frontend/src/lib/components/ReadingEditor.svelte +++ b/visionstate/frontend/src/lib/components/ReadingEditor.svelte @@ -17,6 +17,22 @@ const displays = $derived((app.config?.reading_displays ?? []).filter((d) => d !== 'counter')); const digitLimits = $derived(app.config?.reading_limits.digits ?? [1, 12]); const counterReaders = $derived(app.config?.counter_readers ?? []); + // "Read with": every wheel reader (the default first) and the text reader in one list; with one + // wheel reader, that one is simply "wheels". + const defaultWheels = $derived(app.config?.reading_defaults.wheel_reader ?? ''); + const wheelReaders = $derived( + [...(app.config?.wheel_readers ?? [])].sort( + (a, b) => Number(b.id === defaultWheels) - Number(a.id === defaultWheels), + ), + ); + const severalWheels = $derived(wheelReaders.length > 1); + const wheelsOption = (id: string) => (severalWheels ? `wheels:${id}` : 'wheels'); + const wheelsChosen = $derived( + value.counter_reader === 'wheels' + ? (wheelReaders.find((w) => w.id === (value.wheel_reader || defaultWheels)) ?? wheelReaders[0]) + : undefined, + ); + const readWith = $derived(wheelsChosen ? wheelsOption(wheelsChosen.id) : value.counter_reader); const deviceClasses = $derived(app.config?.reading_device_classes ?? []); const decimals = $derived(app.config?.reading_limits.decimals ?? [0, 4]); const suggested = $derived.by(() => { @@ -37,6 +53,13 @@ } } + function setReadWith(choice: string) { + const [reader, wheels] = choice.split(':'); + value.counter_reader = reader as ReadingSettings['counter_reader']; + if (wheels) value.wheel_reader = wheels; + else if (reader === 'wheels' && wheelReaders[0]) value.wheel_reader = wheelReaders[0].id; + } + function setMode(mode: ReadingMode) { value.mode = mode; if (mode === 'time_left') value.device_class = 'duration'; @@ -137,14 +160,18 @@ {:else}

A counter that reads lower than before is rejected; under Sensor output you can also limit how much it diff --git a/visionstate/frontend/src/lib/types.ts b/visionstate/frontend/src/lib/types.ts index 421ca97..c3c0a87 100644 --- a/visionstate/frontend/src/lib/types.ts +++ b/visionstate/frontend/src/lib/types.ts @@ -115,6 +115,8 @@ export interface ReadingSettings { digits: number; /** Mechanical counters only: what reads the wheels. */ counter_reader: CounterReader; + /** counter_reader "wheels": the wheel reader's id ("" = the default). */ + wheel_reader: string; /** Values and times left, and counters with change_limit "step": the largest change between two * readings (0 = no limit). */ max_step: number; @@ -560,6 +562,8 @@ export interface AppConfig { reading_modes: ReadingMode[]; reading_displays: ReadingDisplay[]; counter_readers: CounterReader[]; + /** The wheel readers a counter can read with (the default is in reading_defaults). */ + wheel_readers: { id: string; name: string; description: string }[]; reading_device_classes: string[]; /** Mechanical counters: the share of each digit field's width that is read. */ reading_counter_cell_share: number; @@ -586,8 +590,8 @@ export interface Status { reader: string | null; reader_name: string | null; reader_error: string; - /** Mechanical counters read wheel by wheel; loaded on first use. */ - wheel_reader: string | null; + /** Mechanical counters read wheel by wheel: the wheel readers loaded (each on first use). */ + wheel_readers: string[]; wheel_reader_name: string | null; wheel_reader_error: string; /** The face model, and how many people and sensors that recognise faces there are. */ @@ -769,9 +773,11 @@ export interface SettingsInfo { detectors: DetectorInfo[]; reader: string; readers: DetectorInfo[]; - /** The wheel reader for mechanical counters (one model for now, nothing to choose). */ + /** The wheel reader new counters get; each counter names its own. */ wheel_reader: string; wheel_readers: DetectorInfo[]; + /** The wheel readers counters read with. */ + wheel_readers_in_use: string[]; /** Recognising faces on object sensors: the recogniser chosen; downloaded when a sensor turns faces on. */ face_model: string; face_models: FaceModelInfo[]; diff --git a/visionstate/frontend/src/pages/Settings.svelte b/visionstate/frontend/src/pages/Settings.svelte index a238470..9698691 100644 --- a/visionstate/frontend/src/pages/Settings.svelte +++ b/visionstate/frontend/src/pages/Settings.svelte @@ -294,8 +294,8 @@ {wheelReader.name} · included {wheelReader.description} {wheelReader.license}, - source. Used by counters set to - Read with → Wheel reader (the default).source. New counters read with it + (Read with on a counter's Settings tab).

{/if} diff --git a/visionstate/frontend/src/pages/sensor/ReadingLiveTab.svelte b/visionstate/frontend/src/pages/sensor/ReadingLiveTab.svelte index 443717c..aba40c3 100644 --- a/visionstate/frontend/src/pages/sensor/ReadingLiveTab.svelte +++ b/visionstate/frontend/src/pages/sensor/ReadingLiveTab.svelte @@ -1,7 +1,7 @@