Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -128,6 +128,14 @@ this project uses [semantic versioning](https://semver.org/).
One sequence and one window. [ADR-0023](docs/adr/0023-clone-against-a-measured-front-end.md) is accepted: the clone
stays opt-in, with the inflation.

- A wider real-front-end measurement ([`docs/real_front_end.md`](docs/real_front_end.md)): six windows of five Machine
Hall sequences, the design committed before the run, the measured errors of each window replayed into the stochastic
clone. At the measured noise the clone meets the gate in two of six windows, at four times in all six, and the single
anchor in none. The first window was also run again with the run aligned to its frames (the earlier run began 1.08 s
after them, so the last second had no vision), which moved its figures slightly. `navkit euroc vo --wide` runs it.
[ADR-0024](docs/adr/0024-assumed-noise-for-the-opt-in-clone.md) is proposed; the decision section is for the
maintainer to write.

### Changed

- The implementation plan carries the same point-in-time banner as the other baseline records. P5-09 and P5-12 are
Expand Down
8 changes: 4 additions & 4 deletions CONSTRAINTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,9 @@ Invariants this project holds itself to. Every entry is either currently true
or is a stated blocker; nothing here is aspirational. Ratchets are measured,
not estimated, and the command to re-measure each one is given.

Last measured: <!-- metric:tests_collected -->1302<!-- /metric --> tests collected, all of which pass except the deliberate
Last measured: <!-- metric:tests_collected -->1313<!-- /metric --> tests collected, all of which pass except the deliberate
xfails and the skips that `pytest -rs` lists, and <!-- metric:coverage_percent -->94.3<!-- /metric -->%
line coverage (<!-- metric:coverage_lines_hit -->9819<!-- /metric -->/<!-- metric:coverage_lines_total -->10408<!-- /metric --> executable lines) on **CPython <!-- metric:coverage_python -->3.13<!-- /metric -->**, measured
line coverage (<!-- metric:coverage_lines_hit -->10009<!-- /metric -->/<!-- metric:coverage_lines_total -->10614<!-- /metric --> executable lines) on **CPython <!-- metric:coverage_python -->3.13<!-- /metric -->**, measured
with `python scripts/coverage_report.py`. The figures come from `docs/data/metrics.json`, which `scripts/metrics.py` checks in CI. Coverage is quoted from that script alone:
it is the project's dependency-free tracer (S1 forbids adding `pytest-cov`), and
a number taken from a different tool is not comparable with the module figures
Expand Down Expand Up @@ -83,7 +83,7 @@ These are floors, not goals. Each must not regress; raising one is welcome.

| Ratchet | Floor | Current (CPython 3.13) | Re-measure with |
|---|---:|---:|---|
| Tests collected | 405 | <!-- metric:tests_collected -->1302<!-- /metric --> | `pytest`, `scripts/metrics.py --check` |
| Tests collected | 405 | <!-- metric:tests_collected -->1313<!-- /metric --> | `pytest`, `scripts/metrics.py --check` |
| Tests passing | 300 | all collected, except the deliberate xfails and the listed skips | `pytest -rs` |
| Line coverage | 75% | <!-- metric:coverage_percent -->94.3<!-- /metric -->% | `scripts/coverage_report.py`, `scripts/metrics.py --check-coverage` |
| `io/trajectory.py` coverage | 85% | 95.0% | as above |
Expand All @@ -104,7 +104,7 @@ These are floors, not goals. Each must not regress; raising one is welcome.
| `outage_sweep.py` coverage | 80% | 98.4% | as above |
| `fdir/nis_monitor.py` coverage | 80% | 98.8% | as above |
| `degrade/` coverage | 50% | 78.0% (`config.py`) / 84.2% (`inject.py`) | as above |
| Docs | README + architecture + calibration + every ADR | <!-- metric:adr_count -->23<!-- /metric --> ADRs, all in the index | `scripts/metrics.py --check` |
| Docs | README + architecture + calibration + every ADR | <!-- metric:adr_count -->24<!-- /metric --> ADRs, all in the index | `scripts/metrics.py --check` |
| Documented tables match the generated benchmark | exact | yes | `scripts/check_doc_tables.py` |
| Open blockers documented | all | see ROADMAP | manual |

Expand Down
2 changes: 1 addition & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,7 @@ mypy src --ignore-missing-imports
`ruff format src tests scripts` applies formatting; the `--check` form is the
gate.

The suite collects <!-- metric:tests_collected -->1302<!-- /metric --> tests and all of them pass on a fresh clone, except the deliberate
The suite collects <!-- metric:tests_collected -->1313<!-- /metric --> tests and all of them pass on a fresh clone, except the deliberate
xfails and the skips. `pytest -rs` lists the skips: two need TUM VI reference data that is not
vendored (see below), and the doc-table check runs once `python scripts/run_benchmark.py` has
written `results/benchmark.json`. The count and the other numbers the docs quote are in
Expand Down
10 changes: 6 additions & 4 deletions ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ Sequenced by dependency, not by ambition. Each item names its exit test.
Status markers: `[x]` done, `[~]` in progress, `[ ]` not started, `[!]`
blocked.

Measured baseline: <!-- metric:tests_collected -->1302<!-- /metric --> tests collected and <!-- metric:coverage_percent -->94.3<!-- /metric -->% line coverage on CPython
Measured baseline: <!-- metric:tests_collected -->1313<!-- /metric --> tests collected and <!-- metric:coverage_percent -->94.3<!-- /metric -->% line coverage on CPython
<!-- metric:coverage_python -->3.13<!-- /metric --> (`docs/data/metrics.json`, from `scripts/metrics.py` and `scripts/coverage_report.py`). Every
number below re-measures against that, not against a remembered value.

Expand Down Expand Up @@ -260,9 +260,11 @@ this removes it.
- [~] A visual front end with correlated errors, scale ambiguity and outliers.
Outliers and scale drift are survived. Correlated errors are not, unless the
assumed visual noise is inflated by a factor found by trial. A plain two-view front end was
then measured on real frames ([real front end](docs/real_front_end.md), ADR-0023): its errors are
correlated for about 0.25 s, and when they are replayed the clone stays inside the gate only with the assumed
noise inflated by four. One sequence and one window, so the default does not change.
then measured on real frames ([real front end](docs/real_front_end.md); [ADR-0023](docs/adr/0023-clone-against-a-measured-front-end.md) and
[ADR-0024](docs/adr/0024-assumed-noise-for-the-opt-in-clone.md), the second proposed): its
errors are correlated for 0.05 s to 0.5 s, and when they are replayed the clone stays inside the gate in all six
windows of five Machine Hall sequences only with the assumed noise inflated by four. One dataset and one
front end, so the default does not change.

**Exit test:** NEES below 10 with vision enabled and GNSS denied, coverage above
90%, `vision_enabled = True` as the shipped default. Current value 286.2 for the
Expand Down
2 changes: 2 additions & 0 deletions docs/adr/0023-clone-against-a-measured-front-end.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,3 +30,5 @@ configuration. The next step is more sequences and windows, so that the replay r
The finding is limited to this front end, this window and a known translation scale. The factor of four is a trial
value that held in one replay, not a derived constant. A front end with another error structure may need another
factor.

Update: the wider measurement ran, and its result is in [ADR-0024](0024-assumed-noise-for-the-opt-in-clone.md).
35 changes: 35 additions & 0 deletions docs/adr/0024-assumed-noise-for-the-opt-in-clone.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
# ADR-0024: The assumed visual noise of the opt-in clone

- Status: proposed
- Date: 2026-10
- Related: [ADR-0017](0017-stochastic-clone-for-the-visual-update.md), [ADR-0023](0023-clone-against-a-measured-front-end.md), [real front end](../real_front_end.md)

## Context

ADR-0023 kept the stochastic clone opt-in and asked for more windows and sequences before anything changed. The wider
measurement ran: six windows of five Machine Hall sequences, a plain two-view front end, the measured errors of each
window replayed into the filter. With the assumed visual noise at the measured size, the clone meets the gate (NEES below
10, 2-sigma coverage above 90%) in two of six windows. At twice the measured size, in two of six. At four times, in all
six. At eight times, in all six. The single anchor meets it in none. The tightest window at four times has a coverage of
91%.

All of it is one dataset, one front end that is not a production one, and a translation with the true scale.

## Options

1. **Keep the clone opt-in and record four times as the supported assumption.** `vision_enabled` stays False. The docs
say that an opt-in user of a feature-tracking front end should assume four times the measured visual noise, and that
this holds on the windows measured so far.
2. **Add the factor to the configuration as a named setting.** A documented preset so that the opt-in path does not
depend on a number found in a page. Code and docs change; the default does not.
3. **Make the clone the default with four times the noise.** It needs a monocular scale, a second dataset and a front end
that is not mine first.

## Decision

To be written by the maintainer.

## Consequences

Whatever is chosen, the claim stays limited to the Machine Hall windows measured, this front end and a known scale. The
factor of four is an empirical value, not a derived constant.
1 change: 1 addition & 0 deletions docs/adr/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,3 +28,4 @@ what it cost, and what was rejected.
| [ADR-0021](0021-fault-classification-from-innovations.md) | Fault classification from the GNSS innovations | accepted |
| [ADR-0022](0022-multipath-signature.md) | A multipath signature on the GNSS innovations | accepted |
| [ADR-0023](0023-clone-against-a-measured-front-end.md) | The clone against a measured front end | accepted |
| [ADR-0024](0024-assumed-noise-for-the-opt-in-clone.md) | The assumed visual noise of the opt-in clone | proposed |
10 changes: 5 additions & 5 deletions docs/data/metrics.json
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
{
"adr_count": 23,
"coverage_lines_hit": 9819,
"coverage_lines_total": 10408,
"coverage_percent": 94.34,
"adr_count": 24,
"coverage_lines_hit": 10009,
"coverage_lines_total": 10614,
"coverage_percent": 94.3,
"coverage_python": "3.13",
"tests_collected": 1302
"tests_collected": 1313
}
30 changes: 15 additions & 15 deletions docs/data/vo_clone_check.csv
Original file line number Diff line number Diff line change
Expand Up @@ -24,18 +24,18 @@ IMU only,4,1.479515039799907,1.0,64.33376427977204,64.33376427977204,48.22398143
"clone, measured correlation, noise assumed x4",2,1.1598820006352226,1.0,0.3853541069446335,1.0644072764533532,0.6663882330282552
"clone, measured correlation, noise assumed x4",3,1.3663597214021872,1.0,0.8468382431054775,1.8443749062870949,0.6770825779385757
"clone, measured correlation, noise assumed x4",4,0.9313382116502497,1.0,0.3461272565693066,0.8911346077089927,0.6696683908514616
"replay, anchor",0,162853.3177202277,0.045,238.5619630685604,238.5619630685604,0.2738550680289662
"replay, anchor",1,3902.892646131298,0.22133333333333333,40.40266180548525,40.40266180548525,0.27385520648124717
"replay, anchor",2,191652.38118501462,0.0375,259.8637367849311,259.8637367849311,0.2738551681094766
"replay, anchor",3,23497.854123016754,0.05,93.63839724696794,93.63839724696794,0.27385521817424435
"replay, anchor",4,15496.151456115414,0.09766666666666667,66.38652929287211,66.38652929287211,0.2738550878280069
"replay, clone",0,34.587376134194805,0.378,0.4410105430927355,2.0679822624887816,0.2736007756934708
"replay, clone",1,26.97402561069759,0.5571666666666667,0.3076255905617748,2.0177121447026427,0.27360843161141396
"replay, clone",2,30.985558012009587,0.535,0.24979233572188436,2.124069403692931,0.2736292940411207
"replay, clone",3,26.427569590150807,0.568,0.1168379946950831,1.9991510229137401,0.27359348189691307
"replay, clone",4,23.426648244921132,0.5501666666666667,0.05654685195852756,1.8729005865056703,0.27360567393794893
"replay, clone, noise assumed x4",0,4.967967593940472,0.895,0.2982003603348016,2.315065365306037,0.6728826925288418
"replay, clone, noise assumed x4",1,4.699123780634335,0.8981666666666667,0.8655240935112204,2.315638468206952,0.6703569638405952
"replay, clone, noise assumed x4",2,4.305557510468603,0.9071666666666667,0.7537660844255686,2.3080299325352716,0.6682262200970166
"replay, clone, noise assumed x4",3,3.9705814909040975,0.9166666666666666,0.7721493615124901,2.208643976825519,0.6729112733384796
"replay, clone, noise assumed x4",4,3.393820812165554,0.9923333333333333,0.6021227381077545,2.1003063486798905,0.6691948130013253
"replay, anchor",0,116531.45162424895,0.05515747375437427,188.16518312586584,188.16518312586584,0.27151730613749975
"replay, anchor",1,6256.248351268813,0.21179803366105648,45.44315739524699,45.44315739524699,0.27151758056193304
"replay, anchor",2,139207.26769054652,0.04849191801366439,204.36024455280514,204.36024455280514,0.27237845096024843
"replay, anchor",3,16102.044859582255,0.058656890518246955,72.40350579411398,72.40350579411398,0.27151768414633365
"replay, anchor",4,2871.7391727054774,0.13031161473087818,28.4397065807435,28.4397065807435,0.2715174671175912
"replay, clone",0,36.078571492441505,0.3741043159473421,0.5322040312592278,2.131259598338551,0.2736847909936279
"replay, clone",1,27.191822359381977,0.5484085985669055,0.40274847834356925,2.0281016058614076,0.2736925558371559
"replay, clone",2,30.986753783465435,0.5327445425762373,0.24973690931341544,2.128940397469289,0.2737154885043541
"replay, clone",3,27.092290386059318,0.5655724045992334,0.19095256792112186,2.0262522423217586,0.27366704758011784
"replay, clone",4,24.83966107677349,0.5424095984002666,0.23403105288862897,1.9248198071294806,0.27368766644561277
"replay, clone, noise assumed x4",0,4.977135696293209,0.8950174970838194,0.4142387267865148,2.381071400000901,0.6662037845719603
"replay, clone, noise assumed x4",1,4.435673577277113,0.9150141643059491,0.7398779535557101,2.2975612749742296,0.6644617442506766
"replay, clone, noise assumed x4",2,4.075097865891809,0.9166805532411265,0.6923764484038278,2.2960069179343803,0.6633675135651395
"replay, clone, noise assumed x4",3,3.8730539801078536,0.9346775537410431,0.4781507195930287,2.214863977962591,0.666187616840367
"replay, clone, noise assumed x4",4,3.5801741639212428,0.9761706382269622,0.6357296692568984,2.1714827797725076,0.6636345605641294
Loading
Loading