Skip to content

docs: close B1 as characterised, not solved (ADR-0036) - #187

Merged
Telschow merged 4 commits into
feat/gm-tuningfrom
docs/b1-close-out
Oct 24, 2026
Merged

Telschow merged 4 commits into
feat/gm-tuningfrom
docs/b1-close-out

Conversation

@Telschow

Copy link
Copy Markdown
Owner

Part of #44. Part of #113. I did not close them: the maintainer closes them, as ADR-0036 says.

Who wrote this. The maintainer asked the reviewer session, which normally only reviews and writes briefs, to implement this package. So this PR was not reviewed by an independent session first. I ran the two-axis code review (Standards and Spec, as separate sub-agents) on the staged diff, and the findings and what I changed are below. A reviewer other than me should still read it.

Stacked. It builds on the branch of PR #179 (feat/gm-tuning), because the CSVs it reads (gm_tuning.csv, gm_tuning_robustness.csv) exist only there. The base of this PR is that branch. When #179 merges, GitHub retargets this PR to main.

What it does

B1 is closed as characterised, not solved, by ADR-0036. The shipped configuration does not change and vision_enabled stays False.

  1. A generated oracle result. navkit gate usefulness-s2 --csv docs/data/gm_s2_usefulness.csv --page docs/calibration.md runs three subjects in scenario S2 on development seeds 25 to 49 (the command takes no seed and refuses seed 100 and up): the plain clone, and the Gauss-Markov model at tau 2 s with rho 0.9 and with rho 0.99 (near the generator's own error process). It reports the six conditions of the ADR-0030 gate and the per-seed comparison with the no-vision control. About 95 s; a second run is byte-identical.
  2. ADR-0036. Context, Options (a) to (c), the Decision, Consequences. Dated notes (2026-10-24) on ADR-0030, ADR-0031 and ADR-0034 only. Index row and nav entry.
  3. Pages made consistent. README (sixth row of the evidence table and one lead-in sentence), CONSTRAINTS B1 current-status paragraph (the generated held-out block, its markers and the history paragraphs are untouched), docs/status.md, one dated line and the B1 bullet in ROADMAP, a pointer to ADR-0036 at the top of the three calibration sections, and the new generated section on the calibration page.
  4. tests/test_front_page_claims.py now has six rows and ties row 6 to three committed CSVs.

The new CSV

Subject Control window ATE, median (m) Candidate window ATE, median (m) Lower ATE in Ratio median Ratio 90th pct Ratio max NEES conditions Usefulness conditions
plain clone (no model) 7.10 7.29 15 of 25 0.71 3.00 5.89 fail: upper, coverage fail: better, worst_ratio
Gauss-Markov model, tau 2 s, rho 0.9 7.10 4.63 19 of 25 0.54 1.98 2.79 hold fail: better, worst_ratio
Gauss-Markov model, tau 2 s, rho 0.99 7.10 4.72 19 of 25 0.54 1.97 2.72 hold fail: better, worst_ratio

The numbers agree with the scratch measurement I made before writing the brief. They are a DIAGNOSTIC on development seeds, not a gate decision and not a held-out evaluation. They show that this near-exact model holds the three NEES conditions in S2 and does not meet the usefulness conditions, and that making it more exact does not change which conditions hold. They do not show that no estimator can meet those conditions, and the page says so.

Tests and evidence

  • Written first. tests/test_gm_usefulness.py (11 tests) and tests/test_gm_usefulness_page.py (9) failed with an import error on the base. The two page-dependent tests of test_front_page_claims.py (six rows, ADR-0036 on the pages) failed against the old README, CONSTRAINTS and status page and pass now.
  • Mutation checks (each reverted and diffed back to the staged file). Dropping the seed guard fails the refusal test. Making the near-exact subject the plain clone fails 3 tests.
  • Gates. ruff, ruff format and mypy are clean. metrics.py --check, check_doc_tables.py (against a benchmark generated to a scratch path), check_links.py (106 files) and mkdocs build --strict pass. The golden test is unchanged. tests/test_ci_docs_tests.py required the new page test to be added to the docs-only list in ci.yml, and it is.
  • Full suite: 2406 passed, 32 skipped (the data-dependent tests; no datasets in this worktree), 2 xfailed, 0 failed. My first full run stopped at one failure that my own change caused: a guard that bans the old verdict word in code found it quoted in my forbidden-word test lists. I changed those checks to a word-boundary regex.

The code review, and what I did about it

  • Spec axis: no missing requirement and no wrong implementation. Two small unrequested edits to the ROADMAP (a status-marker change and a second note). I reverted both and kept the one dated line and the B1 bullet.
  • Standards axis, fixed. The generated sentences contained hand-typed literals (the seed count, the rho values, the 2 s correlation time). They now come from the rows and the subject labels. The ADR's "25 seeds" is gone. A wrong comment on SEEDS is corrected, _gm is renamed and its time constant named, splice_section is reused from gm_tuning, the sentence about a more exact model no longer depends on subject order, and the seed-guard test uses SEED_LIMIT.
  • Standards axis, not changed. csv_text and parse_csv repeat the field-type dispatch of the helpers in gm_tuning.py. Those are private and tied to their own field lists, so reusing them needs a refactor outside this package. Noted for later.
  • For the maintainer to confirm. CLAUDE.md says the maintainer writes an ADR's Decision section. The Decision text in ADR-0036 is the text I drafted in the brief and you accepted by asking for this package, the same pattern as ADR-0027 to ADR-0035. Please confirm it, or rewrite it.

Things to know

  • The generated held-out status block in CONSTRAINTS.md still ends with "Any change to the candidate is tested on a new held-out set, starting at seed 125". That is generated from its CSV and I left it. It stays true if a new ADR adds a candidate, as ADR-0036 allows.
  • docs/results.md has no sentence that calls B1 open or a candidate pending, so it is not in the diff.
  • Deep-Urban-1 is not mentioned: that is PR feat(urbannav): Deep-Urban-1, the aided run on real urban GNSS (#131) #180. Its page should link ADR-0036 when it merges.

🤖 Generated with Claude Code

Telschow and others added 4 commits October 24, 2026 14:39
…ested

navkit gate usefulness-s2 runs the plain clone and two Gauss-Markov subjects in S2 on
development seeds 25 to 49 and reports the six conditions of the ADR-0030 gate with the
per-seed comparison against the no-vision control. Development seeds only: the command
takes no seed and refuses seed 100 and up.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
ADR-0036, the generated usefulness section on the calibration page, a sixth row in the
README evidence table, the CONSTRAINTS B1 current status, the status page and the
roadmap, and the front-page claims test tied to the committed CSVs.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Keep both test files in the ci.yml pytest list.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01L6ro5x4WRA4P7sQHLyPPGa
Keep the superset test list in ci.yml.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01L6ro5x4WRA4P7sQHLyPPGa
Comment on lines +284 to +285
"| Subject | Runs | Control window ATE, median (m) | Candidate window ATE, median (m) | "
"Runs with the lower ATE | ATE ratio, median | ATE ratio, 90th percentile | ATE ratio, maximum |",
@Telschow
Telschow merged commit 877c5da into main Oct 24, 2026
21 checks passed
@Telschow
Telschow deleted the docs/b1-close-out branch October 24, 2026 13:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants