Repository navigation
docs: close B1 as characterised, not solved (ADR-0036) - #187
Merged
Merged
Conversation
…ested navkit gate usefulness-s2 runs the plain clone and two Gauss-Markov subjects in S2 on development seeds 25 to 49 and reports the six conditions of the ADR-0030 gate with the per-seed comparison against the no-vision control. Development seeds only: the command takes no seed and refuses seed 100 and up. Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
ADR-0036, the generated usefulness section on the calibration page, a sixth row in the README evidence table, the CONSTRAINTS B1 current status, the status page and the roadmap, and the front-page claims test tied to the committed CSVs. Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Keep both test files in the ci.yml pytest list. Co-Authored-By: Claude Sonnet 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01L6ro5x4WRA4P7sQHLyPPGa
Keep the superset test list in ci.yml. Co-Authored-By: Claude Sonnet 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01L6ro5x4WRA4P7sQHLyPPGa
Comment on lines
+284
to
+285
| "| Subject | Runs | Control window ATE, median (m) | Candidate window ATE, median (m) | " | ||
| "Runs with the lower ATE | ATE ratio, median | ATE ratio, 90th percentile | ATE ratio, maximum |", |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #44. Part of #113. I did not close them: the maintainer closes them, as ADR-0036 says.
Who wrote this. The maintainer asked the reviewer session, which normally only reviews and writes briefs, to implement this package. So this PR was not reviewed by an independent session first. I ran the two-axis code review (Standards and Spec, as separate sub-agents) on the staged diff, and the findings and what I changed are below. A reviewer other than me should still read it.
Stacked. It builds on the branch of PR #179 (
feat/gm-tuning), because the CSVs it reads (gm_tuning.csv,gm_tuning_robustness.csv) exist only there. The base of this PR is that branch. When #179 merges, GitHub retargets this PR to main.What it does
B1 is closed as characterised, not solved, by ADR-0036. The shipped configuration does not change and
vision_enabledstays False.navkit gate usefulness-s2 --csv docs/data/gm_s2_usefulness.csv --page docs/calibration.mdruns three subjects in scenario S2 on development seeds 25 to 49 (the command takes no seed and refuses seed 100 and up): the plain clone, and the Gauss-Markov model at tau 2 s with rho 0.9 and with rho 0.99 (near the generator's own error process). It reports the six conditions of the ADR-0030 gate and the per-seed comparison with the no-vision control. About 95 s; a second run is byte-identical.docs/status.md, one dated line and the B1 bullet in ROADMAP, a pointer to ADR-0036 at the top of the three calibration sections, and the new generated section on the calibration page.tests/test_front_page_claims.pynow has six rows and ties row 6 to three committed CSVs.The new CSV
The numbers agree with the scratch measurement I made before writing the brief. They are a DIAGNOSTIC on development seeds, not a gate decision and not a held-out evaluation. They show that this near-exact model holds the three NEES conditions in S2 and does not meet the usefulness conditions, and that making it more exact does not change which conditions hold. They do not show that no estimator can meet those conditions, and the page says so.
Tests and evidence
tests/test_gm_usefulness.py(11 tests) andtests/test_gm_usefulness_page.py(9) failed with an import error on the base. The two page-dependent tests oftest_front_page_claims.py(six rows, ADR-0036 on the pages) failed against the old README, CONSTRAINTS and status page and pass now.metrics.py --check,check_doc_tables.py(against a benchmark generated to a scratch path),check_links.py(106 files) andmkdocs build --strictpass. The golden test is unchanged.tests/test_ci_docs_tests.pyrequired the new page test to be added to the docs-only list inci.yml, and it is.The code review, and what I did about it
SEEDSis corrected,_gmis renamed and its time constant named,splice_sectionis reused fromgm_tuning, the sentence about a more exact model no longer depends on subject order, and the seed-guard test usesSEED_LIMIT.csv_textandparse_csvrepeat the field-type dispatch of the helpers ingm_tuning.py. Those are private and tied to their own field lists, so reusing them needs a refactor outside this package. Noted for later.Things to know
CONSTRAINTS.mdstill ends with "Any change to the candidate is tested on a new held-out set, starting at seed 125". That is generated from its CSV and I left it. It stays true if a new ADR adds a candidate, as ADR-0036 allows.docs/results.mdhas no sentence that calls B1 open or a candidate pending, so it is not in the diff.🤖 Generated with Claude Code