[Telschow]

Part of #112.

Since ADR-0029 a single run's verdict uses the effective sample size, so the mismatch sweep (5 seeds per cell, single-run verdicts) cannot resolve a mild mismatch from one run. This adds the ensemble version, the way docs/calibration.md does for the benchmark cases.

What changed

• navkit sweep mismatch --ensemble-csv PATH [--ensemble-page PATH] [--ensemble-compare PATH] [--ensemble-seeds N]. It extends the existing command. The existing outputs are unchanged: --compare docs/data/mismatch_sweep.csv still matches, and a test checks the single-run CSV and JSON are identical with and without the new flags.
• src/navkit/mismatch_ensemble.py: each (case, factor) is run for seeds 25 to 49 through the sweep's own run_case path, the per-epoch NEES is kept, and ensemble_nees scores it (band for 25 runs and 3 dof). Only the statistics docs/calibration.md already defines: grand mean and the shares inside, below and above the band, for the full run and, for outage_control, the outage scope.
• docs/data/mismatch_ensemble.csv (deterministic; a fresh run is byte-identical).
• docs/mismatch.md: a new section "The ensemble over seeds". The table and the sentences are generated from the two CSVs and a test rebuilds them.
• CHANGELOG.md, the documentation-test list in ci.yml, and the generated test-count markers (metrics.py --write; PR chore(metrics): quote the test count rounded down to a hundred, so PRs stop conflicting #174 is not merged).

Result

Band for the ensemble mean over N = 25 runs with 3 degrees of freedom: 2.118 to 4.034. Shares are of the scored epochs.

Case	Assumed noise x	Scope	Scored epochs	Grand mean NEES	Inside the band	Below	Above
gnss_only	0.25	full	6001	44.56	0.0%	0.0%	100.0%
gnss_only	0.5	full	6001	20.57	0.0%	0.0%	100.0%
gnss_only	0.7	full	6001	6.11	4.4%	0.0%	95.6%
gnss_only	0.8	full	6001	4.592	21.7%	0.0%	78.3%
gnss_only	0.9	full	6001	3.523	87.0%	0.1%	12.8%
gnss_only	1	full	6001	2.888	94.9%	4.7%	0.4%
gnss_only	1.25	full	6001	1.961	26.5%	73.5%	0.0%
gnss_only	2	full	6001	0.995	2.6%	97.4%	0.0%
gnss_only	4	full	6001	0.6246	4.0%	96.0%	0.0%
outage_control	0.25	full	6001	36.79	0.0%	0.0%	100.0%
outage_control	0.25	outage	3000	30.62	0.0%	0.0%	100.0%
outage_control	0.5	full	6001	13.1	0.0%	0.0%	100.0%
outage_control	0.5	outage	3000	10.48	0.0%	0.0%	100.0%
outage_control	0.7	full	6001	4.514	51.5%	0.0%	48.5%
outage_control	0.7	outage	3000	3.596	99.1%	0.0%	0.9%
outage_control	0.8	full	6001	3.218	71.8%	0.0%	28.2%
outage_control	0.8	outage	3000	2.396	100.0%	0.0%	0.0%
outage_control	0.9	full	6001	2.557	50.3%	48.3%	1.4%
outage_control	0.9	outage	3000	1.933	3.7%	96.3%	0.0%
outage_control	1	full	6001	2.147	45.0%	54.7%	0.4%
outage_control	1	outage	3000	1.649	0.0%	100.0%	0.0%
outage_control	1.25	full	6001	1.573	5.3%	94.7%	0.0%
outage_control	1.25	outage	3000	1.322	0.0%	100.0%	0.0%
outage_control	2	full	6001	1.114	2.5%	97.5%	0.0%
outage_control	2	outage	3000	1.234	0.0%	100.0%	0.0%
outage_control	4	full	6001	1.103	4.0%	96.0%	0.0%
outage_control	4	outage	3000	1.469	0.0%	100.0%	0.0%

My rule for resolved, which is a choice and not a test: the mild factors are 0.8, 0.9 and 1.25. The ensemble resolves one when its mean is outside the band, on the side the mismatch predicts, for more than 50.0% of the scored epochs and for a larger share than at the true noise. One run resolves it when its bulk test reads the predicted way in at least 4 of 5 seeds.

• gnss_only: with the assumed noise below the true noise, the ensemble mean is above the band for 100.0%, 100.0%, 95.6%, 78.3% and 12.8% of the scored epochs at factors 0.25, 0.5, 0.7, 0.8 and 0.9, a share that rises or stays level as the mismatch grows.
• gnss_only: with the assumed noise above the true noise, the ensemble mean is below the band for 73.5%, 97.4% and 96.0% of the scored epochs at factors 1.25, 2 and 4, a share that does not keep rising as the mismatch grows.
• gnss_only: at the true noise (factor 1) the ensemble mean is inside the band for 94.9% of the scored epochs, below it for 4.7% and above it for 0.4%.
• gnss_only: of the mild factors 0.8, 0.9 and 1.25, the ensemble resolves 0.8 and 1.25 and not 0.9.
• outage_control: with the assumed noise below the true noise, the ensemble mean is above the band for 100.0%, 100.0%, 48.5%, 28.2% and 1.4% of the scored epochs at factors 0.25, 0.5, 0.7, 0.8 and 0.9, a share that rises or stays level as the mismatch grows.
• outage_control: with the assumed noise above the true noise, the ensemble mean is below the band for 94.7%, 97.5% and 96.0% of the scored epochs at factors 1.25, 2 and 4, a share that does not keep rising as the mismatch grows.
• outage_control: at the true noise (factor 1) the ensemble mean is inside the band for 45.0% of the scored epochs, below it for 54.7% and above it for 0.4%. That is not inside the band for most epochs, so for this case read the factors against each other, not against the band.
• outage_control: of the mild factors 0.8, 0.9 and 1.25, the ensemble resolves 1.25 and not 0.8 and 0.9.
• outage_control (outage scope): with the assumed noise below the true noise, the ensemble mean is above the band for 100.0%, 100.0%, 0.9%, 0.0% and 0.0% of the scored epochs at factors 0.25, 0.5, 0.7, 0.8 and 0.9, a share that rises or stays level as the mismatch grows.
• outage_control (outage scope): with the assumed noise above the true noise, the ensemble mean is below the band for 100.0%, 100.0% and 100.0% of the scored epochs at factors 1.25, 2 and 4, a share that is level.
• outage_control (outage scope): at the true noise (factor 1) the ensemble mean is inside the band for 0.0% of the scored epochs, below it for 100.0% and above it for 0.0%. That is not inside the band for most epochs, so for this case read the factors against each other, not against the band.
• outage_control (outage scope): of the mild factors 0.8, 0.9 and 1.25, the ensemble resolves none of them.
• Against the single-run table, gnss_only, mild factors: both resolve 0.8; only the ensemble resolves 1.25; neither resolves 0.9.
• Against the single-run table, outage_control, mild factors: only the ensemble resolves 1.25; only one run resolves 0.8; neither resolves 0.9.
• The ensemble resolves some mild mismatch that one run does not: gnss_only at 1.25; outage_control at 1.25.
• One run resolves some mild mismatch that the ensemble does not: outage_control at 0.8.
• The ensemble does not resolve every mild mismatch: gnss_only at 0.9; outage_control at 0.8 and 0.9; outage_control (outage scope) at 0.8, 0.9 and 1.25.

Sentences changed

• docs/mismatch.md, statement 6: "The ensemble over seeds is the tool that can resolve a mild mismatch" is replaced. The CSV does not support it as written. The ensemble resolves gnss_only at 0.8 and 1.25 (1.25 is one the single run could not), and outage_control at 1.25 on the full run. It does not resolve 0.9 in either case, outage_control at 0.8, or any mild factor in the outage scope. One run resolves outage_control at 0.8 where the ensemble does not. "Resolved" is my rule, stated on the page.
• No sentence in docs/calibration.md makes that claim, so it is not edited.
• New and tested: the true noise is not a fixed point. outage_control at factor 1 is inside the band for all outage epochs on seeds 0 to 24 (benchmark ensemble) and below it for all of them on seeds 25 to 49. I checked my path reproduces the committed benchmark ensemble exactly on seeds 0 to 24, so this is a seed-set effect. I have not explained it.

Runtime and CI

• navkit sweep mismatch with the ensemble, full grid (2 cases, 9 factors, 25 seeds each, plus the 5-seed single-run sweep): 8 min 22 s on a laptop, about 7.4 min of it the ensemble. The new tests add about 45 s to the test jobs.
• I did not add the CSV to the "ensemble reproducibility" job. It took 25 min on the last green run on main, the longest job (test (py3.13)) took 40 min, the repo states no budget for it, and I cannot time a runner from here. The extra 7 to 15 min would put it level with the longest job. The CSV is checked against the page by test, not against a fresh run. Reproduce with navkit sweep mismatch --ensemble-csv /tmp/e.csv --ensemble-compare docs/data/mismatch_ensemble.csv. Say if you want it in the job.

Mutation checks

• Score the square root of the NEES (sigma instead of sigma squared): 3 tests fail (test_a_known_input_is_scored_against_the_band..., ..._agrees_with_ensemble_nees..., ..._half_open_window_only).
• Take the band edges in sigma (square root of the chi-square quantiles): test_a_known_input_is_scored_against_the_band_for_n_runs_and_three_dof fails.
• Drop the factor from the case: test_the_factor_reaches_the_filter_so_two_factors_give_two_results and test_a_cell_is_the_sweeps_own_run_case_path_on_seeds_25_to_27 fail.

Checks

ruff, ruff format, mypy, check_doc_tables.py (benchmark JSON in a scratch path), metrics.py --check, check_links.py, mkdocs build --strict, test_framing.py, test_quoted_numbers.py, the golden test (unchanged), and the full suite once. The full suite had tests/test_doc_tables.py::test_the_real_documents_match_the_committed_benchmark failing; it fails the same on main because the local results/benchmark.json is stale, and check_doc_tables.py passes against a fresh one. Three other failures were mine (an import cycle, a quoted word, a page expectation); each is fixed and its test rerun.

🤖 Generated with Claude Code