Skip to content

feat(urbannav): Deep-Urban-1, the aided run on real urban GNSS (#131) - #180

Merged
Telschow merged 8 commits into
mainfrom
feat/urbannav-deep-urban-run
Oct 24, 2026
Merged

Telschow merged 8 commits into
mainfrom
feat/urbannav-deep-urban-run

Conversation

@Telschow

Copy link
Copy Markdown
Owner

Closes #131

Step 2b of the Deep-Urban-1 package. The data stayed read-only in the maintainer's checkout (through a symlink and --root). Nothing was written into it (0 files modified, checked by modification time after the full run), and no data is committed (S2). The INSPVAX choice (option A, text reference only) is recorded as context in ADR-0026, and this PR adds a dated context section there. No Decision text, no outage rule change, no Tunnel-1 number change.

What changes

  • Rows. docs/data/urbannav_run.csv (aided run, declared and per_fix, seeds 0 to 4) and docs/data/urbannav_data.csv get Deep-Urban-1 rows through the merge writer. Both files are pure additions. The Tunnel-1 lines are byte-identical:

    file Tunnel-1 lines SHA-256 of those lines, before and after
    urbannav_run.csv 925 de65bb1c4a9cee30aaaafc02f8046ab23b339d0bf7379d3a279433ec815783c3
    urbannav_data.csv 24 b9d3fc433e183e15c30c6a44a5c9a4c28b86cdc973fc0da8b439486a480a0b68

    The committed bytes of both files are an exact prefix of the new ones. A page test pins both hashes.

  • New aggregates (aided run only). Gate alpha and level, mean NIS, expected rejections, Wilson 95% intervals of the rejection rate on the raw and the effective count (ADR-0029), effective sample sizes of the horizontal and attitude NEES, NEES by thirds, fixes over 3 receiver sigmas, the receiver's median sigma against the filter's claimed one. The effective count of the rejection rate has one convention: a constant over-the-level series (no rejection) has no autocorrelation to estimate and counts every fix, which fix_rejection_ess_defined flags. The page states it.

  • New CSV and command. navkit urbannav fixerrors writes docs/data/urbannav_fix_errors.csv for both sequences: mean offset, RMS about it, the correlation of the fix error at lags 1, 10, 30 and 60 s (pairs within 0.5 s of the lag, per axis and pooled), and the effective sample size per axis (navkit.eval.statistics, mean removed). A separate file, so no existing row moves.

  • Page. docs/urbannav.md: the Deep-Urban-1 section is final. ATE stays in the CSV and is not in the page tables (the unused height dominates it). Every statement has a test against the CSVs, and the prose has no typed number. One statement is labelled INTERPRETATION (the filter treats fixes as independent; the fix errors are correlated over tens of seconds; the filter was not run with a correlated-error model, so this proves nothing about the cause; the reference's own error is unknown). The stale wording from the step 2a PR is updated (status, the "has not been fetched or run" bullet, "What follows", ADR-0026 "What this does not settle", WB-7, the urbannav help text).

  • Tests. The four tests that assumed no Deep-Urban-1 rows are strengthened: the prefix tests now add a new sequence name, and a second test rewrites Deep-Urban-1 and checks the Tunnel-1 lines are identical. Data-gated tests for the real Deep-Urban-1 files skip without the data.

Sanity checks (scratch runs, not committed; seed 0 unless noted)

Check Result
a Start state and error at 10 s Start is the reference plus the seed's perturbation: 0.98 m, 5.4 deg. At 10 s: 2.62 m (declared), 2.09 m (per_fix). Gravity check 0.04 deg from vertical.
b Attitude convention Reference velocity rotated by the matrices against the position derivative: median 0.069 m/s (p95 0.18). Body-y against the velocity heading: median 1.37 deg offset; Tunnel-1 gives 1.31 deg by the same method (the reference's own body velocity has x/y = 0.024).
c Gravity Specific force rotated to ENU minus (0, 0, 9.80665) over the first 3 s: (-0.034, -0.025, -0.017). Whole run mean (0.000, 0.002, -0.018). Sign: +z up.
d Clock Best lag 0.00 s (RMS and correlation); the repo search returns 0.01 s (one step). At +/-0.26 s (reference and fixes shifted together): ATE 12.24 to 12.25 / 12.46 m, horizontal NEES 13.84 to 14.07 / 14.79, attitude NEES 14.1 to 19.8 / 22.0. Per_fix and a reference-only shift behave the same. Zero shift is best on every measure.
e NEES by thirds Horizontal 14.1 / 22.4 / 4.9 (expected 2), attitude 20.3 / 12.9 / 9.2 (expected 3). The excess is in every third.
f 60 s dead reckoning bound Accelerometer only with the reference attitude: median 61 m, max 72 m. Aided error at the same instants: median 3.4 m, max 13.2 m. Aided is below the bound in 25 of 25 windows. The bound is loose (no bias removed).
g Fixes after large-sigma stretches None rejected. Gate level 16.27, largest NIS among them 1.9. Same answer for three definitions of "large".

The maintainer's facts held: 1539 fixes, all GGA quality 2, 12 satellites, zero outages, 614782 IMU samples, reference codes 1 to 5, 10.8% of fixes (166) more than 3 sigma off. The filter sees 1537 fixes, because two lie outside the IMU span.

Rows (ranges over the five seeds)

declared per_fix
Fixes offered / used / rejected 1537 / 1537 / 0 1537 / 1537 / 0
Rejections expected at alpha 0.001 1.5 1.5
Wilson 95% interval of the rate, raw and effective [0, 0.00249] [0, 0.00249]
Horizontal NEES (2 dof) / its effective sample size 13.8 / 18.0 to 18.1 15.1 / 13.5 to 13.6
Attitude NEES (3 dof) / its effective sample size 13.5 to 14.1 / 62 to 164 13.1 to 13.3 / 179 to 196
Pooled 2-sigma coverage (Gaussian 0.9545) 0.606 to 0.608 0.571 to 0.573
Horizontal RMSE 4.35 m 4.30 m
Mean NIS (3 dof; gate level 16.27) 0.50 0.52

Fix statistics (same in both): median 2.36 m, p95 7.50 m, p99 12.97 m; mean of (difference / receiver sigma) squared 3.285; 166 fixes over 3 sigma, 48 over 10 m, the gate rejects none.

Fix-error time behaviour (new CSV). Deep-Urban-1: pooled correlation 0.948 at 10 s, 0.760 at 30 s, 0.472 at 60 s; effective sample size 16.5 (east) and 17.9 (north) for 1539 fixes. These match the maintainer's hand measurements (0.95, 0.47, about 17). Tunnel-1: 0.233 at 10 s pooled (per axis 0.077 and 0.356), about zero at 30 and 60 s; effective sample size 66.8 and 18.3. The ESS and the long lags agree with the hand values. The 10 s value for Tunnel-1 depends on the estimator and axis (0.04 to 0.36 over the variants tried; the hand value 0.15 is inside that range), because there are only 250 fixes and a few tens-of-metres ones at the tunnel edges. The page claims only that Tunnel-1 is less correlated than Deep-Urban-1 at 10 s, which a test checks.

Runtime

One run: 38.6 s of filter time (62.8 us per IMU step), about 72 s wall with scoring (two at once). The ten-run command: 13 min 38 s. PR #169 had extrapolated 350 to 440 us per step.

Checks

ruff, ruff format, mypy: clean. check_doc_tables.py against a freshly generated benchmark in a scratch path, metrics.py --check after --write (only the test count moved), check_links.py (0 broken), mkdocs build --strict: pass. tests/test_framing.py, tests/test_quoted_numbers.py, test_quoted_metrics.py, test_ci_docs_tests.py after git add: pass. Full suite WITH the data: 2352 passed, 1 skipped (results/benchmark.json not generated), 2 xfailed, 0 failed. Full suite with the data hidden: 2319 passed, 34 skipped (every skip is a data test, the new Deep-Urban-1 ones included), 2 xfailed, 0 failed. Mutation checks, each reverted: a rejected fix in one seed's CSV row fails 3 page tests; a lowered Deep-Urban-1 lag-10 correlation fails the interpretation test and the block test.

Risks and what this does not show

  • One sequence, one receiver, one recording. The seeds perturb only the start state.
  • The reference is a post-processed estimate whose own error is not known, so the correlated difference could partly be in the reference. The interpretation is labelled and is not a result: no correlated-error model was run.
  • The NEES means rest on a few effective samples (13 to 18 for the horizontal series).
  • The gate rejecting nothing is consistent with its alpha (the alpha lies inside the Wilson interval), but the rows do not show why it rejects none of the far fixes. The page does not give a mechanism.
  • The clock-shift and dead-reckoning checks are scratch runs; no code for them is committed.
  • The coverage ratchet (coverage_report.py) was not run locally.

🤖 Generated with Claude Code

Telschow and others added 2 commits October 24, 2026 11:39
…e sizes (#131)

The aided run (a sequence without an outage) now also reports the gate's alpha and
the rejections expected at it, the Wilson 95% interval of the rejection rate on the
raw count and on the effective count (ADR-0029), and the effective sample size of the
horizontal and attitude NEES series with the upper-bound flag. A constant
over-the-level series has no estimate and counts every fix, which the new
fix_rejection_ess_defined flag says. Aided rows only: the Tunnel-1 quantities and
rows are untouched.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Adds the Deep-Urban-1 rows (aided run, declared and per-fix variants, five seeds)
to the run and data CSVs through the merge writer, and the Tunnel-1 lines of both
files are byte-identical to before. The aided run now also reports counts of far
fixes, the mean NIS and the gate level, the claimed sigma against the receiver's,
and the NEES by third of the run.

New: `navkit urbannav fixerrors` and docs/data/urbannav_fix_errors.csv, the time
correlation of the receiver's own fix errors against the reference for both
sequences. The page gets the Deep-Urban-1 findings, each tied to the CSVs by a
test, an INTERPRETATION that says what it does not prove, and no typed number.
ADR-0026 gets a dated context section (no Decision text). Data-gated tests for the
real files skip without the data. The byte-prefix tests now use a new sequence
name, so they still prove that the committed rows are untouched.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Telschow and others added 5 commits October 24, 2026 15:50
CI measured 93.76% against the 94.31% recorded, a gap above the 0.5 points allowed.
Regenerated with scripts/metrics.py --write --coverage after merging origin/main.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01L6ro5x4WRA4P7sQHLyPPGa
…etic sequence

CI has no real dataset, so urbannav_check and urbannav_fix_errors were at 73% and 58% line
coverage, and total coverage fell 0.55 points below metrics.json. These tests run measure(),
the CSV and page-block paths and main() on the generated fixture. Both modules are now above 99%.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01QnVDRzD2rMdsXzjGpA5up9
Generated by scripts/metrics.py --write --coverage after the new study tests.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01QnVDRzD2rMdsXzjGpA5up9
@Telschow
Telschow merged commit a0a2d03 into main Oct 24, 2026
21 checks passed
@Telschow
Telschow deleted the feat/urbannav-deep-urban-run branch October 25, 2026 06:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

UrbanNav follow-up: Deep-Urban-1 rows and the INSPVAX decision (left over from #95)

1 participant