Skip to content

averin export perf: linear-time verifier D6/M4/anchor checks and AVERIN_EXPORT_TIMEOUT 504 (feir-ops#26) - #58

Merged
zachyking merged 16 commits into
mainfrom
wave1/averin-export-perf
Oct 28, 2026
Merged

zachyking merged 16 commits into
mainfrom
wave1/averin-export-perf

Conversation

@zachyking

@zachyking zachyking commented Oct 28, 2026 •

Copy link
Copy Markdown
Member

Post-release 1.

Problem

feir-ops#26: whole-history GET /v2/export and averin-verify bundle were superlinear (28.5 MiB 86 s, 41.7 MiB 177 s, 110.6 MiB 1,208 s), so the export reached the fixed 10 minute deadline at about 110 to 130 MiB and the connection was cut. The export runs the same verifier before its first byte.

Verifier (core)

  • LogCursor gives the same (count, gapless, root) as committed_broker_log + grant_head_root for a sequence of frontiers. State is reused only after the walk from the new frontier has reached every known member of the previous frontier; otherwise it resets and recomputes in full. The old per-frontier functions stay as the fallback and the test oracle.
  • Reach DP for the headless checks, one anchor DP (earliest canonical anchor time per record) instead of one committed set per anchored checkpoint.
  • Memory (review fix): the federation per-broker index and reach vectors are built one broker at a time and dropped at the end of its iteration, and the headless M4 check finds every committing broker with one walk per distinct headless frontier. Peak extra memory is O(N + entries) however many brokers a bundle has (the first version held all of them at once). DagIndex refuses a topo order that lists a child before its parent.
  • Report JSON, issues and their order are byte-identical.

Export deadline (server)

  • AVERIN_EXPORT_TIMEOUT: Go duration, default 10m, range 1m to 60m; unparsable or out of range stops startup. Logged at startup.
  • A build (load, self-verify, disclosures, marshal) that has not finished by the deadline is answered 504 with a generic body (duration and the month hint) before any body byte; counted in averin_bundle_deadline_exceeded_total{route}. /v2/verify waiters get the same 504; the shared flight keeps running for other waiters. The deadline context bounds only the bundle-view build (and on Postgres the 45 s read session cuts first); the wait for a bundle slot and the Rust verify are not bounded by it.
  • The export build runs in a worker. After a 504 the worker owns the concurrency slot until the build ends, so the cap of 4 holds. A Rust verification already running cannot be interrupted; its result is dropped.
  • The response write deadline is the build deadline plus 30 s, and is reset to a full window once the body is ready. A write past its own deadline after the first byte is still cut.
  • One log line per whole-history export: build ms, verify ms, bytes (no project content).
  • Not measured: /v2/export time to first byte at 10k/40k/80k records, a same-database export bytes golden between origin/main and this build, and the anchored-history (TSA pinned) performance measurement from design section 6. The deadline and 504 are covered by unit tests with a slowed store. feir-ops#26 should stay open for those measurements, and for the follow-ups in feir-os and govder (pass the setting, map the 504).

Tests and mutants

  • cargo test -p averin-decision-core --lib committed: seeded random differential tests against the old code, plus (new) counters proving the cursor ran for every head and no frontier took the per-frontier code on an honest history, headless checks answered through the index, 300 brokers equal to the old code, and the topo-order refusal.
  • tests/adversarial.rs: d6_cursor_*, and a new equal-boundary test (key compromised exactly at the anchor time stays trusted, anchored_before answered by the anchor index). Non-canonical stamps that sort before canonical ones were added to the anchor-time test.
  • Server: export_deadline_internal_test.go (parse table, 504 before any byte, slot held after 504 and released at the end, verify 504).
  • Eleven mutants registered in formal/kit-claims.json (three original, eight new: log index, cursor, reach, federation partition, anchor index each forced unused; canonical-guard removal; exclusive boundary; parent-order check removal). All killed with the claim's detector, and each patch changes a symbol the claim covers.

Measurements (same host, dev profile, user CPU of averin-verify, prefixes of the proof run 6 export)

Reviewer rerun, unloaded: 4.57 s, 9.97 s, 24.23 s user at 10k, 20k, 42.7k records. t(20k)/t(10k) = 2.18 (target <= 2.6). t(42.7k)/t(10k) = 5.30 on the unloaded rerun (5.74 on the loaded run); the pre-registered limit of 5.5 was missed on the loaded run and met on the unloaded one. Full 116 MB export: report JSON sha256 identical to origin/main, user CPU 529.2 s to 24.2 s, RSS 684 MB to 698 MB.

Gates (after merging origin/main of 2026-10-28, including #57 and #59)

  • cargo fmt --check, cargo clippy --all-targets -- -D warnings, make test-core pass.
  • make test-server: vet passes; go test ./... with CGO_LDFLAGS='-ldl -lpthread -lm' passes on all packages (the host's old glibc needs the flags).
  • make test-verifier: 53 pass, 0 fail (the pin files it rewrites locally were restored, not committed).
  • python3 scripts/formal/check_claims.py --claims formal/kit-claims.json: 789 checks, 0 failures; python3.12 scripts/check-claims.py: 53 claims; formal/production/check-production.py OK.
  • Relocked covers: nine verify_bundle_with claims and advance (only counters and the lazy build were added; reports unchanged), and two main.go claims (one more env read; posture and content wiring untouched).

Risks

  • verifier/averin_decision_core.wasm.sha256 and verifier/index.html are not repinned (a local build does not reproduce the committed pin; a repin must come from the canonical build).
  • A crafted bundle whose checkpoints do not extend each other still costs the old per-frontier time, bounded by the 256 MiB cap.
  • Peak memory O(N) per partition during the per-broker loop; the per-broker duplicate-seq check still does one O(N) walk per broker (unchanged from before).

🤖 Generated with Claude Code

zachyking and others added 3 commits October 28, 2026 19:41
The offline verifier rebuilt every checkpoint's grant log from the whole history
(committed_set walk, record_trust scan, sort, hash fold), so a history with C
checkpoints and N records cost O(C x N). /v2/export runs the same verifier before
its first byte, so the export inherited the superlinear time.

LogCursor (core/src/verify/committed.rs) gives the same (count, gapless, root) per
frontier while doing work proportional to what is new in each frontier. State from
the previous frontier is reused only after the walk from the new frontier has reached
every known member of the previous one (the committed set is closed under parents,
so the previous closure is then contained); otherwise the cursor resets and
recomputes in full. New entries are merged at their sorted (broker_seq, record
position) position and the fold resumes from the stored accumulator there, never from
a bundle value. Also: a reach DP for the "headless checkpoint commits grants" checks
(D6 and M4), a cursor per broker for the federation heads, and one backward pass for
the anchor times plus one committed_set over the union of anchored frontiers
(replacing one committed set per anchored checkpoint held in memory).

No check is skipped; the report is byte-identical (golden comparison on prefixes of
the 8 h soak export). The old per-frontier functions stay as the fallback and as the
test oracle.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@zachyking

Copy link
Copy Markdown
Member Author

Gate output (host: WSL, old glibc; cargo and go under nice -n 10 taskset -c 0-11):

  • cargo test --workspace (make test-core): exit=0, every test result: line ok ... 0 failed (adversarial: 311 passed; 0 failed; 2 ignored; lib: 66 passed; 0 failed).
  • make test-verifier: 53 pass, 0 fail, Ran 53 tests across 12 files.
  • make test-server: rebuilt the staticlib (Finished dev profile), then go vet and go test -a failed at LINK time on this host for the test binaries (undefined reference to dlsym, old glibc, not related to this change). Rerun by hand with CGO_LDFLAGS="-ldl -lpthread -lm" go vet ./... && go test ./... in server/: every package ok, no failure line.
  • python3 formal/production/check-production.py: production refinement: checks OK (stale, call-path, cfg, glue).
  • python3 scripts/check-claims.py (python 3.12): validated references and CI IDs for 52 claims (textual check only).
  • scripts/formal/check_claims.py: 687 check(s), 687 failure(s) on this branch AND on origin/main (schema errors on every claim, the same before and after), so it gives no signal here.
  • sfw make supply-chain not run: no dependency or crypto change.

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Oct 28, 2026 •

Copy link
Copy Markdown

Deploying averin-verify with  Cloudflare Pages  Cloudflare Pages

Latest commit: 56dd780
Status: ✅  Deploy successful!
Preview URL: https://f7191356.averin-verify.pages.dev
Branch Preview URL: https://wave1-averin-export-perf.averin-verify.pages.dev

View logs

zachyking and others added 12 commits October 28, 2026 20:28
…-path guards (feir-ops#26)

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…k nine verify_bundle_with claims; docs: state measured ratios

The nine relocked claims (temporal-revocation-ordering-tests, pop-intent-and-disclosure-regressions, production-call-path-gate, approver-identity-in-signed-evidence, month-segment-verifier, gateway-refusal-counted-when-proven, gateway-refusal-shape-fail-closed, gateway-refusal-never-widens-surface, oldest-uncovered-received-at) still hold: verify_bundle_with changed only in how anchored_before and the closed set are computed (exactly equal, golden and mutant checked) and in passing topo_order through.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…lbacks, harden DagIndex

The per-broker LogIndex and reach vectors are built inside the per-broker loop and dropped at its end, and the headless M4 check finds every committing broker with one walk per distinct headless frontier, so peak extra memory is O(N) however many brokers a crafted bundle has. DagIndex refuses a topo_order that lists a child before its parent. Per-thread counters (slow_path_probe) record per-frontier fallbacks, cursor advances and anchor fallbacks so a silent revert of the speed-up fails a test. Adds the equal-boundary anchored-before test and non-canonical stamps that sort before canonical ones.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…y_bundle_with and advance covers

The relocked statements still hold: the edits to verify_bundle_with and advance add only slow-path counters and the lazy per-broker build; every report is byte-identical and the detectors pass.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…bundle_with and build_inner in the incremental-log claim

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…te, worker-held slot

A whole-history export or verify that has not finished by the deadline (default 10m, range 1m to 60m, invalid value stops startup) is answered 504 with a generic body before any body byte. The export build runs in a worker; once the handler gives up the worker owns the concurrency slot until the build ends. A verify waiter gets the same 504 while the shared flight continues for others. Counter averin_bundle_deadline_exceeded_total{route} and one timing log line per whole-history export. Docs: CONFIGURATION, API, LIMITATIONS, TESTING.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
The posture-report-live and durable-records-need-durable-content statements still hold: the edit only reads one more environment variable and sets the build deadline; posture and content-store wiring are untouched, and the full server test suite passes.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@zachyking zachyking changed the title core: linear-time D6 grant-log check in the offline verifier (feir-ops#26) averin export perf: linear-time verifier D6/M4/anchor checks and AVERIN_EXPORT_TIMEOUT 504 (feir-ops#26) Oct 28, 2026
@zachyking
zachyking merged commit c5e65a7 into main Oct 28, 2026
36 checks passed
zachyking added a commit that referenced this pull request Oct 30, 2026
verifier: repin the browser WASM to main (post-#57/#58)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant