Skip to content

Stage categorical NativeLink authority and canonical runtime foundations - #87

Draft
b7r6 wants to merge 168 commits into
mainfrom
categorical-elimination
Draft

b7r6 wants to merge 168 commits into
mainfrom
categorical-elimination

Conversation

@b7r6

@b7r6 b7r6 commented Oct 23, 2026 •

Copy link
Copy Markdown
Owner

What and why

This draft starts the categorical elimination plan: one committed scheduler transition model, fixed storage and worker roles, and explicit ownership and recovery boundaries. It collects the experimental implementation, related protocol repairs, and the evidence needed to review each stage. Production factories still construct the legacy runtime.

The series includes:

  • A deterministic scheduler, independent reference model, replay and mutation harnesses, and OpenRaft 0.9.25 authority with durable, replicated, and volatile recovery experiments.
  • Worker attempt identities, reconciliation, subprocess fencing, resource accounting, progress and deadline handling, and scheduler-owned accepted-result publication experiments.
  • Fixed listener experiments; bounded CAS, AC and ByteStream handling; digest-context, upload-identity and producer-completion repairs.
  • Strict canonical configuration parsing and migration inventory, descriptor-based namespace and storage capabilities, private journal ownership, a closed recovered owner with supervised lifecycle and clock, and descriptor-based worker staging capabilities.

Start with the source plan, contract, and morning ledger. The ledger links individual changes, failures, qualification receipts and remaining work. Most changed files are archived evidence; the implementation and design records are organized by crate and stage.

How was this verified?

  • Final GitHub CI is green at 39936f3748: all 168 normal Bazel targets and all 168 ASAN targets executed and passed, all 10 remote-execution examples passed, and fuzzing completed 2,890,093 executions. Web, docs, pre-commit, local-execution, CodeQL and PR validation checks also passed; the two existing schedule-only Nix jobs were skipped. See the normal/RBE run, ASAN run, and fuzz run.
  • The current CI repair pass fixes a fair-mutex deadlock in scheduler listing prefetch while preserving incremental dispatch, ordering, bounded reads and cancellation. It also corrects sanitizer memory fixtures to measure actual process-group PSS, enables required Ubuntu worker namespaces, isolates a filesystem fixture's fork/lock race, and updates nightly lint assertions.
  • On the combined Rust source at 50097d6375, all 168 Bazel test targets passed uncached under ASAN, with nightly Clippy and rustfmt checks passing. The sanitizer run retained all case selections and target deadlines. The sanitizer workflow separates compilation from tests and bounds nested concurrency for hosted runners.
  • The final test-fixture improvement bounds allocation batches by the measured PSS deficit and the existing memory window; all four affected worker/RPC targets passed uncached again in both normal and ASAN builds, with the enforcement assertions and action/RPC deadlines retained. The allocation setup allowance scales at five seconds per started GiB (15 seconds for the three-GiB retry-budget fixture); individual observer I/O operations still have five-second limits. Controlled CPU-load runs retained the setup-timeout failures; after the repair, all 22 RPC cases passed with four CPU competitors, and the retry-budget case passed for both hashes with eight competitors across four CPUs, preserving all six actual memory kills.
  • The final local normal Bazel check passed all 168 targets using valid cached results, after the four affected targets passed uncached. An earlier local run timed out in the OpenRaft storage suite; the unchanged 24-case target passed on rerun and in the subsequent GitHub run. Original failures and reruns are retained locally; this is not described as an entirely uncached local normal pass.
  • At 50097d6375, a fresh Rust remote build against the corrected server completed 886 remote actions with an empty cache. The build and test passed, both scheduler status endpoints returned HTTP 200, and the server logged no errors.
  • Earlier stage-specific evidence remains linked from the ledger. The canonical-owner checkpoint 78ab50c3b6 ran six uncached targets with 378 passes and one existing ignore, plus a server build; its packet binds source inputs and actual executable digests. Fault campaigns retain original failures and distinguish compiler/prerequisite failures from detected semantic faults.

The closed recovered owner and private worker materialization are included with their separate fault campaigns and the combined root check. The next-stage handoff identifies the remaining layout synchronization, cold initialization/sealing, fixed feature admission and canonical coordinator dependencies. It is a source-only proposal.

Risk

This is a large experimental series touching authority state, persistence, worker lifecycle and protocol behavior. The categorical milestones remain open: production construction, cold bootstrap and sealing, combined workload admission, deployment trust and placement, migration, and final removal of legacy implementations still need work. Redis remains in the repository. Experimental formats and synthetic workload envelopes do not establish a production compatibility or capacity promise. Keep this PR in draft while those boundaries are reviewed and qualified.

AI assistance

OpenAI Codex (Astra), with collaborating agents, implemented the series, ran checks, and performed independent agent reviews. b7r6 directed the work and explicitly requested this draft PR. Human review is pending; no claim is made that every line has already received human review.

b7r6 added 30 commits October 22, 2026 22:54
Add a deterministic authority with independently implemented reference tests,
adversarial replay and mutation checks, an OpenRaft 0.9.25/redb journal, and
real three-member recovery fixtures. Preserve complete request/result facts,
resource charges, bounded starvation holds, and fenced memory retry attempts.

Add an identity-bearing worker recipient and additive protocol, namespace
quiescence receipts, and a durable-authority/subprocess test that permits AC
publication only after acceptance. Preserve uncertainty across cancellation,
reconnect, and restart. Production factories remain on the legacy runtime.

Record retention experiments, synthetic measurements, exact evidence, and
open qualification gates. Aggregate completion-byte credits and meaningful
transfer progress remain subsequent work. Isolate an existing filesystem test
from shared /tmp paths and preserve immutable evidence through repository hooks.

Validation: Cargo workspace check and tests (1546 passed, 5 ignored); all 135
Bazel test targets plus focused followup; 20/20 semantic mutants caught after
repairing one insufficient fixture; repository hooks and docs gates passed.
b7r6 added 30 commits October 23, 2026 22:04
Add descriptor-bound worker input and action capabilities with retained namespace custody, exact bounded copying, read-only completed inputs, isolated metadata and same-family checks.

Keep the module private while qualifying 33 frozen worker cases in both normal and redb suites. Preserve actual binaries, build events and exact source guards for 16 runtime controls and 17 invocations, with refusal-preserving error-contract outcomes distinguished.

Restored ownership68/journal80 suites and normal/redb libraries pass; independent review and complete archived evidence accompany the change. Production factory and execution integration remain separate.
Size allocation batches from measured process-group PSS and the existing workload window, reducing repeated smaps scans. Keep the five-second calibration deadline, cumulative allocation cap, and actual enforcement assertions.
Controlled CPU contention reproduces the three-GiB fixture calibration failure before memory enforcement can report its breach. Allow five seconds of setup per started GiB while keeping observer I/O, action and RPC deadlines, measured memory windows, and all enforcement and retry assertions unchanged.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant