From 74650c6a603c3e369c57ce4fe88a9ad8eaea7005 Mon Sep 17 00:00:00 2001 From: Bohao Tang Date: Mon, 16 Nov 2026 17:09:31 +0800 Subject: [PATCH 1/2] fix(ci): drop byte-identity gates that fail on dependency bumps --- .agents/knowledge/README.md | 2 +- .../integrator-docs-site-ssot-links.md | 2 +- .../spoke-codegen-pipeline.md | 7 +- ...ci-assembled-committed-native-artifacts.md | 20 ++- .agents/specs/connect/connect-cpp-binding.md | 2 +- .github/workflows/ci.yml | 6 - .github/workflows/cpp-connect.yml | 16 +- .github/workflows/xcframework.yml | 11 +- CONTRIBUTING.md | 4 +- crates/spoke-connect/bindings/cpp/README.md | 11 +- docs/reference/protocol.md | 2 +- docs/zh/reference/protocol.md | 2 +- package.json | 2 +- .../codegen/assert-generated-rust-size.mjs | 48 ------ tooling/codegen/assert-opaque-json-types.mjs | 2 +- tooling/connect/apply-xcframework-artifact.sh | 7 +- .../apply-xcframework-artifact.test.mjs | 9 +- tooling/connect/cpp-build.mjs | 74 +-------- tooling/connect/cpp-build.test.mjs | 78 --------- tooling/connect/verify-binding-regen.sh | 84 ---------- tooling/connect/verify-xcframework-drift.sh | 148 ------------------ tooling/connect/xcframework-manifest.sh | 27 ++++ 22 files changed, 65 insertions(+), 499 deletions(-) delete mode 100755 tooling/codegen/assert-generated-rust-size.mjs delete mode 100755 tooling/connect/verify-binding-regen.sh delete mode 100755 tooling/connect/verify-xcframework-drift.sh create mode 100755 tooling/connect/xcframework-manifest.sh diff --git a/.agents/knowledge/README.md b/.agents/knowledge/README.md index 47a59bb4..2e9fd1b1 100644 --- a/.agents/knowledge/README.md +++ b/.agents/knowledge/README.md @@ -6,7 +6,7 @@ Durable protocol and tooling facts. Normative specs live in [`specs/`](../specs/ | Doc | Category | Source | Summary | |-----|----------|--------|---------| -| [`architecture-patterns/spoke-codegen-pipeline.md`](architecture-patterns/spoke-codegen-pipeline.md) | architecture-patterns | current `schemas/**/*.schema.json`, `tooling/codegen` and generated trees | jstt+typify orchestrator over **114** hand-authored schemas; each generated tree is **114 modules + 5 barrels = 119 files**; `verify-codegen` asserts the schema count, opaque-JSON types, the generated-Rust size bound and zero drift, then runs `cargo check --workspace`; rust-gen fail-fast; **presence-sensitive arrays** — six byte-bound modules module-wide, plus `Content.cognition` in all 15 carrier modules / 17 clone types (unrelated arrays keep `Vec` + `is_empty`); typify nominal-clone canonical-import guide and Builder construction; **regeneration ripple** — zero-drift proof covers generated bytes, not downstream compilability (land ABI + consumer updates together) | +| [`architecture-patterns/spoke-codegen-pipeline.md`](architecture-patterns/spoke-codegen-pipeline.md) | architecture-patterns | current `schemas/**/*.schema.json`, `tooling/codegen` and generated trees | jstt+typify orchestrator over **114** hand-authored schemas; each generated tree is **114 modules + 5 barrels = 119 files**; `verify-codegen` regenerates both trees, asserts the schema count and opaque-JSON types, then runs `cargo check --workspace`; rust-gen fail-fast; **presence-sensitive arrays** — six byte-bound modules module-wide, plus `Content.cognition` in all 15 carrier modules / 17 clone types (unrelated arrays keep `Vec` + `is_empty`); typify nominal-clone canonical-import guide and Builder construction; **regeneration ripple** — zero-drift proof covers generated bytes, not downstream compilability (land ABI + consumer updates together) | | [`architecture-patterns/mind-axis-ownership-boundary.md`](architecture-patterns/mind-axis-ownership-boundary.md) | architecture-patterns | current cognition schema, data contract and Memory contract | One current authority per holder: `KnowledgeEntry.modules.cognition`; open `modules.mental` keeps nine fields with `beliefs` as derivative summary; `WorldFactRecord` stays world-side; `MindState`/`MindDelta` are derivative L5 presentation with RFC 6901 pointers; Memory `Content.cognition` is recorded-at-time and never mutates current authority; complete `(kind, namespace, id)` reference identity; `assessed` vs `no_record` projection; dual-SSOT prevention and rejection table | | [`architecture-patterns/mind-axis-ops-extraction-gate.md`](architecture-patterns/mind-axis-ops-extraction-gate.md) | architecture-patterns | current `spoke-operations` cognition exports and shipped consumers | `spoke-operations` ships only pure validators (cognition record/content/module/observation/filter/selection/projection, MindState RFC 6901 pointer gate) plus deterministic match/select/project serving KE `Scope`/assemble and Memory `cognition_filter`; checker Finding patterns, belief revision, delta apply/replay, ordering, scoring and ranking stay out; further helper families gated on two consumers with dual-language parity and shared vectors | | [`architecture-patterns/connect-error-taxonomy-layering.md`](architecture-patterns/connect-error-taxonomy-layering.md) | architecture-patterns | compound 2026-08-13 (protocol_version_mismatch) | Four-layer error chain (CoreError→ConnectError→RemoteAdapterError→FfiError); dual FFI surface; ordinal stability; frozen contract rule | diff --git a/.agents/knowledge/architecture-patterns/integrator-docs-site-ssot-links.md b/.agents/knowledge/architecture-patterns/integrator-docs-site-ssot-links.md index 74d94f89..bc10920f 100644 --- a/.agents/knowledge/architecture-patterns/integrator-docs-site-ssot-links.md +++ b/.agents/knowledge/architecture-patterns/integrator-docs-site-ssot-links.md @@ -64,7 +64,7 @@ When consolidating fragmented conceptual content (e.g., merging the old 7-page ` 1. **Binding package coordinates** — inline on the native-bindings how-to (NuGet `42ch.Spoke.Connect`, Maven `dev.42ch:spoke-connect`, SPM `SpokeConnect`, Go module path, PyPI `spoke-connect`). Do not leave coordinates behind a spec link. 2. **Domain Profile open-string vocabulary tables** — per-profile vocabulary (beat types, activation fields, pack dialect keys, placement hints) on the consolidated `explanation/domain-profiles.md`. Full mapping matrices remain agent SSOT in specs. -3. **Schema inventory count + codegen verify posture** — explicit subsection on `reference/protocol.md` (114 committed `*.schema.json` files, mirrored by `tooling/codegen/assert-schema-count.mjs`; `verify-codegen` drift gate). +3. **Schema inventory count + codegen verify posture** — explicit subsection on `reference/protocol.md` (114 committed `*.schema.json` files, mirrored by `tooling/codegen/assert-schema-count.mjs`; `verify-codegen` regeneration posture). 4. **Discovery / peering boundary** (explicit peering as the discovery path) — affirmative paragraph on `reference/connect.md`. 5. **Operations purity boundary** (no I/O, storage, LLM, HTTP, ranking, retrieval, silent auto-promote) — affirmative form on `reference/ops.md` and `how-to/orchestrate-ops.md`. diff --git a/.agents/knowledge/architecture-patterns/spoke-codegen-pipeline.md b/.agents/knowledge/architecture-patterns/spoke-codegen-pipeline.md index e3fc6b40..9d18642b 100644 --- a/.agents/knowledge/architecture-patterns/spoke-codegen-pipeline.md +++ b/.agents/knowledge/architecture-patterns/spoke-codegen-pipeline.md @@ -28,14 +28,13 @@ Hand-authored JSON Schema must produce both TypeScript (`@42ch/spoke-schemas`) a 1. **SSOT** — only `schemas/**/*.schema.json` are hand-authored; generated trees are committed. 2. **Orchestrator** — `tooling/codegen` (`pnpm -F @42ch/spoke-codegen codegen` → `node tooling/codegen/src/run.mjs`) walks schemas, localizes `$ref`s for typify, and emits mirrored `generated/{common,connect,data,ops}/` trees in both packages. -3. **Verify** — `pnpm run verify-codegen` runs `codegen` → `tooling/codegen/assert-schema-count.mjs` → `tooling/codegen/assert-opaque-json-types.mjs` → `node --test tooling/codegen/test/deep-ref-hoist.test.mjs` → `tooling/codegen/assert-generated-rust-size.mjs` → `git diff --exit-code` on both generated trees → `cargo check --workspace`. +3. **Verify** — `pnpm run verify-codegen` runs `codegen` → `tooling/codegen/assert-schema-count.mjs` → `tooling/codegen/assert-opaque-json-types.mjs` → `node --test tooling/codegen/test/deep-ref-hoist.test.mjs` → `cargo check --workspace`. 4. **Fail-fast counts** — `assert-schema-count.mjs` holds `EXPECTED_SCHEMA_COUNT = 114`; `tooling/codegen/rust-gen/src/main.rs` holds the same constant and returns non-zero unless exactly 114 Rust schema modules are written. Bump both constants in the same commit as the schema change and the regenerated output. 5. **Closed ops responses** — mutually exclusive success/error shapes use draft-07 `oneOf` (see `assemble-response.schema.json`). 6. **Opaque JSON** — `#/definitions/OpaqueJson` must be an **empty schema object `{}`**. Consuming properties `$ref` that definition (e.g. `ComputableLogChange.previous` / `.next`, `ExtractionRunMetadata.coverage_hint`). Generators emit any-JSON types (`unknown` / `OpaqueJson` in TS; `serde_json::Value` in Rust). 6b. **Annotated `$ref` normalization** — a bare `$ref` to the untyped `{}` target emits `unknown`, but a `$ref` carrying a sibling `description` makes jstt emit an object-index map (`{ [k: string]: unknown | undefined }`), silently narrowing any-JSON to object shape. The orchestrator normalizes annotation-bearing opaque refs to the `allOf` form before generation, and `assert-opaque-json-types.mjs` checks the generated types with a negative control that names the regressed field. Connect `payload` / `auth` / `proof` are the precedent; adapting such consumers is a type-level mechanical change (object-map → `unknown`), not a wire change. 7. **Duplicate generated types (strategy A)** — document typify nominal duplication as known generator behavior; integrators use canonical `common/` imports. No orchestrator dedupe. -8. **Generated Rust size bound** — `assert-generated-rust-size.mjs` caps the three rustfmt-formatted memory response files (`memory_exports_get_response.rs`, `memory_imports_stage_request.rs`, `memory_receipts_get_response.rs`) at 2 MiB to catch runaway shared-definition growth. Raise the bound only with the legitimate schema change and the regenerated file in the same commit. -9. **Release tooling tests** — `pnpm run test:release` runs pure unit tests for lockstep assert/bump scripts (temp fixtures; optional `SPOKE_REPO_ROOT` for harness isolation). Wired into the CI `typescript` job. +8. **Release tooling tests** — `pnpm run test:release` runs pure unit tests for lockstep assert/bump scripts (temp fixtures; optional `SPOKE_REPO_ROOT` for harness isolation). Wired into the CI `typescript` job. ## Schema inventory and generated layout @@ -78,7 +77,7 @@ Raw struct literals (`Scope { scope_id: ..., ... }`) are **field-exhaustive by t Regenerating binding trees changes the **interface downstream host code must implement**, not only the generated files. A new callback method on `PortsHandler` makes every existing implementation of that interface (per-language smoke files, host staging) uncompilable until it implements the method: -- The ripple lands **outside** the generated trees, so a zero-drift, committed regeneration can leave channels unbuildable when their handlers were not updated in the same change. +- The ripple lands **outside** the generated trees, so a committed regeneration can leave channels unbuildable when their handlers were not updated in the same change. - Per-task diff review cannot see this class: each diff looks correct in isolation, and the breakage appears only when the generated tree and its consumers compile together. - Handle it deliberately: land the generated ABI and the downstream handler updates as **one task**, or make the regenerating task's acceptance include "the consuming smoke/host sources still compile". Zero-drift proof covers generated bytes; it says nothing about downstream compilability. diff --git a/.agents/knowledge/tooling-decisions/ci-assembled-committed-native-artifacts.md b/.agents/knowledge/tooling-decisions/ci-assembled-committed-native-artifacts.md index c9a830c7..558f04d7 100644 --- a/.agents/knowledge/tooling-decisions/ci-assembled-committed-native-artifacts.md +++ b/.agents/knowledge/tooling-decisions/ci-assembled-committed-native-artifacts.md @@ -1,7 +1,7 @@ --- module: connect / native artifact provenance date: 2026-08-18 -last_updated: 2026-11-08 +last_updated: 2026-11-16 problem_type: tooling_decision category: tooling-decisions severity: medium @@ -9,12 +9,10 @@ tags: - native-artifacts - xcframework - provenance - - source-freshness - lfs - reproducibility applies_when: - Refreshing committed native carriers - - Verifying source-to-artifact freshness --- # Source-bound committed native artifacts @@ -36,22 +34,22 @@ A simulated compiler can exercise source-identity decisions in an isolated Git r | Carrier | Existing producer and retained evidence | |---|---| | C/C++ | `cpp-connect.yml` produces macOS and Windows artifacts. Each RID keeps its own source revision, target, observed Rust/compiler versions, flags, artifact hashes and both `headerSha256`/`hppSha256`. Apply each RID's files and entry from its own producer artifact. | -| Swift | `xcframework.yml` uses the pinned, locked staged build for the three-slice bundle. The uploaded manifest and producer record bind transferred files to the run's source and observed compiler. The committed-versus-built decision follows upload. | +| Swift | `xcframework.yml` uses the pinned, locked staged build for the three-slice bundle. The uploaded manifest and producer record bind transferred files to the run's source and observed compiler. A committed consumer smoke exercises the committed macOS slice. | | Go Darwin | The existing maintainer recipe builds both committed Darwin targets, copies their dylibs and assigns `@rpath/libspoke_connect.dylib`. Retain the local source, toolchain, SDK, flags, full-byte comparisons and applied hashes. | -Both C header hashes are required per RID; missing or mismatching values fail verification. Header byte representation follows the checkout's LF attributes. Native binaries and each RID's metadata remain one source-bound pair; a bundled sibling entry is not evidence for a different RID's build. +Both C header hashes are recorded per RID; header byte representation follows the checkout's LF attributes. Native binaries and each RID's metadata remain one source-bound pair; a bundled sibling entry is not evidence for a different RID's build. Swift static archives and C dynamic carriers use git-lfs. Applied LFS pointer OIDs must agree with the recorded artifact hashes. Go dylibs remain ordinary Git objects for module-proxy consumption. The C import library and provenance are ordinary Git objects. ### Keep the Swift refresh path usable and fail-closed -The committed Swift consumer smoke exercises the committed macOS slice. The staged producer builds and uploads the refresh bundle, then the two-tree verifier compares it with the committed tree. A stale committed carrier can therefore fail freshness while leaving a valid refresh artifact available. +The committed Swift consumer smoke exercises the committed macOS slice. The staged producer builds and uploads the refresh bundle; the manifest and producer record travel with the artifact, so a maintainer can apply a validated refresh without comparing two build trees. -`apply-xcframework-artifact.sh ` requires the expected completed workflow/job and successful build, manifest, provenance and upload steps. A failed run is usable only when its failed checks are confined to the explicitly named committed freshness and consumer-smoke checks. Other failures, incomplete/cancelled runs or inconsistent outcomes are rejected. +`apply-xcframework-artifact.sh ` requires the expected completed workflow/job and successful build, manifest, provenance and upload steps. A failed run is usable only when its failed checks are confined to the explicitly named consumer-smoke check. Other failures, incomplete/cancelled runs or inconsistent outcomes are rejected. Normal application matches the checkout HEAD to the run's source. Before replacing the bundle, the script validates every transferred manifest entry, binds producer `sourceHead` to the run head, checks the embedded manifest against the transferred manifest and requires an observed compiler value. The sidecar records the actual run conclusion, source and application head. An intentional explicit source override is recorded separately. -A second download of the same build can verify transfer and application identity. Independent reproducibility requires another real source build with equivalent inputs. The producer's failing old-tree comparison, the applied artifact's passing integrity check and later delivery CI retain their distinct results. +A second download of the same build can verify transfer and application identity. Independent reproducibility requires another real source build with equivalent inputs. The transfer manifest proves the shipped bytes; behavior comes from the committed carrier checks. ### Compare independent builds with identical complete inputs @@ -65,16 +63,16 @@ Producer-level normalization of `Info.plist` `AvailableLibraries` ordering stabi ## Why This Matters -Source-bound producer records explain which implementation consumers receive. Required committed-carrier checks preserve drift detection, while upload-before-decision and the guarded application policy make a legitimate refresh possible. Independent cold builds and whole-file mismatch controls establish reproducibility within their declared input boundary. +Source-bound producer records explain which implementation consumers receive. Committed-carrier symbol and smoke checks cover behavior, while the transfer manifest and the guarded application policy make a legitimate refresh possible. Independent cold builds and whole-file mismatch controls establish reproducibility within their declared input boundary. ## When to Apply - A compiled dependency or FFI implementation change affects committed carriers. -- A required freshness check reports drift and a genuine refresh artifact is available. +- A maintainer refreshes a committed carrier from a CI artifact. - A native comparison needs to distinguish changed inputs from same-input nondeterminism. ## References - [Native-binding channels](../../adr/implemented/architecture/2026-08-03-connect-binding-channels.md) - [Maintainer carrier procedures](../../../CONTRIBUTING.md) -- Source: `.github/workflows/cpp-connect.yml`, `.github/workflows/xcframework.yml`, `tooling/connect/cpp-build.mjs`, `apply-xcframework-artifact.sh`, `verify-xcframework-drift.sh`. +- Source: `.github/workflows/cpp-connect.yml`, `.github/workflows/xcframework.yml`, `tooling/connect/cpp-build.mjs`, `apply-xcframework-artifact.sh`, `tooling/connect/xcframework-manifest.sh`. diff --git a/.agents/specs/connect/connect-cpp-binding.md b/.agents/specs/connect/connect-cpp-binding.md index 230d99f0..e214cad9 100644 --- a/.agents/specs/connect/connect-cpp-binding.md +++ b/.agents/specs/connect/connect-cpp-binding.md @@ -149,7 +149,7 @@ C++20 generator integration, vcpkg/Conan, public Release archives, additional pl - `crates/spoke-connect/bindings/cpp/native/provenance.json` — per-RID source revision, toolchain and artifact hashes (`headerSha256`, `hppSha256`, artifact SHA-256). - `crates/spoke-connect/bindings/cpp/Smoke/` (`main.cpp`, `convenience.cpp`, `support.hpp`) and `crates/spoke-connect/bindings/cpp/ue/` (`SpokeConnect.Build.cs`, `README.md`). - `crates/spoke-connect-capi/` — the workspace-private carrier crate (`publish = false`). -- `tooling/connect/cpp-symbol-check.mjs`, `tooling/connect/cpp-smoke.mjs`, `tooling/connect/cpp-build.mjs` — drift gate, smoke runner and native producer/verifier. +- `tooling/connect/cpp-symbol-check.mjs`, `tooling/connect/cpp-smoke.mjs`, `tooling/connect/cpp-build.mjs` — drift gate, smoke runner and native producer. - `.github/workflows/cpp-connect.yml` — per-PR carrier lanes; `.gitattributes` — git-lfs rules for the carrier libraries. - `crates/spoke-connect/tests/fixtures/golden-hello.json` — the golden hello bytes the smokes assert. diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 5e899871..95cb308a 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -248,12 +248,6 @@ jobs: cargo build -p spoke-connect --features ffi,remote-adapter cargo test -p spoke-connect --features ffi,remote-adapter - # First-party uniffi languages only (Swift, Python). C# and Go use vendored - # uniffi-bindgen forks under bindings/{csharp,go}/bindgen/ and are excluded. - - name: Verify binding regeneration (no drift) - if: needs.changes.outputs.code == 'true' - run: ./tooling/connect/verify-binding-regen.sh - csharp-connect: needs: changes name: csharp-connect diff --git a/.github/workflows/cpp-connect.yml b/.github/workflows/cpp-connect.yml index f5b7a0fc..4a312feb 100644 --- a/.github/workflows/cpp-connect.yml +++ b/.github/workflows/cpp-connect.yml @@ -121,12 +121,6 @@ jobs: throw "cl.exe version probe failed with exit code $LASTEXITCODE" } - - name: Verify committed Windows provenance - if: needs.changes.outputs.code == 'true' - id: committed_provenance - continue-on-error: true - run: node tooling/connect/cpp-build.mjs --verify --target x86_64-pc-windows-msvc - - name: Check committed C ABI symbols if: needs.changes.outputs.code == 'true' id: committed_symbols @@ -176,7 +170,7 @@ jobs: retention-days: 5 - name: Fail on committed carrier errors - if: always() && (steps.committed_provenance.outcome == 'failure' || steps.committed_symbols.outcome == 'failure' || steps.committed_smoke.outcome == 'failure') + if: always() && (steps.committed_symbols.outcome == 'failure' || steps.committed_smoke.outcome == 'failure') run: exit 1 @@ -236,12 +230,6 @@ jobs: - - name: Verify committed macOS provenance - if: needs.changes.outputs.code == 'true' - id: committed_provenance - continue-on-error: true - run: node tooling/connect/cpp-build.mjs --verify --target aarch64-apple-darwin - - name: Check C ABI symbols if: needs.changes.outputs.code == 'true' id: committed_symbols @@ -284,5 +272,5 @@ jobs: retention-days: 5 - name: Fail on committed carrier errors - if: always() && (steps.committed_provenance.outcome == 'failure' || steps.committed_symbols.outcome == 'failure' || steps.committed_smoke.outcome == 'failure') + if: always() && (steps.committed_symbols.outcome == 'failure' || steps.committed_smoke.outcome == 'failure') run: exit 1 diff --git a/.github/workflows/xcframework.yml b/.github/workflows/xcframework.yml index a05163f3..3bf41756 100644 --- a/.github/workflows/xcframework.yml +++ b/.github/workflows/xcframework.yml @@ -14,7 +14,7 @@ on: - crates/spoke-connect/Cargo.toml - Cargo.lock - tooling/connect/build-swift-xcframework.sh - - tooling/connect/verify-xcframework-drift.sh + - tooling/connect/xcframework-manifest.sh - crates/spoke-connect/bindings/swift/generated/** - crates/spoke-connect/bindings/swift/Smoke/** - Package.swift @@ -101,7 +101,7 @@ jobs: if: needs.changes.outputs.code == 'true' # The apply script verifies the transferred artifact against this manifest. run: | - ./tooling/connect/verify-xcframework-drift.sh --manifest \ + ./tooling/connect/xcframework-manifest.sh \ "${{ runner.temp }}/xcframework-stage/spoke_connectFFI.xcframework" \ > "${{ runner.temp }}/xcframework-stage/spoke_connectFFI.xcframework.sha256" @@ -131,13 +131,6 @@ jobs: if-no-files-found: error retention-days: 5 - - name: Verify committed xcframework freshness - if: needs.changes.outputs.code == 'true' - run: | - ./tooling/connect/verify-xcframework-drift.sh \ - crates/spoke-connect/bindings/swift/xcframework/spoke_connectFFI.xcframework \ - "${{ runner.temp }}/xcframework-stage/spoke_connectFFI.xcframework" - - name: Fail on committed Swift consumer smoke if: always() && steps.committed_smoke.outcome == 'failure' run: exit 1 diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index fee69f0e..3c956c5b 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -121,7 +121,7 @@ Refresh the committed artifact from a run in one command (requires the `gh` CLI, ./tooling/connect/apply-xcframework-artifact.sh ``` -The script verifies the run's provenance — the `Xcframework` workflow, completed producer steps for the build, manifest, provenance, and upload, and either success or failures limited to the committed freshness and consumer smoke checks. The producer record must name the run head as both its declared `sourceHead` and its actual `checkoutRevision`, so a run built from a different checkout — a pull-request virtual merge records the merge commit, not the PR head — can never be applied as a head build. The run `headSha` must match the current checkout (override with `--allow-sha ` only to deliberately pin an older build; the override never relaxes the producer-record checks). It verifies the artifact's hash manifest and producer record before applying the built tree and staging the LFS pointers. The copy itself is content-aware — `rsync -c` compares checksums rather than size and mtime, because same-size files sharing a coarse mtime would otherwise be skipped, reporting success while leaving the old bytes — and after the copy the script re-verifies the delivered tree against the artifact's hash manifest before any staging or sidecar write, so a transfer that reports success without delivering the authenticated bytes refuses with no staged changes and no provenance sidecar. Commit with the suggested line — `build(connect): refresh xcframework from CI artifact ` — using normal maintainer credentials. A fresh build may differ from a later rebuild; retain the actual two-tree comparison and investigate drift rather than treating successful application as reproducibility. +The script verifies the run's provenance — the `Xcframework` workflow, completed producer steps for the build, manifest, provenance, and upload, and either success or failures limited to the consumer-smoke check. The producer record must name the run head as both its declared `sourceHead` and its actual `checkoutRevision`, so a run built from a different checkout — a pull-request virtual merge records the merge commit, not the PR head — can never be applied as a head build. The run `headSha` must match the current checkout (override with `--allow-sha ` only to deliberately pin an older build; the override never relaxes the producer-record checks). It verifies the artifact's hash manifest and producer record before applying the built tree and staging the LFS pointers. The copy itself is content-aware — `rsync -c` compares checksums rather than size and mtime, because same-size files sharing a coarse mtime would otherwise be skipped, reporting success while leaving the old bytes — and after the copy the script re-verifies the delivered tree against the artifact's hash manifest before any staging or sidecar write, so a transfer that reports success without delivering the authenticated bytes refuses with no staged changes and no provenance sidecar. Commit with the suggested line — `build(connect): refresh xcframework from CI artifact ` — using normal maintainer credentials. A fresh build may differ from a later rebuild; retain the actual two-tree comparison and investigate drift rather than treating successful application as reproducibility. ## Refreshing the C carrier natives @@ -129,7 +129,7 @@ The script verifies the run's provenance — the `Xcframework` workflow, complet The Windows lane of [`cpp-connect.yml`](.github/workflows/cpp-connect.yml) checks out with `lfs: true`, builds the carrier (`node tooling/connect/cpp-build.mjs --target x86_64-pc-windows-msvc --toolchain 1.98.1`) and uploads `spoke-connect-win-x64`: the DLL, the import library and the refreshed `provenance.json`. Refresh the committed carrier by copying that artifact over `native/win-x64/` and committing the binaries together with the record that describes them — the recorded `headerSha256` / `hppSha256` / artifact hashes must match the committed bytes, never the other way round. -Both carrier lanes check the C ABI and run the C++ smoke against the committed carrier before building, then repeat those checks on the rebuilt carrier. `node tooling/connect/cpp-build.mjs --verify --target ` is available to verify the recorded hashes of the committed C and C++ headers and native. The C `.h` and C++ `.hpp` headers are pinned to LF (`text eol=lf` in `.gitattributes`) so the recorded `headerSha256` and `hppSha256` describe their bytes on every platform. +Both carrier lanes check the C ABI and run the C++ smoke against the committed carrier before building, then repeat those checks on the rebuilt carrier. The C `.h` and C++ `.hpp` headers are pinned to LF (`text eol=lf` in `.gitattributes`) so the recorded `headerSha256` and `hppSha256` describe their bytes on every platform. ## Release diff --git a/crates/spoke-connect/bindings/cpp/README.md b/crates/spoke-connect/bindings/cpp/README.md index c1897e62..d3b14718 100644 --- a/crates/spoke-connect/bindings/cpp/README.md +++ b/crates/spoke-connect/bindings/cpp/README.md @@ -139,13 +139,10 @@ source and refreshes that RID's `provenance.json` entry with the revision, toolchain and hashes of the shipped files. It accepts the two committed targets, `aarch64-apple-darwin` and `x86_64-pc-windows-msvc`. -`node tooling/connect/cpp-build.mjs --verify --target ` is the read-only -counterpart consumers run: it re-reads the committed entry and hashes the -committed C and C++ headers and staged native, so a copy that no longer matches the record -fails instead of being quietly re-recorded. The C `.h` and C++ `.hpp` headers -are pinned to LF (`text eol=lf` in `.gitattributes`) for that check's sake — -the recorded `headerSha256` and `hppSha256` describe their respective header -bytes on every platform, independent of checkout end-of-line conversion. +The C `.h` and C++ `.hpp` headers are pinned to LF (`text eol=lf` in +`.gitattributes`), so the recorded `headerSha256` and `hppSha256` describe their +respective header bytes on every platform, independent of checkout +end-of-line conversion. ## Header durability diff --git a/docs/reference/protocol.md b/docs/reference/protocol.md index 4c2b383b..7ab229bb 100644 --- a/docs/reference/protocol.md +++ b/docs/reference/protocol.md @@ -20,7 +20,7 @@ Six interaction envelopes ([`schemas/connect/`](https://github.com/42ch-dev/spok ## Schema inventory and codegen posture -The wire inventory is **114 committed `*.schema.json` files**: 2 common + 24 data + 82 ops + 6 connect envelopes. The optional `spoke-memory` family contributes 79 of them (13 data models + 33 request/response pairs). Baseline integrators use **21** of those files (2 common + 9 data + 10 baseline ops): the nine baseline data schemas include `ToolDescriptor`, while `CognitionRecord` and `MindState` are the two optional non-memory data files. `schemas/` is the only hand-authored wire truth; generated TypeScript (`@42ch/spoke-schemas`) and Rust (`spoke-schemas`) output is committed and mirrors the schema tree. `pnpm run verify-codegen` fails the build if the generated tree drifts from `schemas/`; schema changes and regenerated output land in the same commit. +The wire inventory is **114 committed `*.schema.json` files**: 2 common + 24 data + 82 ops + 6 connect envelopes. The optional `spoke-memory` family contributes 79 of them (13 data models + 33 request/response pairs). Baseline integrators use **21** of those files (2 common + 9 data + 10 baseline ops): the nine baseline data schemas include `ToolDescriptor`, while `CognitionRecord` and `MindState` are the two optional non-memory data files. `schemas/` is the only hand-authored wire truth; generated TypeScript (`@42ch/spoke-schemas`) and Rust (`spoke-schemas`) output is committed and mirrors the schema tree. `pnpm run verify-codegen` regenerates both trees, checks the opaque-JSON type contract and compiles the workspace; schema changes and regenerated output land in the same commit. ## Extensions contract diff --git a/docs/zh/reference/protocol.md b/docs/zh/reference/protocol.md index 5c41b00b..6653bbc9 100644 --- a/docs/zh/reference/protocol.md +++ b/docs/zh/reference/protocol.md @@ -20,7 +20,7 @@ SPOKE 是面向叙事产品的共享**线上方言(wire dialect)**:一套 ## Schema 清单与代码生成姿态 -线上清单为 **114 个已提交的 `*.schema.json`**:2 common + 24 data + 82 ops + 6 connect 信封。可选 `spoke-memory` 族贡献其中 79 个(13 个数据模型 + 33 组请求/响应配对)。基线集成方使用其中的 **21** 个 schema 文件(2 common + 9 data + 10 基线 ops):九个基线数据 schema 包含 `ToolDescriptor`,而 `CognitionRecord` 与 `MindState` 是两个可选的非记忆数据文件。`schemas/` 是唯一手写线上真源;生成的 TypeScript(`@42ch/spoke-schemas`)与 Rust(`spoke-schemas`)输出提交入库并镜像 schema 树。`pnpm run verify-codegen` 在生成树偏离 `schemas/` 时令构建失败;schema 变更与重新生成输出落在同一提交。 +线上清单为 **114 个已提交的 `*.schema.json`**:2 common + 24 data + 82 ops + 6 connect 信封。可选 `spoke-memory` 族贡献其中 79 个(13 个数据模型 + 33 组请求/响应配对)。基线集成方使用其中的 **21** 个 schema 文件(2 common + 9 data + 10 基线 ops):九个基线数据 schema 包含 `ToolDescriptor`,而 `CognitionRecord` 与 `MindState` 是两个可选的非记忆数据文件。`schemas/` 是唯一手写线上真源;生成的 TypeScript(`@42ch/spoke-schemas`)与 Rust(`spoke-schemas`)输出提交入库并镜像 schema 树。`pnpm run verify-codegen` 重新生成两棵生成树,检查 opaque JSON 类型契约并编译工作区;schema 变更与重新生成输出落在同一提交。 ## 扩展契约 diff --git a/package.json b/package.json index e443ab52..f2fbec02 100644 --- a/package.json +++ b/package.json @@ -12,7 +12,7 @@ "//": "Cold-clone gate is `ci:typescript` (inlined superset of the ci.yml `typescript` job: `build:operations` precedes every typecheck that resolves @42ch/spoke-operations from dist/, then the demo typechecks + `test:demo`). The composite `typecheck` script is a warm-tree helper only.", "scripts": { "codegen": "pnpm -F @42ch/spoke-codegen codegen", - "verify-codegen": "pnpm run codegen && node tooling/codegen/assert-schema-count.mjs && node tooling/codegen/assert-opaque-json-types.mjs && node --test tooling/codegen/test/deep-ref-hoist.test.mjs && node tooling/codegen/assert-generated-rust-size.mjs && git diff --exit-code -- packages/spoke-schemas/src/generated crates/spoke-schemas/src/generated && cargo check --workspace", + "verify-codegen": "pnpm run codegen && node tooling/codegen/assert-schema-count.mjs && node tooling/codegen/assert-opaque-json-types.mjs && node --test tooling/codegen/test/deep-ref-hoist.test.mjs && cargo check --workspace", "typecheck": "pnpm -F @42ch/spoke-schemas typecheck && pnpm -F @42ch/spoke-operations typecheck && pnpm -F @42ch/spoke-connect typecheck && pnpm -F @42ch/spoke-demo-server typecheck && pnpm -F @42ch/spoke-demo-client typecheck", "build:schema": "pnpm -F @42ch/spoke-schemas build", "build:operations": "pnpm -F @42ch/spoke-operations build", diff --git a/tooling/codegen/assert-generated-rust-size.mjs b/tooling/codegen/assert-generated-rust-size.mjs deleted file mode 100755 index 2fcec663..00000000 --- a/tooling/codegen/assert-generated-rust-size.mjs +++ /dev/null @@ -1,48 +0,0 @@ -#!/usr/bin/env node -/** - * Assert the formatted memory Rust files stay within MAX_BYTES. - * - * typify emits each file as one long line; `generateRust()` runs stable rustfmt on - * these three files. This bound catches pathological schema growth (e.g. runaway - * shared definitions) without flagging normal generation drift. - * - * Bump procedure: also record the measured post-format sizes in the task evidence. - * If a legitimate schema change pushes a file over the bound, raise MAX_BYTES in the - * same commit as the regenerated file. - */ - -import { statSync } from "node:fs"; -import { dirname, join } from "node:path"; -import { fileURLToPath } from "node:url"; - -const MAX_BYTES = 2 * 1024 * 1024; - -const REPO_ROOT = join(dirname(fileURLToPath(import.meta.url)), "../.."); -const GENERATED_DIR = join(REPO_ROOT, "crates/spoke-schemas/src/generated/ops"); -const FILES = [ - "memory_exports_get_response.rs", - "memory_imports_stage_request.rs", - "memory_receipts_get_response.rs", -]; - -let failed = false; - -for (const file of FILES) { - const bytes = statSync(join(GENERATED_DIR, file)).size; - - if (bytes > MAX_BYTES) { - failed = true; - console.error( - `Generated Rust file too large: ${file} is ${bytes} bytes (limit ${MAX_BYTES}).`, - ); - } else { - console.log(`Generated Rust size OK: ${file} ${bytes} bytes`); - } -} - -if (failed) { - console.error( - "Bump procedure: see header comment in tooling/codegen/assert-generated-rust-size.mjs", - ); - process.exit(1); -} diff --git a/tooling/codegen/assert-opaque-json-types.mjs b/tooling/codegen/assert-opaque-json-types.mjs index 07d640e5..0cb982dd 100755 --- a/tooling/codegen/assert-opaque-json-types.mjs +++ b/tooling/codegen/assert-opaque-json-types.mjs @@ -11,7 +11,7 @@ * This script fails when a field whose schema `$ref`s the canonical `OpaqueJson` definition is * emitted as an object map. * - * Wire it from `pnpm run verify-codegen` alongside `assert-schema-count.mjs`. + * Wire it from `pnpm run verify-codegen`. */ import { readdirSync, readFileSync, statSync } from "node:fs"; diff --git a/tooling/connect/apply-xcframework-artifact.sh b/tooling/connect/apply-xcframework-artifact.sh index 27d81c3d..7c3a6644 100755 --- a/tooling/connect/apply-xcframework-artifact.sh +++ b/tooling/connect/apply-xcframework-artifact.sh @@ -31,8 +31,8 @@ # objects). # # Provenance gate (fail-closed, runs before the download): accept successful -# runs or failures limited to the committed freshness and consumer smoke checks, -# with successful build, manifest, provenance and upload steps in either case. +# runs or failures limited to the consumer smoke check, with successful build, +# manifest, provenance and upload steps in either case. # The run head must match HEAD unless --allow-sha pins it. The override only # relaxes that destination check; the producer record must still show a # checkout of exactly the declared run head. @@ -137,7 +137,6 @@ try: "Upload xcframework artifact", } allowed_failures = { - "Verify committed xcframework freshness", "Fail on committed Swift consumer smoke", } if len(names) != len(set(names)): @@ -149,7 +148,7 @@ try: raise ValueError("incomplete or cancelled step") failures = {name for name, conclusion in outcomes.items() if conclusion == "failure"} if failures - allowed_failures or bool(failures) != (sys.argv[1] == "failure"): - raise ValueError("failure is not limited to committed freshness or consumer smoke") + raise ValueError("failure is not limited to the consumer smoke") except (KeyError, TypeError, ValueError) as error: print("error: invalid producer job: %s" % error, file=sys.stderr) sys.exit(1) diff --git a/tooling/connect/apply-xcframework-artifact.test.mjs b/tooling/connect/apply-xcframework-artifact.test.mjs index e823ecf6..81a3e48a 100644 --- a/tooling/connect/apply-xcframework-artifact.test.mjs +++ b/tooling/connect/apply-xcframework-artifact.test.mjs @@ -13,13 +13,12 @@ const required = [ "Record xcframework build provenance", "Upload xcframework artifact", ]; -const freshness = "Verify committed xcframework freshness"; const smoke = "Fail on committed Swift consumer smoke"; const digest = (text) => createHash("sha256").update(text).digest("hex"); const bundle = "crates/spoke-connect/bindings/swift/xcframework/spoke_connectFFI.xcframework"; function runFixture({ - failures = [freshness], missing, invalidSource, invalidManifest, invalidProvenance, invalidCheckout, + failures = [smoke], missing, invalidSource, invalidManifest, invalidProvenance, invalidCheckout, status = "completed", conclusion = "failure", jobStatus = "completed", jobConclusion, cancelledStep, advanceHead, allowSha, transferWithoutDelivery, } = {}) { @@ -100,7 +99,7 @@ function runFixture({ } describe("xcframework artifact application outcome gate", () => { - for (const failures of [[freshness], [freshness, smoke], [smoke]]) { + for (const failures of [[smoke]]) { it(`applies a validated artifact after ${failures.join(" and ")}`, () => { const { result, contents, record, producerSha } = runFixture({ failures }); assert.equal(result.status, 0, result.stderr); @@ -111,14 +110,14 @@ describe("xcframework artifact application outcome gate", () => { }); } for (const [label, options] of [ - ["unrelated failure", { failures: [freshness, "unrelated"] }], + ["unrelated failure", { failures: [smoke, "unrelated"] }], ["missing prerequisite", { missing: required[0] }], ["wrong source", { invalidSource: true }], ["invalid manifest", { invalidManifest: true }], ["invalid producer provenance", { invalidProvenance: true }], ["producer checkout that is not the run head", { invalidCheckout: true }], ["job conclusion does not match the run", { jobConclusion: "success" }], - ["cancelled producer step", { cancelledStep: freshness }], + ["cancelled producer step", { cancelledStep: smoke }], ["incomplete run", { status: "in_progress" }], ["cancelled run", { conclusion: "cancelled" }], ]) { diff --git a/tooling/connect/cpp-build.mjs b/tooling/connect/cpp-build.mjs index 72ce8fe3..6afc5185 100755 --- a/tooling/connect/cpp-build.mjs +++ b/tooling/connect/cpp-build.mjs @@ -33,7 +33,6 @@ * Usage: * node tooling/connect/cpp-build.mjs --target aarch64-apple-darwin --toolchain 1.98.1 * node tooling/connect/cpp-build.mjs --target x86_64-pc-windows-msvc --toolchain 1.98.1 - * node tooling/connect/cpp-build.mjs --verify --target aarch64-apple-darwin * * `--toolchain ` is passed to cargo as `+`; when omitted, the * caller's default toolchain is used. Both carrier lanes pin stable `1.98.1` @@ -42,13 +41,6 @@ * maintainer-local one — so build with that exact toolchain, not whichever * default happens to be installed. * - * `--verify` is the read-only consumer check: it re-reads the committed - * provenance and hashes the committed headers and staged native for one RID, - * failing when a recorded SHA-256 no longer describes those bytes. It builds - * nothing and writes nothing, so it runs against any checkout — including one - * whose header bytes drifted from the record (CRLF from a Windows - * `core.autocrlf=true` checkout), which is the drift the build path hides by - * rewriting the record it just hashed. */ import { createHash } from "node:crypto"; @@ -109,7 +101,7 @@ function fail(message) { } function parseArgs(argv) { - const args = { target: null, toolchain: null, verify: false }; + const args = { target: null, toolchain: null }; for (let index = 0; index < argv.length; index += 1) { const flag = argv[index]; if (flag === "--target") { @@ -118,19 +110,14 @@ function parseArgs(argv) { } else if (flag === "--toolchain") { args.toolchain = argv[index + 1]; index += 1; - } else if (flag === "--verify") { - args.verify = true; } else { - fail(`unknown argument '${flag}' (usage: --target [--toolchain ] [--verify])`); + fail(`unknown argument '${flag}' (usage: --target [--toolchain ])`); } } if (!args.target) fail("--target is required"); if (!TARGETS[args.target]) { fail(`unsupported target '${args.target}' (supported: ${Object.keys(TARGETS).join(", ")})`); } - if (args.verify && args.toolchain) { - fail("--verify only reads the committed files; drop --toolchain rather than ignoring it"); - } return args; } @@ -261,67 +248,10 @@ export function readProvenance(path) { return provenance; } -/** - * The committed entry for `spec.rid` has to describe the bytes a consumer - * receives: both headers and every staged native the entry lists. The - * read-only shape is the point — the build path records what it just hashed, - * so only a reader can catch a checkout whose bytes no longer match the record. - */ -function verifyProvenance(spec) { - const entry = readProvenance(PROVENANCE).nativeArtifacts[spec.rid]; - if (!entry) { - fail(`provenance has no '${spec.rid}' entry, so nothing records what was shipped`); - } - - const problems = []; - const headerSha256 = sha256(HEADER); - if (entry.headerSha256 !== headerSha256) { - problems.push( - `header: recorded ${entry.headerSha256}, committed bytes ${headerSha256} ` + - `(${relative(REPO_ROOT, HEADER)})`, - ); - } - const hppSha256 = sha256(HPP); - if (entry.hppSha256 !== hppSha256) { - problems.push( - `hpp: recorded ${entry.hppSha256}, committed bytes ${hppSha256} ` + - `(${relative(REPO_ROOT, HPP)})`, - ); - } - for (const artifact of spec.artifacts) { - const recorded = (entry.artifacts ?? []).find((item) => item.path === artifact.staged); - const actual = sha256(join(CPP_DIR, "native", spec.rid, artifact.staged)); - if (!recorded) { - problems.push(`native/${spec.rid}/${artifact.staged}: missing from the entry`); - } else if (recorded.sha256 !== actual) { - problems.push( - `native/${spec.rid}/${artifact.staged}: recorded ${recorded.sha256}, ` + - `committed bytes ${actual}`, - ); - } - } - - if (problems.length > 0) { - fail( - `committed provenance does not describe the committed ${spec.rid} bytes:\n ` + - problems.join("\n "), - ); - } - console.log( - `cpp-build: verified ${spec.rid} provenance (header ${headerSha256}, ` + - `hpp ${hppSha256}, natives ${spec.artifacts.length})`, - ); -} - function main() { const args = parseArgs(process.argv.slice(2)); const spec = TARGETS[args.target]; - if (args.verify) { - verifyProvenance(spec); - return; - } - assertBuildSourcesCommitted(); const cargoArgs = ["build", "--locked", "-p", CARRIER_PACKAGE, "--release", "--target", args.target, "--target-dir", TARGET_DIR_FLAG]; diff --git a/tooling/connect/cpp-build.test.mjs b/tooling/connect/cpp-build.test.mjs index 6a06a46d..1845b8f9 100644 --- a/tooling/connect/cpp-build.test.mjs +++ b/tooling/connect/cpp-build.test.mjs @@ -154,81 +154,3 @@ describe("dependency source identity", () => { }); } }); - -/** - * `--verify` is the read-only consumer gate wired into CI: it must accept the - * bytes the build path just recorded and fail closed on any later drift, - * including a committed entry that lost the `.hpp` hash altogether. - */ -describe("verify committed provenance", () => { - const PROVENANCE_PATH = "crates/spoke-connect/bindings/cpp/native/provenance.json"; - const setupVerifyFixture = () => { - const root = realpathSync(mkdtempSync(join(tmpdir(), "cpp-build-verify-"))); - const bin = join(root, "bin"); - const script = join(root, "tooling/connect/cpp-build.mjs"); - const git = (...args) => execFileSync("git", args, { cwd: root, encoding: "utf8" }).trim(); - const put = (path, content, options) => { - const absolute = join(root, path); - mkdirSync(dirname(absolute), { recursive: true }); - writeFileSync(absolute, content, options); - }; - const run = (...args) => spawnSync(process.execPath, [script, ...args], { - cwd: root, - encoding: "utf8", - env: { ...process.env, PATH: `${bin}:${process.env.PATH}` }, - }); - mkdirSync(dirname(script), { recursive: true }); - copyFileSync(fileURLToPath(new URL("./cpp-build.mjs", import.meta.url)), script); - git("init", "-q"); - git("config", "user.name", "Fixture"); - git("config", "user.email", "fixture@example.invalid"); - put("Cargo.toml", "workspace\n"); - put("crates/spoke-connect/src/lib.rs", "initial\n"); - put("crates/spoke-connect/Cargo.toml", "package\n"); - put("crates/spoke-connect/bindings/cpp/include/spoke_connect.h", "header\n"); - put("crates/spoke-connect/bindings/cpp/include/spoke_connect.hpp", "header\n"); - git("add", "."); - git("commit", "-qm", "initial"); - put("bin/cargo", "#!/bin/sh\nmkdir -p target/cpp/x86_64-pc-windows-msvc/release\nprintf native > target/cpp/x86_64-pc-windows-msvc/release/spoke_connect_capi.dll\nprintf import > target/cpp/x86_64-pc-windows-msvc/release/spoke_connect_capi.dll.lib\n", { mode: 0o755 }); - put("bin/rustc", "#!/bin/sh\nprintf 'rustc fixture\\n'\n", { mode: 0o755 }); - put("bin/cl.exe", "#!/bin/sh\nprintf 'Compiler Version fixture\\n'\n", { mode: 0o755 }); - const build = run("--target", "x86_64-pc-windows-msvc"); - assert.equal(build.status, 0, build.stderr); - const verify = () => run("--verify", "--target", "x86_64-pc-windows-msvc"); - return { root, verify, put, PROVENANCE_PATH }; - }; - - it("passes on the bytes the build path just recorded", () => { - const { root, verify } = setupVerifyFixture(); - try { - const result = verify(); - assert.equal(result.status, 0, result.stderr); - } finally { - rmSync(root, { recursive: true, force: true }); - } - }); - - it("fails closed when the committed entry has no .hpp hash", () => { - const { root, verify, put, PROVENANCE_PATH } = setupVerifyFixture(); - try { - const provenance = JSON.parse(readFileSync(join(root, PROVENANCE_PATH), "utf8")); - delete provenance.nativeArtifacts["win-x64"].hppSha256; - put(PROVENANCE_PATH, `${JSON.stringify(provenance, null, 2)}\n`); - const result = verify(); - assert.notEqual(result.status, 0); - } finally { - rmSync(root, { recursive: true, force: true }); - } - }); - - it("fails closed when the committed .hpp bytes drift from the record", () => { - const { root, verify, put } = setupVerifyFixture(); - try { - put("crates/spoke-connect/bindings/cpp/include/spoke_connect.hpp", "drifted header\n"); - const result = verify(); - assert.notEqual(result.status, 0); - } finally { - rmSync(root, { recursive: true, force: true }); - } - }); -}); diff --git a/tooling/connect/verify-binding-regen.sh b/tooling/connect/verify-binding-regen.sh deleted file mode 100755 index d04db0f7..00000000 --- a/tooling/connect/verify-binding-regen.sh +++ /dev/null @@ -1,84 +0,0 @@ -#!/usr/bin/env bash -# Verify committed Swift/Python uniffi bindings match fresh bindgen output (no drift). -# -# Usage (from repo root): -# ./tooling/connect/verify-binding-regen.sh -# -# C# and Go are intentionally excluded: they require vendored uniffi-bindgen forks -# under crates/spoke-connect/bindings/{csharp,go}/bindgen/ (see connect-uniffi-bindgen-fork.md). - -set -euo pipefail - -REPO_ROOT="$(cd "$(dirname "$0")/../.." && pwd)" -cd "${REPO_ROOT}" - -if command -v rustup >/dev/null 2>&1; then - if ! rustup toolchain list 2>/dev/null | grep -q '^nightly'; then - echo "==> install nightly toolchain (binding regen)" - rustup toolchain install nightly --profile minimal - fi -fi - -CARGO=(cargo +nightly) - -FFI_FEATURES="ffi,remote-adapter" -BINDGEN_FEATURES="ffi,bindgen-cli,remote-adapter" -STAGING="/tmp/binding-drift-check" -SWIFT_OUT="${STAGING}/swift" -PYTHON_OUT="${STAGING}/python" - -COMMITTED_SWIFT="${REPO_ROOT}/crates/spoke-connect/bindings/swift/generated" -COMMITTED_PYTHON="${REPO_ROOT}/crates/spoke-connect/bindings/python/spoke_connect/__init__.py" - -rm -rf "${STAGING}" -mkdir -p "${SWIFT_OUT}" "${PYTHON_OUT}" - -echo "==> build production cdylib (${FFI_FEATURES})" -"${CARGO[@]}" build -p spoke-connect --features "${FFI_FEATURES}" - -TARGET_DIR="$("${CARGO[@]}" metadata --no-deps --format-version 1 | python3 -c 'import json,sys; print(json.load(sys.stdin)["target_directory"])')" -CDYLIB="${TARGET_DIR}/debug/libspoke_connect.so" -if [[ ! -f "${CDYLIB}" ]]; then - CDYLIB="${TARGET_DIR}/debug/libspoke_connect.dylib" -fi -if [[ ! -f "${CDYLIB}" ]]; then - CDYLIB="${TARGET_DIR}/debug/spoke_connect.dll" -fi -if [[ ! -f "${CDYLIB}" ]]; then - echo "error: cdylib not found under ${TARGET_DIR}/debug/" >&2 - exit 1 -fi - -echo "==> regenerate Swift bindings" -"${CARGO[@]}" run -p spoke-connect --features "${BINDGEN_FEATURES}" --bin uniffi-bindgen -- \ - generate --library "${CDYLIB}" --language swift --out-dir "${SWIFT_OUT}" - -echo "==> regenerate Python bindings" -"${CARGO[@]}" run -p spoke-connect --features "${BINDGEN_FEATURES}" --bin uniffi-bindgen -- \ - generate --library "${CDYLIB}" --language python --out-dir "${PYTHON_OUT}" - -REGEN_PYTHON="${PYTHON_OUT}/spoke_connect.py" -if [[ ! -f "${REGEN_PYTHON}" ]]; then - echo "error: expected ${REGEN_PYTHON} from python bindgen" >&2 - exit 1 -fi - -DRIFT=0 - -echo "==> diff Swift generated/" -if ! diff -ru "${SWIFT_OUT}" "${COMMITTED_SWIFT}"; then - echo "::error::Committed Swift bindings differ from fresh uniffi-bindgen output. Regenerate per tooling/connect/build-swift-xcframework.sh." >&2 - DRIFT=1 -fi - -echo "==> diff Python __init__.py" -if ! diff -u "${REGEN_PYTHON}" "${COMMITTED_PYTHON}"; then - echo "::error::Committed Python bindings differ from fresh uniffi-bindgen output. Regenerate per crates/spoke-connect/bindings/python/README.md." >&2 - DRIFT=1 -fi - -if [[ "${DRIFT}" -ne 0 ]]; then - exit 1 -fi - -echo "==> binding regeneration OK (no drift)" diff --git a/tooling/connect/verify-xcframework-drift.sh b/tooling/connect/verify-xcframework-drift.sh deleted file mode 100755 index 65d356a7..00000000 --- a/tooling/connect/verify-xcframework-drift.sh +++ /dev/null @@ -1,148 +0,0 @@ -#!/usr/bin/env bash -# Verify that a CI-built xcframework matches the committed one, file by file. -# -# Usage (from repo root): -# ./tooling/connect/verify-xcframework-drift.sh -# ./tooling/connect/verify-xcframework-drift.sh --manifest -# -# The committed xcframework IS the baseline. When a `provenance.json` sidecar -# sits beside the committed bundle, this script also checks that the committed -# producer manifest still describes the committed bytes and that the record -# agrees about the manifest digest and inventory; a tree without a sidecar -# claims no provenance, so plain temporary/older invocations stay valid. For -# each tree it emits one -# " " -# line per file (SHA-256 via `shasum -a 256`, paths sorted with LC_ALL=C), -# diffs the two manifests, prints the unified diff, and exits non-zero on any -# mismatch. Run after an LFS-smudged checkout so holds real -# bytes, not LFS pointers. -# -# `shasum -a 256` is the BSD/macOS standard and the macOS-runner-locked choice — -# GNU coreutils' sha256sum is not guaranteed on macOS images, so keep this -# script on macOS runners. -# -# --manifest emits the same sorted per-file manifest on stdout and is -# the single generator for the artifact manifest format (the xcframework -# workflow's "Write xcframework hash manifest" step calls this mode; -# apply-xcframework-artifact.sh consumes it via `shasum -a 256 -c`). - -set -euo pipefail - -# simplify: newline-separated paths assume no filenames with newlines. The -# xcframework layout is fixed (Info.plist, /libspoke_connect.a, -# Headers/*), so a byte-exact per-file manifest over LF-split names suffices. -manifest() { - local dir="$1" - (cd "${dir}" && find . -type f | LC_ALL=C sort | xargs shasum -a 256) -} - -if [[ "${1:-}" == "--manifest" ]]; then - if [[ $# -ne 2 || ! -d "$2" ]]; then - echo "usage: $0 --manifest " >&2 - exit 2 - fi - manifest "$2" - exit 0 -fi - -if [[ $# -ne 2 ]]; then - echo "usage: $0 " >&2 - exit 2 -fi - -COMMITTED_DIR="$1" -BUILT_DIR="$2" - -for dir in "${COMMITTED_DIR}" "${BUILT_DIR}"; do - if [[ ! -d "${dir}" ]]; then - echo "error: not a directory: ${dir}" >&2 - exit 2 - fi -done - -WORK="$(mktemp -d)" -trap 'rm -rf "${WORK}"' EXIT - -manifest "${COMMITTED_DIR}" > "${WORK}/committed.sha256" -manifest "${BUILT_DIR}" > "${WORK}/built.sha256" - -rc=0 -if diff -u "${WORK}/committed.sha256" "${WORK}/built.sha256" > "${WORK}/drift.diff"; then - echo "no drift: ${COMMITTED_DIR} matches ${BUILT_DIR} ($(wc -l < "${WORK}/committed.sha256" | tr -d ' ') files)" -else - echo "drift detected between committed and built xcframework:" >&2 - cat "${WORK}/drift.diff" >&2 - echo "commit count: committed $(wc -l < "${WORK}/committed.sha256" | tr -d ' ') files, built $(wc -l < "${WORK}/built.sha256" | tr -d ' ') files" >&2 - rc=1 -fi - -# The two-tree compare above proves "committed == rebuilt". It says nothing -# about the provenance sidecar committed beside the bundle, which is the only -# durable statement about where those bytes came from: a hand-edited bundle, a -# replaced LFS object or a mix of two apply runs would leave a confident-looking -# record describing different content. A tree with no sidecar claims no -# provenance, so plain temporary/older invocations remain valid. -BUNDLE_PARENT="$(cd "$(dirname "${COMMITTED_DIR}")" && pwd)" -BUNDLE_NAME="$(basename "${COMMITTED_DIR}")" -PROVENANCE="${BUNDLE_PARENT}/provenance.json" -NAMED_MANIFEST="${BUNDLE_PARENT}/${BUNDLE_NAME}.sha256" - -if [[ -f "${PROVENANCE}" ]]; then - if ! SIDECAR="$( - python3 - "${PROVENANCE}" <<'PY' -import json -import sys - -try: - with open(sys.argv[1], encoding="utf-8") as handle: - entry = json.load(handle)["xcframework"] - manifest = entry["manifest"] - digest = entry["manifestSha256"] -except Exception as error: # malformed, wrong shape or missing fields - print("error: unreadable provenance record: %s" % error, file=sys.stderr) - raise SystemExit(1) -print(manifest) -print(digest) -PY - )"; then - echo "error: malformed provenance sidecar ${PROVENANCE}" >&2 - rc=1 - else - MANIFEST_NAME="${SIDECAR%%$'\n'*}" - MANIFEST_SHA="${SIDECAR#*$'\n'}" - MANIFEST_SHA="${MANIFEST_SHA%$'\n'}" - if [[ ! "${MANIFEST_NAME}" =~ ^[A-Za-z0-9._-]+$ ]]; then - echo "error: provenance names an unusable manifest path '${MANIFEST_NAME}'" >&2 - rc=1 - else - MANIFEST_PATH="${BUNDLE_PARENT}/${MANIFEST_NAME}" - if [[ ! -f "${MANIFEST_PATH}" ]]; then - echo "error: provenance records manifest '${MANIFEST_NAME}', which is not committed beside the bundle" >&2 - rc=1 - else - ACTUAL_SHA="$(shasum -a 256 "${MANIFEST_PATH}" | cut -d' ' -f1)" - if [[ "${ACTUAL_SHA}" != "${MANIFEST_SHA}" ]]; then - echo "error: manifest digest drift: record ${MANIFEST_SHA}, committed ${ACTUAL_SHA}" >&2 - rc=1 - fi - if ! ( cd "${COMMITTED_DIR}" && shasum -a 256 -c "${MANIFEST_PATH}" ); then - echo "error: committed bundle disagrees with the recorded producer manifest" >&2 - rc=1 - fi - ( cd "${COMMITTED_DIR}" && find . -type f | LC_ALL=C sort ) > "${WORK}/tree.files" - awk '{ $1 = ""; sub(/^ +/, ""); print }' "${MANIFEST_PATH}" \ - | LC_ALL=C sort > "${WORK}/manifest.files" - if ! diff -u "${WORK}/manifest.files" "${WORK}/tree.files" > "${WORK}/inventory.diff"; then - echo "error: committed bundle inventory disagrees with the recorded manifest:" >&2 - cat "${WORK}/inventory.diff" >&2 - rc=1 - fi - fi - fi - fi -elif [[ -f "${NAMED_MANIFEST}" ]]; then - echo "error: ${NAMED_MANIFEST} is committed without ${PROVENANCE} — incomplete provenance sidecar" >&2 - rc=1 -fi - -exit "${rc}" diff --git a/tooling/connect/xcframework-manifest.sh b/tooling/connect/xcframework-manifest.sh new file mode 100755 index 00000000..1f77405b --- /dev/null +++ b/tooling/connect/xcframework-manifest.sh @@ -0,0 +1,27 @@ +#!/usr/bin/env bash +# Emit the xcframework transfer manifest: one " " line +# per file, sorted by path. +# +# Usage (from repo root): +# ./tooling/connect/xcframework-manifest.sh +# +# This is the single generator for the artifact manifest format: the +# xcframework workflow writes it beside the uploaded bundle and +# apply-xcframework-artifact.sh consumes it with `shasum -a 256 -c` before and +# after the transfer. +# +# `shasum -a 256` is the BSD/macOS standard and the macOS-runner-locked choice — +# GNU coreutils' sha256sum is not guaranteed on macOS images, so keep this +# script on macOS runners. + +set -euo pipefail + +if [[ $# -ne 1 || ! -d "$1" ]]; then + echo "usage: $0 " >&2 + exit 2 +fi + +# simplify: newline-separated paths assume no filenames with newlines. The +# xcframework layout is fixed (Info.plist, /libspoke_connect.a, +# Headers/*), so a byte-exact per-file manifest over LF-split names suffices. +(cd "$1" && find . -type f | LC_ALL=C sort | xargs shasum -a 256) From 7b29e0c81e641113f329060298ce0dfac8454e98 Mon Sep 17 00:00:00 2001 From: Bohao Tang Date: Mon, 16 Nov 2026 17:21:21 +0800 Subject: [PATCH 2/2] docs: align verify-codegen and carrier refresh text with current checks --- .agents/knowledge/README.md | 2 +- .../architecture-patterns/spoke-codegen-pipeline.md | 2 +- CONTRIBUTING.md | 6 +++--- 3 files changed, 5 insertions(+), 5 deletions(-) diff --git a/.agents/knowledge/README.md b/.agents/knowledge/README.md index 2e9fd1b1..2db0a6c4 100644 --- a/.agents/knowledge/README.md +++ b/.agents/knowledge/README.md @@ -6,7 +6,7 @@ Durable protocol and tooling facts. Normative specs live in [`specs/`](../specs/ | Doc | Category | Source | Summary | |-----|----------|--------|---------| -| [`architecture-patterns/spoke-codegen-pipeline.md`](architecture-patterns/spoke-codegen-pipeline.md) | architecture-patterns | current `schemas/**/*.schema.json`, `tooling/codegen` and generated trees | jstt+typify orchestrator over **114** hand-authored schemas; each generated tree is **114 modules + 5 barrels = 119 files**; `verify-codegen` regenerates both trees, asserts the schema count and opaque-JSON types, then runs `cargo check --workspace`; rust-gen fail-fast; **presence-sensitive arrays** — six byte-bound modules module-wide, plus `Content.cognition` in all 15 carrier modules / 17 clone types (unrelated arrays keep `Vec` + `is_empty`); typify nominal-clone canonical-import guide and Builder construction; **regeneration ripple** — zero-drift proof covers generated bytes, not downstream compilability (land ABI + consumer updates together) | +| [`architecture-patterns/spoke-codegen-pipeline.md`](architecture-patterns/spoke-codegen-pipeline.md) | architecture-patterns | current `schemas/**/*.schema.json`, `tooling/codegen` and generated trees | jstt+typify orchestrator over **114** hand-authored schemas; each generated tree is **114 modules + 5 barrels = 119 files**; `verify-codegen` regenerates both trees, asserts the schema count and opaque-JSON types, then runs `cargo check --workspace`; rust-gen fail-fast; **presence-sensitive arrays** — six byte-bound modules module-wide, plus `Content.cognition` in all 15 carrier modules / 17 clone types (unrelated arrays keep `Vec` + `is_empty`); typify nominal-clone canonical-import guide and Builder construction; **regeneration ripple** — compiling generated bytes does not cover downstream handlers (land ABI + consumer updates together) | | [`architecture-patterns/mind-axis-ownership-boundary.md`](architecture-patterns/mind-axis-ownership-boundary.md) | architecture-patterns | current cognition schema, data contract and Memory contract | One current authority per holder: `KnowledgeEntry.modules.cognition`; open `modules.mental` keeps nine fields with `beliefs` as derivative summary; `WorldFactRecord` stays world-side; `MindState`/`MindDelta` are derivative L5 presentation with RFC 6901 pointers; Memory `Content.cognition` is recorded-at-time and never mutates current authority; complete `(kind, namespace, id)` reference identity; `assessed` vs `no_record` projection; dual-SSOT prevention and rejection table | | [`architecture-patterns/mind-axis-ops-extraction-gate.md`](architecture-patterns/mind-axis-ops-extraction-gate.md) | architecture-patterns | current `spoke-operations` cognition exports and shipped consumers | `spoke-operations` ships only pure validators (cognition record/content/module/observation/filter/selection/projection, MindState RFC 6901 pointer gate) plus deterministic match/select/project serving KE `Scope`/assemble and Memory `cognition_filter`; checker Finding patterns, belief revision, delta apply/replay, ordering, scoring and ranking stay out; further helper families gated on two consumers with dual-language parity and shared vectors | | [`architecture-patterns/connect-error-taxonomy-layering.md`](architecture-patterns/connect-error-taxonomy-layering.md) | architecture-patterns | compound 2026-08-13 (protocol_version_mismatch) | Four-layer error chain (CoreError→ConnectError→RemoteAdapterError→FfiError); dual FFI surface; ordinal stability; frozen contract rule | diff --git a/.agents/knowledge/architecture-patterns/spoke-codegen-pipeline.md b/.agents/knowledge/architecture-patterns/spoke-codegen-pipeline.md index 9d18642b..c894e9a9 100644 --- a/.agents/knowledge/architecture-patterns/spoke-codegen-pipeline.md +++ b/.agents/knowledge/architecture-patterns/spoke-codegen-pipeline.md @@ -79,7 +79,7 @@ Regenerating binding trees changes the **interface downstream host code must imp - The ripple lands **outside** the generated trees, so a committed regeneration can leave channels unbuildable when their handlers were not updated in the same change. - Per-task diff review cannot see this class: each diff looks correct in isolation, and the breakage appears only when the generated tree and its consumers compile together. -- Handle it deliberately: land the generated ABI and the downstream handler updates as **one task**, or make the regenerating task's acceptance include "the consuming smoke/host sources still compile". Zero-drift proof covers generated bytes; it says nothing about downstream compilability. +- Handle it deliberately: land the generated ABI and the downstream handler updates as **one task**, or make the regenerating task's acceptance include "the consuming smoke/host sources still compile". Compiling the generated trees says nothing about downstream handler sources. ## What not to do diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 3c956c5b..70618e7e 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -24,7 +24,7 @@ Useful scripts (repo root): | Script | Purpose | |--------|---------| | `pnpm run codegen` | Regenerate TypeScript and Rust types from `schemas/` | -| `pnpm run verify-codegen` | Codegen + schema-count assert + `git diff` on generated trees | +| `pnpm run verify-codegen` | Codegen + schema-count assert + opaque-JSON type contract + `cargo check --workspace` | | `pnpm run typecheck` | Typecheck all workspace TS packages (`@42ch/spoke-schemas`, `@42ch/spoke-operations`, `@42ch/spoke-connect`, `@42ch/spoke-demo-server`, `@42ch/spoke-demo-client`) — warm-tree helper; the cold-clone gate is `ci:typescript` | | `pnpm run ci:typescript` | Cold-clone gate mirroring the ci.yml `typescript` job stage chain in dependency order, then the demo typechecks + `test:demo` | | `pnpm run build:schema` / `build:operations` | Build published npm packages | @@ -121,13 +121,13 @@ Refresh the committed artifact from a run in one command (requires the `gh` CLI, ./tooling/connect/apply-xcframework-artifact.sh ``` -The script verifies the run's provenance — the `Xcframework` workflow, completed producer steps for the build, manifest, provenance, and upload, and either success or failures limited to the consumer-smoke check. The producer record must name the run head as both its declared `sourceHead` and its actual `checkoutRevision`, so a run built from a different checkout — a pull-request virtual merge records the merge commit, not the PR head — can never be applied as a head build. The run `headSha` must match the current checkout (override with `--allow-sha ` only to deliberately pin an older build; the override never relaxes the producer-record checks). It verifies the artifact's hash manifest and producer record before applying the built tree and staging the LFS pointers. The copy itself is content-aware — `rsync -c` compares checksums rather than size and mtime, because same-size files sharing a coarse mtime would otherwise be skipped, reporting success while leaving the old bytes — and after the copy the script re-verifies the delivered tree against the artifact's hash manifest before any staging or sidecar write, so a transfer that reports success without delivering the authenticated bytes refuses with no staged changes and no provenance sidecar. Commit with the suggested line — `build(connect): refresh xcframework from CI artifact ` — using normal maintainer credentials. A fresh build may differ from a later rebuild; retain the actual two-tree comparison and investigate drift rather than treating successful application as reproducibility. +The script verifies the run's provenance — the `Xcframework` workflow, completed producer steps for the build, manifest, provenance, and upload, and either success or failures limited to the consumer-smoke check. The producer record must name the run head as both its declared `sourceHead` and its actual `checkoutRevision`, so a run built from a different checkout — a pull-request virtual merge records the merge commit, not the PR head — can never be applied as a head build. The run `headSha` must match the current checkout (override with `--allow-sha ` only to deliberately pin an older build; the override never relaxes the producer-record checks). It verifies the artifact's hash manifest and producer record before applying the built tree and staging the LFS pointers. The copy itself is content-aware — `rsync -c` compares checksums rather than size and mtime, because same-size files sharing a coarse mtime would otherwise be skipped, reporting success while leaving the old bytes — and after the copy the script re-verifies the delivered tree against the artifact's hash manifest before any staging or sidecar write, so a transfer that reports success without delivering the authenticated bytes refuses with no staged changes and no provenance sidecar. Commit with the suggested line — `build(connect): refresh xcframework from CI artifact ` — using normal maintainer credentials. Builds are not bit-reproducible across runs; the transfer manifest and producer record identify the applied bytes. ## Refreshing the C carrier natives `crates/spoke-connect/bindings/cpp/native/` holds the two committed C carriers (`osx-arm64/libspoke_connect_capi.dylib`, `win-x64/spoke_connect_capi.dll`) with the Windows import library and `provenance.json`. The dynamic libraries are git-lfs objects; `spoke_connect_capi.dll.lib` and `provenance.json` stay ordinary Git objects, so a checkout can read and diff the record without an LFS fetch. A converted or refreshed carrier needs `git add --renormalize ` — a plain `git add` sees unchanged bytes and stages nothing. -The Windows lane of [`cpp-connect.yml`](.github/workflows/cpp-connect.yml) checks out with `lfs: true`, builds the carrier (`node tooling/connect/cpp-build.mjs --target x86_64-pc-windows-msvc --toolchain 1.98.1`) and uploads `spoke-connect-win-x64`: the DLL, the import library and the refreshed `provenance.json`. Refresh the committed carrier by copying that artifact over `native/win-x64/` and committing the binaries together with the record that describes them — the recorded `headerSha256` / `hppSha256` / artifact hashes must match the committed bytes, never the other way round. +The Windows lane of [`cpp-connect.yml`](.github/workflows/cpp-connect.yml) checks out with `lfs: true`, builds the carrier (`node tooling/connect/cpp-build.mjs --target x86_64-pc-windows-msvc --toolchain 1.98.1`) and uploads `spoke-connect-win-x64`: the DLL, the import library and the refreshed `provenance.json`. Refresh the committed carrier by copying that artifact over `native/win-x64/` and committing the binaries together with the `provenance.json` the build wrote. Both carrier lanes check the C ABI and run the C++ smoke against the committed carrier before building, then repeat those checks on the rebuilt carrier. The C `.h` and C++ `.hpp` headers are pinned to LF (`text eol=lf` in `.gitattributes`) so the recorded `headerSha256` and `hppSha256` describe their bytes on every platform.