Repository navigation
fix(ci): verify committed codegen output instead of a regenerated worktree - #177
Conversation
[Medium impact] The PR appears safe to merge. Summary
No actionable issues found. Reviews (1) · Last reviewed commit: "fix(ci): verify committed codegen output..." · Reviewed by Greptile |
✅ Verdict:
|
| Findings | Count |
|---|---|
| 🔴 must-fix | 0 |
| 🟠 should-fix | 0 |
| 🟡 may-fix | 2 |
| 🔵 nit | 1 |
📋 Review
What this PR does: This PR is a follow-up to #176 that reshapes verify-codegen: it drops the leading pnpm run codegen step so the chain validates the committed generated trees (schema count, opaque-JSON contract, deep-ref-hoist test, @42ch/spoke-schemas typecheck, cargo check --workspace) instead of compiling freshly regenerated worktree files. Alongside it the PR rewrites the surrounding documentation so the stated contract matches what is actually gated — codegen as an author step, provenance.json hashes as build audit data rather than a CI gate, committed C carriers covered by the symbol check and C++ smoke, and reproducibility scoped to identical inputs. The settled verdict is ship it at 88% (good) with no must-fix or should-fix items; what remains is one knowledge-doc posture correction, one residual wording nit, and the question of whether removing the generator from CI should be paired with a re-added generatability or dirty-tree check.
Findings
🟡 Knowledge doc presents CI as verifying committed generated trees, but no chain step compares them to schemas/
-
Evidence:
[DOCS-01] Knowledge doc presents CI as verifying committed generated trees, but no chain step compares them to
schemas/- Evidence:
.agents/knowledge/architecture-patterns/spoke-codegen-pipeline.md:31— Decision step 3 now reads: "pnpm run verify-codegenchecks the committed trees:tooling/codegen/assert-schema-count.mjs→tooling/codegen/assert-opaque-json-types.mjs→node --test …deep-ref-hoist.test.mjs→@42ch/spoke-schemastypecheck →cargo check --workspace. Regeneration is the author step (pnpm run codegen); CI verifies the committed output that ships, never a regenerated worktree." - Evidence:
tooling/codegen/assert-schema-count.mjs:29-31—EXPECTED_SCHEMA_COUNT = 114; the script only countsschemas/**/*.schema.jsonfiles and never enumeratespackages/spoke-schemas/src/generatedor the Rust generated tree, so a schema body edit that skips regeneration leaves the count at 114 and passes. - Evidence:
tooling/codegen/assert-opaque-json-types.mjs:1-20— the checker reads committed generated TS (packages/spoke-schemas/src/generated) and asserts the opaque-JSON shape; it is self-consistent against stale output, not proof of freshness. - Evidence:
.github/workflows/ci.yml:87-89— theverify-codegenjob runs onlypnpm run verify-codegen;.github/workflows/release.yml:163-164does the same before publish. Neither adds a post-stepgit diff --exit-code, and grep across.github/workflows/finds no othercodegeninvocation. - Impact: The doc is the durable tooling fact surface loaded before touching
schemas/. Its "CI verifies the committed output that ships" wording, placed immediately after removal of the regeneration step, tells a reader that a greenverify-codegencertifies the committed trees are current with the hand-authored SSOT declared in the same doc's Problem statement ("must produce both TypeScript and Rust without drift"). No step in the chain does that: typecheck andcargo checkcomp…
- Evidence:
-
Location:
.agents/knowledge/architecture-patterns/spoke-codegen-pipeline.md:31 -
Category: DOCS
-
Merge class: may-fix
🔵 README index summary drops the generated-tree freshness dimension the row previously carried
-
Evidence:
[DOCS-02] README index summary drops the generated-tree freshness dimension the row previously carried
- Evidence:
.agents/knowledge/README.md:9— the codegen row summary changed from "verify-codegenregenerates both trees, asserts the schema count and opaque-JSON types, then runscargo check --workspace" to "verify-codegenchecks the committed trees — schema count, opaque-JSON types, TypeScript typecheck andcargo check --workspace". - Impact: Both halves of the new wording are factually correct against the current script (
package.json:15:assert-schema-count→assert-opaque-json-types→deep-ref-hoisttest →pnpm -F @42ch/spoke-schemas typecheck→cargo check --workspace) and againstpackages/spoke-schemas/package.json:37(typecheck: tsc --noEmitexists). The residual reader cost is only that the index row now reads as a complete verification list while the freshness question (see DOCS-01) is the same one; this is a wording consequence of the same change, not an independent wrong outcome. - Effort: XS
- Risk: LOW — summary text only.
- Confidence: HIGH (opened the row and both scripts).
- Merge class: nit
- Fix sketch: Align the summary row's wording with the corrected posture once DOCS-01 is settled; no behavior change.
- Evidence:
-
Location:
.agents/knowledge/README.md:9 -
Category: DOCS
-
Merge class: nit
🟡 verify-codegen no longer executes the generator, so CI never proves the schema set is still generatable
- Evidence:
- Evidence:
package.json:15— the changedverify-codegenchain is nowassert-schema-count.mjs → assert-opaque-json-types.mjs → node --test deep-ref-hoist.test.mjs → pnpm -F @42ch/spoke-schemas typecheck → cargo check --workspace; the leadingpnpm run codegenwas deleted.tooling/codegen/assert-schema-count.mjs:6-9still describes the author procedure as "Runpnpm run codegen; commit schema + generated output + this constant in one commit.pnpm run verify-codegenmust PASS", so step 4 no longer executes step 3. The two CI consumers run this exact chain:.github/workflows/ci.yml:87-89("Verify codegen output":pnpm run verify-codegen) and.github/workflows/release.yml:163-164.tooling/codegen/assert-opaque-json-types.mjs:14("Wire it frompnpm run verify-codegen") is unaffected because that script readsschemas/, not generated output. - Impact: Before this change the CI job proved two things — the committed trees compile and
schemas/still generates. After it, the generator (tooling/codegen/src/run.mjs→ jstt + typify, plusrust-gen's exact-114 fail-fast) is exercised by no CI job at all. Concretely: a PR that editsschemas/**/*.schema.json, fails to regenerate (the case the new docs warn about), and commits stale-but-compiling generated output now passesverify-codegengreen, whereas the previous chain failed atcodegen/rust-genfor any schema the generators reject. The committed generated trees that ship (and thatdocs/reference/protocol.md:23/docs/zh/reference/protocol.md:23now describe as "checked") can therefore disagree with the hand-authored SSOT with no red check. Note the pre-existing part: drift itself was never asserted (nogit diff --exit-codeafter regeneration), so this finding is limited to the newly removed generatability guard, not to a new drift detector that never existed. - Effort: XS — restore one
pnpm run codegenstep, or run it in a throwaway state and assert a clean tree. - **R…
- Evidence:
- Location:
package.json:15 - Category: dx
- Merge class: may-fix
Revalidated findings
none
Linked-issue AC
none — the PR links no issue (the issues array is empty); only #176 review threads are referenced in prose, which carry no testable acceptance criteria here.
✅ Verified
- Read package.json:15: the changed verify-codegen chain is assert-schema-count.mjs → assert-opaque-json-types.mjs → node --test deep-ref-hoist.test.mjs → pnpm -F @42ch/spoke-schemas typecheck → cargo check --workspace, with the leading pnpm run codegen removed.
- Read .github/workflows/ci.yml:87-89 and .github/workflows/release.yml:163-164: both invoke only pnpm run verify-codegen, with no post-step git diff --exit-code; a grep across .github/workflows/ found no other codegen invocation.
- Read tooling/codegen/assert-schema-count.mjs in full: EXPECTED_SCHEMA_COUNT = 114, counting only schemas/**/*.schema.json and never enumerating packages/spoke-schemas/src/generated or the Rust generated tree.
- Read tooling/codegen/assert-opaque-json-types.mjs:1-20: the checker reads committed generated TS, so it is self-consistent against stale output rather than proof of freshness.
- Read packages/spoke-schemas/package.json:37: a typecheck: tsc --noEmit script exists, so the typecheck leg of the new chain is real.
- Read .agents/knowledge/architecture-patterns/spoke-codegen-pipeline.md:31 and .agents/knowledge/README.md:9 in full, checking each rewritten sentence against the scripts above.
- ❓ The PR-body verification commands (assert-schema-count, assert-opaque-json-types, deep-ref-hoist test, spoke-schemas typecheck, cargo +nightly check --workspace, twin-parity.mjs) are author-claimed only; this seat read statically and did not re-execute them.
🗑️ Considered & rejected
- A "new drift detector" regression: rejected as a new defect — no git diff --exit-code against generated paths ever existed in this chain, so the PR changes generatability coverage, not drift detection; the finding is scoped to the removed codegen step only.
- The README index row (.agents/knowledge/README.md:9) as a substantive wrongness: rejected — both halves of the new wording check out against package.json:15 and packages/spoke-schemas/package.json:37; it survives only as a nit that trails DOCS-01's wording outcome.
- Publishing stale-but-compiling trees from release.yml as an independent finding: rejected — it is the same root cause as the removed generator step and is folded into that one finding rather than counted twice.
- provenance.json hash / committed-carrier coverage concerns (PR threads 1 and 3): rejected — removal of the C carrier hash gate was the stated goal of #176 and the PR resolves those threads by documenting the actual contract; nothing in this seat's file scope contradicts that posture.
- The stale rustfmt rationale at .github/workflows/ci.yml:73 (comment and component install now unjustified): noted but not elevated — it is cross-domain residue explicitly outside this seat's file scope, recorded in the fix sketch for the workflow reviewer.
- Any must-fix or should-fix escalation: rejected — the two may-fix items are documentation and posture accuracy, touching no runtime behavior, and the verdict stands at ship it.
🛠️ Plan to fix
Expand fix plan
1. **DOCS-01 — correct the verification posture in the pipeline doc** (`.agents/knowledge/architecture-patterns/spoke-codegen-pipeline.md:31`, plus the nearby step-4 `rust-gen` "exactly 114 Rust schema modules" clause). Replace "CI verifies the committed output that ships" with wording that states precisely what the chain proves — count, opaque-JSON contract, TS typecheck, Rust compile, all properties of the committed bytes — and that freshness against `schemas/` is author discipline (`pnpm run codegen` + commit), not a CI gate. Make the Verify list and step 4 consistent with that posture so a reader no longer infers a zero-drift proof.
2. **Decide the generatability posture** (`package.json:15`, `.github/workflows/ci.yml:87-89`, `.github/workflows/release.yml:163-164`). Either (a) restore `pnpm run codegen && git diff --exit-code -- packages/spoke-schemas/src/generated crates/spoke-schemas/src/generated`, keeping both workflow consumers on the same chain and accepting the `cargo run --release` cost of `rust-gen`, or (b) keep the current author-only posture and record it explicitly in the codegen knowledge/pipeline doc: no CI job exercises `tooling/codegen/src/run.mjs`, jstt, typify or `rust-gen`'s fail-fast, so a reviewer must run `pnpm run codegen` by hand. Whichever is chosen, update the author-procedure comment at `tooling/codegen/assert-schema-count.mjs:6-9` so step 4 no longer claims to execute step 3.
3. **DOCS-02 — align the index summary last** (`.agents/knowledge/README.md:9`): once DOCS-01's final wording is settled, make the codegen row's summary reflect the same posture (properties verified vs. author-only freshness). No behavior change.
4. **Workflow residue handoff**: flag to the workflow owner that `.github/workflows/ci.yml:73` still installs the `rustfmt` component for a comment claiming `verify-codegen` runs `rustfmt` on generated memory types, which `cargo check` no longer does — outside this seat's scope, but it should land in the same change or be tracked.
5. **Verification gates after the edits**: `node tooling/codegen/assert-schema-count.mjs`, `node tooling/codegen/assert-opaque-json-types.mjs`, `node --test tooling/codegen/test/deep-ref-hoist.test.mjs`, `pnpm -F @42ch/spoke-schemas typecheck`, `cargo +nightly check --workspace`, plus `node tooling/docs/twin-parity.mjs`; if posture (a) is chosen, additionally confirm both workflows run the extended chain and that a deliberately unregenerated schema edit turns red. Re-run the doc twin-parity check after reconciling the `docs/reference/protocol.md:23` / `docs/zh/reference/protocol.md:23` "checked" wording with the chosen posture.Review 1 · commit 9068d63
| 1. **SSOT** — only `schemas/**/*.schema.json` are hand-authored; generated trees are committed. | ||
| 2. **Orchestrator** — `tooling/codegen` (`pnpm -F @42ch/spoke-codegen codegen` → `node tooling/codegen/src/run.mjs`) walks schemas, localizes `$ref`s for typify, and emits mirrored `generated/{common,connect,data,ops}/` trees in both packages. | ||
| 3. **Verify** — `pnpm run verify-codegen` runs `codegen` → `tooling/codegen/assert-schema-count.mjs` → `tooling/codegen/assert-opaque-json-types.mjs` → `node --test tooling/codegen/test/deep-ref-hoist.test.mjs` → `cargo check --workspace`. | ||
| 3. **Verify** — `pnpm run verify-codegen` checks the committed trees: `tooling/codegen/assert-schema-count.mjs` → `tooling/codegen/assert-opaque-json-types.mjs` → `node --test tooling/codegen/test/deep-ref-hoist.test.mjs` → `@42ch/spoke-schemas` typecheck → `cargo check --workspace`. Regeneration is the author step (`pnpm run codegen`); CI verifies the committed output that ships, never a regenerated worktree. |
There was a problem hiding this comment.
Knowledge doc presents CI as verifying committed generated trees, but no chain step compares them to schemas/ · 🟡 may-fix
[DOCS-01] Knowledge doc presents CI as verifying committed generated trees, but no chain step compares them to schemas/
- Evidence:
.agents/knowledge/architecture-patterns/spoke-codegen-pipeline.md:31— Decision step 3 now reads: "pnpm run verify-codegenchecks the committed trees:tooling/codegen/assert-schema-count.mjs→tooling/codegen/assert-opaque-json-types.mjs→node --test …deep-ref-hoist.test.mjs→@42ch/spoke-schemastypecheck →cargo check --workspace. Regeneration is the author step (pnpm run codegen); CI verifies the committed output that ships, never a regenerated worktree." - Evidence:
tooling/codegen/assert-schema-count.mjs:29-31—EXPECTED_SCHEMA_COUNT = 114; the script only countsschemas/**/*.schema.jsonfiles and never enumeratespackages/spoke-schemas/src/generatedor the Rust generated tree, so a schema body edit that skips regeneration leaves the count at 114 and passes. - Evidence:
tooling/codegen/assert-opaque-json-types.mjs:1-20— the checker reads committed generated TS (packages/spoke-schemas/src/generated) and asserts the opaque-JSON shape; it is self-consistent against stale output, not proof of freshness. - Evidence:
.github/workflows/ci.yml:87-89— theverify-codegenjob runs onlypnpm run verify-codegen;.github/workflows/release.yml:163-164does the same before publish. Neither adds a post-stepgit diff --exit-code, and grep across.github/workflows/finds no othercodegeninvocation. - Impact: The doc is the durable tooling fact surface loaded before touching
schemas/. Its "CI verifies the committed output that ships" wording, placed immediately after removal of the regeneration step, tells a reader that a greenverify-codegencertifies the committed trees are current with the hand-authored SSOT declared in the same doc's Problem s…
| | Doc | Category | Source | Summary | | ||
| |-----|----------|--------|---------| | ||
| | [`architecture-patterns/spoke-codegen-pipeline.md`](architecture-patterns/spoke-codegen-pipeline.md) | architecture-patterns | current `schemas/**/*.schema.json`, `tooling/codegen` and generated trees | jstt+typify orchestrator over **114** hand-authored schemas; each generated tree is **114 modules + 5 barrels = 119 files**; `verify-codegen` regenerates both trees, asserts the schema count and opaque-JSON types, then runs `cargo check --workspace`; rust-gen fail-fast; **presence-sensitive arrays** — six byte-bound modules module-wide, plus `Content.cognition` in all 15 carrier modules / 17 clone types (unrelated arrays keep `Vec` + `is_empty`); typify nominal-clone canonical-import guide and Builder construction; **regeneration ripple** — compiling generated bytes does not cover downstream handlers (land ABI + consumer updates together) | | ||
| | [`architecture-patterns/spoke-codegen-pipeline.md`](architecture-patterns/spoke-codegen-pipeline.md) | architecture-patterns | current `schemas/**/*.schema.json`, `tooling/codegen` and generated trees | jstt+typify orchestrator over **114** hand-authored schemas; each generated tree is **114 modules + 5 barrels = 119 files**; `verify-codegen` checks the committed trees — schema count, opaque-JSON types, TypeScript typecheck and `cargo check --workspace`; rust-gen fail-fast; **presence-sensitive arrays** — six byte-bound modules module-wide, plus `Content.cognition` in all 15 carrier modules / 17 clone types (unrelated arrays keep `Vec` + `is_empty`); typify nominal-clone canonical-import guide and Builder construction; **regeneration ripple** — compiling generated bytes does not cover downstream handlers (land ABI + consumer updates together) | |
There was a problem hiding this comment.
README index summary drops the generated-tree freshness dimension the row previously carried · 🔵 nit
[DOCS-02] README index summary drops the generated-tree freshness dimension the row previously carried
- Evidence:
.agents/knowledge/README.md:9— the codegen row summary changed from "verify-codegenregenerates both trees, asserts the schema count and opaque-JSON types, then runscargo check --workspace" to "verify-codegenchecks the committed trees — schema count, opaque-JSON types, TypeScript typecheck andcargo check --workspace". - Impact: Both halves of the new wording are factually correct against the current script (
package.json:15:assert-schema-count→assert-opaque-json-types→deep-ref-hoisttest →pnpm -F @42ch/spoke-schemas typecheck→cargo check --workspace) and againstpackages/spoke-schemas/package.json:37(typecheck: tsc --noEmitexists). The residual reader cost is only that the index row now reads as a complete verification list while the freshness question (see DOCS-01) is the same one; this is a wording consequence of the same change, not an independent wrong outcome. - Effort: XS
- Risk: LOW — summary text only.
- Confidence: HIGH (opened the row and both scripts).
- Merge class: nit
- Fix sketch: Align the summary row's wording with the corrected posture once DOCS-01 is settled; no behavior change.
| "scripts": { | ||
| "codegen": "pnpm -F @42ch/spoke-codegen codegen", | ||
| "verify-codegen": "pnpm run codegen && node tooling/codegen/assert-schema-count.mjs && node tooling/codegen/assert-opaque-json-types.mjs && node --test tooling/codegen/test/deep-ref-hoist.test.mjs && cargo check --workspace", | ||
| "verify-codegen": "node tooling/codegen/assert-schema-count.mjs && node tooling/codegen/assert-opaque-json-types.mjs && node --test tooling/codegen/test/deep-ref-hoist.test.mjs && pnpm -F @42ch/spoke-schemas typecheck && cargo check --workspace", |
There was a problem hiding this comment.
verify-codegen no longer executes the generator, so CI never proves the schema set is still generatable · 🟡 may-fix
- Evidence:
package.json:15— the changedverify-codegenchain is nowassert-schema-count.mjs → assert-opaque-json-types.mjs → node --test deep-ref-hoist.test.mjs → pnpm -F @42ch/spoke-schemas typecheck → cargo check --workspace; the leadingpnpm run codegenwas deleted.tooling/codegen/assert-schema-count.mjs:6-9still describes the author procedure as "Runpnpm run codegen; commit schema + generated output + this constant in one commit.pnpm run verify-codegenmust PASS", so step 4 no longer executes step 3. The two CI consumers run this exact chain:.github/workflows/ci.yml:87-89("Verify codegen output":pnpm run verify-codegen) and.github/workflows/release.yml:163-164.tooling/codegen/assert-opaque-json-types.mjs:14("Wire it frompnpm run verify-codegen") is unaffected because that script readsschemas/, not generated output. - Impact: Before this change the CI job proved two things — the committed trees compile and
schemas/still generates. After it, the generator (tooling/codegen/src/run.mjs→ jstt + typify, plusrust-gen's exact-114 fail-fast) is exercised by no CI job at all. Concretely: a PR that editsschemas/**/*.schema.json, fails to regenerate (the case the new docs warn about), and commits stale-but-compiling generated output now passesverify-codegengreen, whereas the previous chain failed atcodegen/rust-genfor any schema the generators reject. The committed generated trees that ship (and thatdocs/reference/protocol.md:23/docs/zh/reference/protocol.md:23now describe as "checked") can therefore disagree with the hand-authored SSOT with no red check. Note the pre-existing part: drift itself was never asserted (nogit diff --exit-codeafter regeneration), so this finding is limited to the newly removed generatability guard, not to a new drift detector that never exis…
Follow-up to #176 review threads.
Thread 2 (should-fix) — fixed
verify-codegenrancodegenfirst, so the typecheck andcargo checkthat followed compiled freshly rewritten worktree files, not the committed trees that publish as@42ch/spoke-schemas/spoke-schemas. A schema edit without a committed regeneration passed.It now verifies the committed trees: schema-count and opaque-JSON asserts, the deep-ref-hoist test,
@42ch/spoke-schemastypecheck, thencargo check --workspace. Regeneration is the author step (pnpm run codegen); no byte diff returns.Threads 1 and 3 (should-fix) — resolved by documenting the chosen coverage
Removing the committed C carrier hash gate was the stated goal of #176: CI must not fail on compiled-byte identity. Those threads are addressed by making the docs state the actual contract instead of demanding a check that no longer exists:
provenance.jsonhashes are build audit data, not a CI gate.CONTRIBUTING.mdno longer says the recorded hashes must match committed bytes.Threads 4, 5, 6 (may-fix) — fixed
main; this PR keeps refresh text on the transfer manifest).Verification
node tooling/codegen/assert-schema-count.mjs— OKnode tooling/codegen/assert-opaque-json-types.mjs— OKnode --test tooling/codegen/test/deep-ref-hoist.test.mjs— passpnpm -F @42ch/spoke-schemas typecheck— passcargo +nightly check --workspace— passnode tooling/docs/twin-parity.mjs— OK