feat(control-plane): move post-writeback hook transactions to TypeScript - #3847
Conversation
huangruiteng
left a comment
There was a problem hiding this comment.
Findings:
- [P1] Python compatibility locks still pay the full timeout once per hook, so a contended batch can stall primary writeback for
N × timeout.dispatch_post_writeback_hookskeeps all acquired guards alive in oneExitStack(loopx/control_plane/capability_hooks.py:691-715), but_enter_legacy_post_writeback_guardpasses the unchangedlease_timeout_secondsto every successiveexclusive_file_lock(capability_hooks.py:333-378). The new TypeScript sidecar locks correctly share one batch-wide deadline, but that deadline starts only during finalize and does not bound this earlier Python loop. A focused reproduction holding three legacy receipt locks and calling the dispatcher withlease_timeout_seconds=0.05took about0.195s, returned three timeouts, and invoked no providers; with the default mutation timeout this can cost roughly five seconds per contended hook, up to the admitted 128-hook batch. This contradicts the PR's bounded-crossing/one-deadline goal and can turn an optional post-writeback observer into a long synchronous delay after the primary mutation. Compute one monotonic compatibility deadline before the loop and pass each guard only the remaining budget (or acquire the ordered compatibility set under one bounded helper), then add a multi-locked-hook regression that asserts total elapsed time is bounded near one timeout while free siblings still execute.
Exact head reviewed: 1e5caadc1299527397d940eb8a80c66651e23663.
动机
这个 PR 要解决 post-writeback capability hook 事务由 Python 分散拥有的问题。旧路径对 fresh batch 中每个 hook 分别做 registration/input/result/receipt runtime crossing、journal lock 与 sidecar 写入;随着 provider 增加,会重复跨语言调用,也让 replay、retry、CAS、顺序和 durable receipt 的权威散落在 Python。新设计希望把这些决策统一迁到 TypeScript,同时保留 mixed-version 期间的 Python provider 与旧锁兼容。
改动思路
CLI 在主写回完成后构造 compact source,Python dispatcher 先把整批 registration 送入 capability_hook.post_writeback.transaction 的 preflight。TypeScript 严格解码 source、registration 与 hook input,计算稳定 transaction/dispatch identity,读取 sidecar 并只返回仍需执行的 provider plan。Python 对这些 plan 获取兼容锁、调用 effect-free producer、把 JSON-safe outcome 送回 finalize;TypeScript 再次读取权威状态、做 retry/CAS/duplicate-key 仲裁、原子写 receipt,最后按 canonical hook order 汇总 dispatch。这个 owner 切分是合理的:provider 执行不获得写权限,receipt 与 replay 由 typed runtime 统一拥有。
正向路径:一个 fresh durable writeback 进入 preflight,两个合法 hook 得到各自最小 read scope;provider 返回两个 receipt-bound intent;finalize 在一个 typed transaction 中重验结果、写 sidecar 并返回 canonical intents。相同 source 重放时,preflight 直接读取 terminal receipts,不再调用 provider。
负向路径:若某个 receipt 被并发写入,snapshot/legacy guard 与 TypeScript CAS 会把它归类为 replay 或 conflict,不覆盖赢家;provider failure、malformed receipt、transport overflow 和 retry exhaustion 也按 hook 隔离。当前遗漏的是多个 legacy locks 同时 contention 时没有共享 deadline,导致“隔离”仍可能在线性等待之后才返回。
具体改动
loopx/cli_commands/post_writeback.py不再在 Python 构造最终 hook input/journal,而是把 committed source facts 与 runtime root 交给 batch dispatcher。loopx/control_plane/capability_hooks.py删除 Python journal/receipt authority,新增 registration/outcome transport budgeting、preflight/finalize adapter、provider invocation 与 rolling-upgrade legacy guard;Python 文件从 982 行缩到 759 行。loopx/control_plane/post_writeback_hook_transaction.ts新增完整事务 owner:exact-field decoding、Python-compatible canonical identity、registration admission、receipt inspection、provider plan、retry/replay/conflict、duplicate intent arbitration、batch envelope admission、原子 sidecar 与 canonical dispatch。effect_runtime_handlers.ts用单个 transaction method 替代四个 Python 逐 hook validator method;effect_runtime_io.ts让 mutation lock 接收 caller-supplied timeout,从而支持 TypeScript finalize 的一秒 batch deadline。- 两组测试覆盖 identity parity、read-scope isolation、transport size、fresh/replay crossing、retry、rolling upgrade、并发 single-flight/CAS、malformed receipt、duplicate intent 与 envelope preflight;
tsconfig.control-plane.json把新 owner 和测试纳入 strict typecheck。
关键代码讲解
evaluatePostWritebackHookTransaction是新权威入口:preflight 只授权 provider plan,finalize 必须绑定同一 transaction identity,并拒绝缺失或越界 outcome。inspectTransaction按 hook id 建立 canonical result slots,验证 registration/input,读取 terminal/retryable receipt,并保证 replay、fresh 与 failure 最终仍按稳定顺序聚合。storeReceipt在 TypeScript mutation lock 内重读 sidecar,区分 exact replay、terminal conflict 与合法 retry,再原子写入并同步父目录。dispatch_post_writeback_hooks是兼容 facade:它只执行 admitted provider,但当前在ExitStack中逐个使用完整 legacy timeout,这是本 review 的 blocker。_bounded_post_writeback_transport_outcomes先把 returned result 降为 contract-rejected baseline,再按实际 JSON byte delta 装回可容纳结果,避免一个大 provider 撑破共享 RPC envelope。
对主干的风险
这是默认 control-plane handler 与 committed CLI post-writeback 热路径的迁移,8 files、+4244/-732;主要风险是 replay/CAS identity 漂移、跨版本双写、可选 hook 拖慢主命令、以及巨大 transaction owner 的后续维护成本。范围虽大,但代码集中在一个已接受的迁移 seam,旧 Python authority 有实质删除,测试也覆盖了多数并发负路径;typed state、domain-neutral naming、authority semantics 和 feature-off isolation没有发现额外 blocker。未启用 periodic-report hooks 时 dispatcher 在进入 runtime 前返回,shared effect_runtime_io 的默认 timeout 也保持不变。
当前最关键的反例恰好击中 PR 自己声明的 batch-wide timeout 不变量:TypeScript 只约束 finalize locks,Python rolling-upgrade locks 仍能累计。最小修复不需要重做架构,只需让兼容层复用一个 monotonic deadline并补回归测试。
验证矩阵:
| 范围 | 结果 |
|---|---|
| focused TypeScript transaction tests | 27 passed |
| focused Python post-writeback tests | 31 passed |
| exact-head GitHub control-plane/typecheck | 421 passed, 1 skipped;strict typecheck passed |
| exact-head GitHub Python suite | 5,143 passed, 12 skipped |
| local full TypeScript suite | 421 passed;PostgreSQL integration 因本地缺少 pg 包无法启动;exact-head CI 对应 suite 已通过 |
| Ruff / compile / git diff check | passed |
| public/private boundary scan | passed for all 8 changed files |
| multi-hook compatibility-lock budget reproduction | failed:3 × 0.05s contention 实测约 0.195s |
| risk-based premerge automated checks | 16/16 passed,无 manual hold;最终 gate 因 exact-scope quality receipt invalid 而未通过 |
| SonarCloud Code Analysis | failed;quality gate 为 ERROR / reliability D,公开 issue 主要是 complexity 与 analyzer ordering findings |
| change-quality | fingerprint 4113cb940571f88f70de3527df5a5278101bc2b1e066a9d451a8cebfd29d06a3;receipt cqr_4113cb940571f88f70de fail/invalid |
我的整体评价
把 post-writeback 的 admission、identity、CAS、durable receipt 与 canonical aggregation 收到 TypeScript 是正确的架构方向,也比继续扩张 Python journal owner 更容易形成单一事实源。PR 对 replay、retry、冲突和 transport 的测试非常扎实,future-facing refactor 也确实删除了旧 authority;但兼容锁仍按 hook 累计完整 timeout,直接破坏本次迁移最重要的 bounded batch 保证。这个修复范围很小且应在当前 PR 内完成。结论为 REQUEST_CHANGES / hold;修复共享 deadline、补多锁 contention 测试并让 exact-head gates 回绿后可快速复审。不执行 merge。
English verdict: REQUEST_CHANGES — exact head 1e5caadc1299527397d940eb8a80c66651e23663 correctly moves admission, identity, replay/CAS, durable receipts, and canonical aggregation into the TypeScript owner, with strong focused and exact-head test coverage. However, the Python rolling-upgrade guards still spend the full lock timeout once per hook; three contended hooks at a 0.05s timeout took about 0.195s, so a batch can synchronously delay primary writeback by N × timeout. Use one monotonic compatibility deadline across the batch and add a multi-lock regression. SonarCloud also remains failed. No merge performed.
1e5caad to
4e39fd2
Compare
4e39fd2 to
3fc1c4b
Compare
huangruiteng
left a comment
There was a problem hiding this comment.
Finding
- [P1] Python compatibility locks still consume the full timeout once per hook, so contention remains
N × timeout. At exact head3fc1c4b23064b41490fb77b3ad13d8f367caaa30,dispatch_post_writeback_hooksiterates every provider plan atloopx/control_plane/capability_hooks.py:692-716, and each call to_enter_legacy_post_writeback_guardreceives the same unchangedlease_timeout_secondsat lines 701-706. The helper then passes that full value toexclusive_file_lockat lines 350-355. Holding three admitted legacy receipt locks and dispatching withlease_timeout_seconds=0.05took0.191s, returned threelock_acquire_timeoutfailures, and invoked no provider. This is the same blocker as the previous review: the TypeScript finalize deadline does not bound the earlier Python rolling-upgrade loop. Compute one monotonic compatibility deadline before iterating, pass only the remaining budget to each guard (while preserving free-sibling execution), and add a three-locked-hook regression whose total elapsed time is bounded near one timeout.
Exact head reviewed: 3fc1c4b23064b41490fb77b3ad13d8f367caaa30.
动机
这个 PR 试图消除 post-writeback hook 事务在 Python 与 TypeScript 之间的重复权威。旧实现按 hook 分别完成输入校验、journal/receipt 状态判断、重放与冲突处理,fresh batch 的 runtime crossing 会随 hook 数量增长,也让稳定 identity、CAS、retry 和 durable receipt 的所有权分散。当前改动把这些决策集中到 TypeScript transaction owner,Python 只保留 provider 调用和 mixed-version 兼容锁。受影响的直接调用者是 committed writeback 后的 CLI/capability hook 路径;预期可观察结果是 fresh batch 固定两次 runtime crossing、exact replay 一次 crossing,同时保持主写回不被可选 provider 失败破坏。更小的局部优化无法消除双重 authority,因此迁移方向成立;本 PR 不包含 provider 原生 TS 化,也不改变 benchmark、权限或外部写入策略。
改动思路
正向路径是:loopx/cli_commands/post_writeback.py 从已提交 writeback 构造 compact source,dispatch_post_writeback_hooks 把整批 registration 送到 capability_hook.post_writeback.transaction preflight;TypeScript 以 source、registration 和 canonical order 计算 transaction/dispatch identity,读取 sidecar,只返回尚需执行的 provider plan。Python 获取兼容 guard 并调用 effect-free producer,再把 bounded outcomes 送回 finalize;TypeScript 重新检查权威状态,完成 retry/CAS/duplicate-key 仲裁、原子 receipt 写入和 canonical dispatch 汇总。exact replay 则在 preflight 直接返回 terminal dispatch,不再调用 provider。
负向路径覆盖 malformed registration/result、transport envelope 超限、retry exhaustion、并发 receipt winner 和 sidecar write failure:这些都应隔离到单个 hook,同时保留 primary writeback。当前缺口发生在 finalize 之前:多个 legacy receipt lock 同时被占用时,Python 逐个等待完整 timeout,只有全部等待结束后才进入 finalize;因此“一个 batch-wide deadline”并未覆盖完整事务。
具体改动
loopx/cli_commands/post_writeback.py将 committed source 与 runtime root 交给 batch dispatcher,不再由 Python 创建最终 journal transaction。loopx/control_plane/capability_hooks.py删除大部分 Python receipt authority,新增 registration/outcome envelope budgeting、preflight/finalize adapter、provider lookup 与 rolling-upgrade legacy guard。loopx/control_plane/post_writeback_hook_transaction.ts新增 typed transaction owner,负责 exact-field decode、Python-compatible identity、registration admission、receipt inspection、retry/replay/conflict、duplicate intent arbitration、原子 sidecar 和稳定输出顺序。effect_runtime_handlers.ts用单个capability_hook.post_writeback.transactionmethod 取代多个逐 hook validator;effect_runtime_io.ts允许 mutation lock 接收 caller timeout。- Python 与 TypeScript 测试覆盖 fresh/replay、identity parity、read-scope、transport、retry、CAS、malformed receipt、duplicate intent、envelope admission 和 rolling-upgrade compatibility;
tsconfig.control-plane.json纳入新 owner 与测试。
关键代码讲解
evaluatePostWritebackHookTransaction(post_writeback_hook_transaction.ts:1319)是统一入口:preflight 只签发 provider plan,finalize 必须绑定同一 transaction identity 和完整 outcome 集合。inspectTransaction(:736)读取并分类 terminal、retryable、fresh 与 conflict receipts,建立 canonical result slots;它是 replay/retry 判断的权威读路径。storeReceipt(:910)在 TypeScript mutation lock 内重读 sidecar,区分 exact replay、CAS winner 和合法 retry,再以原子写落盘。dispatch_post_writeback_hooks(capability_hooks.py:600)是 Python compatibility facade:它只执行 preflight admitted provider,但 lines 692-716 仍逐 plan 传入完整 lock timeout,导致本次 blocker。_bounded_post_writeback_transport_outcomes(:418)从 contract-rejected baseline 开始按 JSON byte delta 装回 provider 结果,避免单个大结果拖垮共享 RPC envelope。
对主干的风险
这是默认 control-plane post-writeback 热路径迁移,8 files、+4287/-732;主要风险是跨语言 identity 漂移、rolling-upgrade 双写、CAS winner 被覆盖、可选 hook 拖慢主命令,以及 1,421 行 transaction owner 的维护成本。整体范围虽然大,但围绕一个已接受的迁移 seam,并实质删除了 Python authority;active production caller 已从 CLI committed writeback 接入,scope-fit 与 authority naming 均成立。它不是 default-off 功能,因而不存在 feature-off parity claim;typed state、domain-neutral wording和 guidance/obligation 语义未发现新的 blocker。PR body 也披露了 fresh-path timing 变慢与 compatibility lock 的退役条件。
验证结果:focused Python hook suite 31 passed;focused TypeScript transaction suite 27 passed;strict TypeScript typecheck passed;git diff --check passed。当前 GitHub pytest check 为 cancelled,其他已返回的 DCO、release build、Windows、dependency review 与 SonarCloud checks 成功。最强负向反例仍失败:三把 legacy locks、每把 0.05s timeout 的 exact-head reproduction 实测 0.191s,而不是接近一个共享 0.05s budget;仓库内也仍只有单 locked-hook isolation coverage,没有 multi-locked batch deadline regression。
最小修复是仅调整 Python compatibility facade:在进入 provider-plan loop 前建立 monotonic deadline,每次 lock acquisition 计算 remaining budget,并补 multi-lock test;不需要扩大 TypeScript transaction API。Future-facing pass 已通过集中 typed authority 与删除旧 Python journal 实现,除这个 bounded companion fix 外,不建议再扩张本 PR。
我的整体评价
把 admission、identity、replay/CAS、durable receipt 和 canonical aggregation 收到 TypeScript 是正确且与仓库迁移 RFC 一致的方向,现有测试对 transaction owner 的大多数正负路径也很扎实。代码体量对于这次单一 authority migration 虽高但尚属可解释,最高价值的简化已经是删除 Python journal owner,而不是继续拆出抽象。
不过,上一轮明确要求修复的 batch-wide compatibility timeout 在当前 exact head 仍未实现,且独立复现数据几乎与上一轮一致。这会让可选 observer 在多锁争用时线性拖慢主写回,是默认热路径上的 P1 blocker。结论:REQUEST_CHANGES。完成共享 deadline、补多锁回归,并让 exact-head CI 重新完整结束后再复审;本次不执行 merge。
English verdict: REQUEST_CHANGES — exact head 3fc1c4b23064b41490fb77b3ad13d8f367caaa30 preserves the intended TypeScript ownership migration and passes the focused Python (31) and TypeScript (27) suites plus strict typecheck, but the prior blocker remains. The Python rolling-upgrade loop still gives every legacy lock the full timeout; three contended hooks at 0.05s took 0.191s, so primary writeback can still stall for N × timeout. Use one monotonic compatibility deadline, add a multi-lock regression, and rerun the currently cancelled GitHub pytest check. No merge performed.
|
This is a careful migration — we independently recomputed the golden receipt digests against the retired Python implementation (upstream/main semantics: Three points while reading:
Two smaller notes: the Python top-level exception funnel in Overall: net positive from our side as the authors of the Python journal this replaces (#3691) — the transaction shape is a strict upgrade. The questions above are about making the load-bearing invariants legible, not about correctness we could break. |
Signed-off-by: huangruiteng <[email protected]>
huangruiteng
left a comment
There was a problem hiding this comment.
Re-reviewed exact head f6f7d6a83d145c74cd25fe33e942c75acfd771d3 after the fork update.
The prior P1 blocker is resolved: Python now establishes one monotonic compatibility deadline for the full admitted batch and passes only the remaining budget to each legacy lock. The new three-contended-lock regression bounds total delay near one timeout while confirming that an uncontended sibling can still execute. The branch also incorporates current main; the test conflict was resolved by retaining both the new transaction API coverage and the newer periodic-report coverage.
Validation on the exact head:
- focused Python hook suite: 40 passed
- focused TypeScript transaction suite: 27 passed
- full TypeScript control-plane suite: 546 passed, 1 environment-gated PostgreSQL integration skipped
- strict TypeScript typecheck: passed
- strict mypy (19 files), Ruff,
py_compile, andgit diff --check: passed loopx canary premerge --from-git-diff --goal-id loopx-meta: passed, 16/16 selected checks, 0 holds- change-quality receipt
cqr_64ef1f01fae97231403b: valid for the exact base and diff
The refinement stays within the existing architecture: TypeScript remains the single transaction/receipt authority, Python remains only the provider and rolling-upgrade compatibility layer, and no protocol fields or new abstractions were added. APPROVED pending required GitHub checks.
Summary
capability_hook.post_writeback.transaction.The fresh path now crosses the managed runtime twice for a whole batch (preflight and finalize), while an exact replay crosses it once. The previous path crossed four times per fresh hook and once per replayed hook.
Issue Or Task
Design And Compatibility
Future-facing pass: this PR also moves the lock-timeout option into the existing runtime I/O owner because the transaction needs one batch-wide deadline. A native provider/caller migration is deliberately deferred until those real call sites are TypeScript-owned.
Performance Evidence
Local five-trial characterization with eight hooks:
origin/mainruntime crossingsThe fresh-path timing cost is disclosed: the managed batch transaction performs durable receipt/fsync work at the TypeScript boundary and is slower in this local microbenchmark. The payoff is removal of duplicate Python authority, bounded crossings independent of hook count, and roughly 8.9x faster exact replay.
capability_hooks.pyshrinks from 982 to 759 lines.Validation
twine check; fresh-environment probes from both artifacts.loopx canary premerge --from-git-diffwith the managed-goal policy: 16/16 risk-based checks passed, no warnings or manual holds.git diff --check, exact-path secret/private-state scan, clean worktree, one DCO-signed commit, and merge base equal to latestorigin/main.loopx check --scan-path ...: all 8 PR paths are public-boundary clean; the aggregate command remains non-zero because an unrelated registered project has a pre-existing malformed user gate.No benchmark jobs were launched. This does not change benchmark scoring, task semantics, permissions, external delivery, or public evidence policy.
Type of Change
LoopX Area
Technical Direction
Core control-plane hardening
Long-horizon benchmark evidence
Operator surface and IM integration
Shared Goal Authority and cross-host coordination
Architecture and research incubator
Target base branch:
mainDirection tracker or promotion unit: TypeScript control-plane migration RFC
Boundary Checklist
.loopx/,.codex/goals/, liveACTIVE_GOAL_STATE.md, credentials, private benchmark traces, verifier output, raw agent sessions, internal document links, or local machine paths.Signed-off-bytrailer (git commit -s).