Repository navigation
refactor(turn): centralize native host failure policy - #9146
Conversation
|
This pull request has merge conflicts with Choose the remote for the base repository, not an out-of-date fork. git fetch upstream
git rebase upstream/main
# Resolve each conflict, git add the resolved files, then git rebase --continue.
git push --force-with-lease origin HEADFor a same-repository clone whose Keep the DCO |
Signed-off-by: lusendong.6789 <[email protected]> Co-authored-by: TRAE CLI <[email protected]>
03a1d23 to
94ba859
Compare
cocolord
left a comment
There was a problem hiding this comment.
Reviewer: model_agent; GPT-6; OpenAI; self_reported
Approval conclusion (author-owned PR; GitHub blocks formal self-approval)
APPROVE — exact head 94ba859f4f8dc698c4ef6493d5a20d05fe572d2a。未发现本切片的阻塞缺陷;整体 RFC、原生采用和合入门仍开放。
动机
原生 Host 失败恢复及外部 scheduler worker 的调用方。 Host 返回容量失败或另一个会话的结果时,原来由 Python 表和 worker 分支再次解释重试资格;现在复用既有 TypeScript failure owner,仍将不明或外国会话结果保留为 unknown。 退役重复的重试分配表和 worker 成功/失败选择,保留原会话、有限重试和一次结算。 本 PR 不新增 executor、持久化 owner 或自动启用;Host 成功不等于 Turn 完成、ACK 或扣额许可。 剩余 C1/C4/C9 切换、已安装 worker 采用、整体产品组合和成果/完整成本资格仍开放。
改动思路
重试预算和原生阶段观测共用既有 failure owner;native 错误创建、整数/Mapping 编码、会话 ID、pending witness 与时钟仍归原 IO 适配器。 当前 PR 完成 failure allocation 和 worker 观测迁移,原恢复与结算边界保持;安装态及整体收益继续沿原验收。 正路径是 Capacity failure from the same configured session records original attempt and backoff, retries within three attempts, then original recovery settles once. 负路径是 Foreign-session success/failure, malformed receipt or transport loss cannot clear the original session; capacity exhaustion and output-budget exhaustion cannot auto-retry. 原事实与 effect owner 继续维护身份、权限和结算,新的投影不构成执行授权。做更小的 wrapper 而保留正常规则会继续产生两个需同步修改的位置;另建执行 owner 则没有证据支持。此处复用既有 owner 并保留必要 IO,是可独立回滚的边界。
具体改动
10 个文件,341 行增加、67 行删除。生产变化在既有 failure owner、Python codec、外部 worker 和两个内部 dispatch;测试覆盖整数、Mapping、失败资格、外国会话及真实 worker,tsconfig 加入既有类型检查;双语 RFC 标明源码边界。
规格依据:docs/architecture/rfcs/mechanism-convergence-and-learning-loop-v0.md,预变更规范 revision 2d302ca189cf3798ca4d53b7e01d37353ffc19d6。criterion 5.5:当前规则/真实 caller 迁移 implemented;8:原身份、receipt 与 pending recovery implemented;9:本切片的合法/拒绝/恢复与一次结算控制 implemented;10:整体采用及成果/完整成本要求 deferred,沿原 C1/C4/C9 与安装/效果验收继续。checkpoint 编辑不反向定义这些 oracle。
关键代码讲解
hostFailureRecord 统一 ceiling/backoff;hostFailureRetryRequest 的 automatic_only 仅缩窄自动资格,省略/false 保留显式修复语义。hostFailureObservationRequest 检查 transport、exit、ok 和原 session;失败记录仍由 journal/worker 维护原 attempt。_host_result 保留 native ID 解码、有限 receipt 和 not_before,provider 私有字段不会成为规则输入。
实际调用链:Native adapters -> build_host_failure_record; external_scheduler_worker.run -> _host_result; journal retry -> hostFailureRetryRequest。保留的兼容边界:Native kind validation remains independent so BuiltInHostError construction needs no RPC; Mapping error order, exact decimal integer codec and session-ID decoder remain IO obligations. 退役边界:Python retry table and worker outcome selection removed; distinct explicit-repair and automatic eligibility reuse the same typed allocation.
对主干的风险
最强反例是另外会话的成功结果被当作原工作完成,或者同一容量失败超出原重试预算。两个反例都有真实 worker/SQLite CLI 的负控制与已执行 mutation;当前边界检查保留 malformed/unknown 和大整数拒绝顺序。失败路径增加本地 RPC;它不可替代 native 错误创建,也没有通过新 RPC 授予执行权。 回滚应撤回本切片而保留原 journal/receipt 和恢复身份,不能通过新 Turn 或补扣额掩盖旧未完成工作。新增代码没有不同的 peer/权限模型,状态判定复用 typed route、receipt 和 schema;不依赖任务文本判断状态。能力安装与 RPC 可用均不触发新激活。
PASS:156 current Python boundary/worker/managed CLI cases; 20 TS cases; typecheck; four immutable main native File/SQLite managed CLI controls passed; retained same-fixture base/head failure controls and deliberate mutations inspected. 原失败和 mutation:Capacity ceiling 3->4 fails real SQLite CLI; ignoring session drift fails actual external worker foreign-capacity case. Both sources restored; current worker and managed CLI pass. CI 当前仍有未完成或失败的诊断项,原因未全部核实;独立检查覆盖本次受影响 invariant,CI 颜色不能代替这些证据,也不能被改叫全绿。
语义与 CI 对齐
复用已有共享词汇,局部类型表达原值,未新增持久化 schema 或独立状态 owner。公开/native codec 与拒绝顺序保留;源规范、双语 staged/exact-head gates 和已有风险 premerge 证据已核对。原生隔离控制在 immutable main 亦于同一入口失败,组合原生采用仍 UNVERIFIED;这是整体资格缺口,未被局部 CLI pass 覆盖。PG 普通 Host Turn、已安装切换、真实模型采用与成果/完整成本不由本切片测试证明。
我的整体评价
本 PR 是 justified_increment。当前 PR 完成 failure allocation 和 worker 观测迁移,原恢复与结算边界保持;安装态及整体收益继续沿原验收。 长程义务 preserved,用户入口 preserved,原会话和恢复体验未增加新控制。无渲染 App/Lark 页面变化,visual_review not_applicable。保持历史 codec/receipt 的代价有实际消费者依据;当前已减去正常重复规则,不为减少行数删除必要 IO。剩余 C1/C4/C9 切换、已安装 worker 采用、整体产品组合和成果/完整成本资格仍开放。 没有阻塞代码 finding;最强仍缺验证是安装态与真实模型/独立成果采用。审批结论只针对此 unchanged head,不授予 merge 或清除 CI/整体验收 gate。
English verdict: APPROVE - 94ba859; existing owner reused, scoped real File/SQLite positive/refusal/recovery checks pass; installed/native adoption and full outcome/cost acceptance remain open, and CI/merge holds remain separate.
loopx-agent
left a comment
There was a problem hiding this comment.
Reviewer: model_agent · Antigravity · Google Deepmind · self_reported
动机 / Motivation
原生 Host 失败恢复调用方、external_scheduler_worker 外部调度守护进程以及 Turn journal 恢复路径。 此前,Python 侧 host_failure.py 独立维护一套 _RETRY_POLICIES 重试表与退避计算,scripts/external_scheduler_worker.py 的 _host_result 也在 Python 内自行判定原生阶段成功与同会话失败类型;一旦调整重试上限或跨会话拒绝规则,Python 与 TypeScript 两侧会各自漂移。改动后,两类判断统一收敛至既有 TypeScript owner(host_failure.ts),Python 适配器仅保留本地错误构造、精确整数/Mapping 编解码与状态持久化。
重试预算分配、自动/显式修复资格区分及外部 worker 的 Host 阶段观测全部由 host_failure.ts 裁决;现有重试上限、指数退避上限、原会话绑定、未知/跨会话失败降级为 unknown 及一次结算契约均保持等价。 本 PR 不新增 executor、持久化状态 store、全局权限规则或自动激活开关;Host 成功观测仅确认进程与回执传输,不授予 Turn 完成、scheduler ACK 或配额扣减权限。
与主线收敛范围相比,整体机制收敛 RFC 中的 C1/C4/C9 其余路径切换、已安装 worker 采用、完整产品组合以及真实模型同预算成果/全成本验收仍保持开放。 若不做本次收口,任何对退避或重试上限的策略调整仍需同步修改 Python 与 TypeScript 两份实现,且 worker 的成功/跨会话观测语义无法在控制面单一边界下被独立约束。
改动思路 / Approach
将重试预算表与原生 Host 阶段观测收归既有 TypeScript 策略 owner(host_failure.ts),把易变策略规则与不可变的本地 IO 适配边界(BuiltInHostError 无 RPC 构造、Decimal 大整数编解码、Mapping 校验优先序与 worker 状态持久化)明确拆离。 当前 PR 边界仅覆盖原生失败记录分配、自动/显式重试资格区分及外部 worker 阶段观测这组内聚机制,不引入新状态 owner 或运行模式开关,可按单一切片独立审阅与回滚。
| 环节 | 改动前(1a3ab75a53) |
改动后(94ba859f4f) |
|---|---|---|
| 重试表与记录分配 | Python host_failure.py 自持 _RETRY_POLICIES |
经 host_failure.record RPC 调用 TS hostFailureRecord |
| 自动重试资格 | Python 本地比较 attempt < max_attempts |
经 host_failure.retry 传入 automatic_only: true |
| 外部 worker 阶段观测 | _host_result 在 Python 内分支判断成功与同会话 |
剥离私有字段后交由 host_failure.observe 裁决 |
| 本地出错构造与 IO 适配 | BuiltInHostError 本地校验 HOST_FAILURE_KINDS |
保持本地无 RPC 错误构造,Decimal 精确传输 attempt_text |
上述分工(基于源码检验,非运行实测替代)保证了纯类型规则归 TS、进程/串口/Mapping 适配归 Python;成功观测返回 failure: null 仅表达传输确认,Turn 结算与配额消费仍受原交易日志与控制器门禁约束。
具体改动 / Changes
本次共修改 10 个文件(+341 / -67),包括核心控制面与运行脚本 4 个、双语 RFC 源码 checkpoint 2 个、新增与扩展测试 3 个以及 tsconfig.control-plane.json 1 处类型检查纳管:
- 生产运行时(4 个文件):
loopx/control_plane/turn_driver/host_failure.ts、loopx/control_plane/turn_driver/host_failure.py、loopx/control_plane/effect_runtime_handlers.ts、scripts/external_scheduler_worker.py。 - 文档与规范检查点(2 个文件):
docs/architecture/rfcs/mechanism-convergence-and-learning-loop-v0.md与.zh-CN.md同步追加本切片源码边界说明(check_rfc_bilingual.py通过)。 - 测试与构建配置(4 个文件):新增
tests/control_plane/test_host_failure_policy_transport.py与tests/control_plane_ts/host_failure_policy.test.ts,扩展tests/test_external_scheduler_worker.py跨会话反例,并将新 TS 测试加入tsconfig.control-plane.json。
规格映射依据:docs/architecture/rfcs/mechanism-convergence-and-learning-loop-v0.md(变更前固定基线 1a3ab75a53107b29ee1b8bff56e2b8263259dec6)及关联控制器统一契约:
- 准则
5.5(单一普通策略 owner、迁出重复重试表并保留内核不变量与反例测试):implemented ——host_failure.py已删除_RETRY_POLICIES与HOST_RETRY_STRATEGY,external_scheduler_worker.py已移除本地结果分支,且 Python/TS 边界与 CLI 测试全部通过。 - 准则
9(隔离 File/SQLite 真实入口下验证原会话重试、拒绝跨会话/未知结果与一次结算):implemented ——test_external_scheduler_worker.py与test_loopx_turn_managed_step.py在真实入口验证上述不变量。 - 准则
C4.1与C4.2(采用 typed Turn 观测/续跑边界并收敛外部 worker 重复策略,保留唤醒与不确定性语义):implemented(切片范围内)——host_failure.observe与host_failure.retry承接决策,worker 保留进程唤醒与 pending witness。 - 准则
10(全量 C1/C4/C9 切换、已安装 Host 采用及同预算成果/全成本验收):deferred —— 父级 RFC 仍开放,需后续真实模型与安装态实验单独验证。
关键代码讲解
hostFailureRecord与hostFailureRecordRequest(loopx/control_plane/turn_driver/host_failure.ts):统一使用BigInt与hostRetryPolicies计算retryable、max_attempts及上限 300 秒的指数退避;对超过Number.MAX_SAFE_INTEGER的尝试次数在 TS 侧序列化为十进制字符串,避免 JSON 精度丢失。hostFailureObservationRequest(loopx/control_plane/turn_driver/host_failure.ts):校验transport_ok、timed_out、error_kind、exit_zero、ok、has_host_failure与prior === null || prior === session,仅在合法同会话且native_failure.attempt === 1时提取原生失败类型,其余异常、超限、跨会话或损坏元数据一律降级为unknown,避免伪造或外泄私有诊断。hostFailureRetryRequest(loopx/control_plane/turn_driver/host_failure.ts):新增可选布尔参数automatic_only。省略或为false时完全保留原有显式修复判定(如允许auth_failed人工修复重试,仅阻止output_budget_exhausted);设为true时收窄为必须retryable === true且未达尝试上限。build_host_failure_record与host_failure_retry_available(loopx/control_plane/turn_driver/host_failure.py):前者改用Decimal格式化attempt_text经host_failure.recordRPC 生成标准记录后回填原始 Pythonint;后者与explicit_host_failure_retry_available共同复用host_failure.retryRPC,彻底退役 Python 本地退避与次数上限逻辑,同时保留HOST_FAILURE_KINDS集合使BuiltInHostError构造不依赖外部运行时。_host_result(scripts/external_scheduler_worker.py):保留进程退出码、JSON 解析、agent作用域会话 ID 校验与本地normalize_host_failure_record清洗(确保private_diagnostic等私有字段不进入 RPC),再通过host_failure.observe获取最终失败记录与归属会话。
对主干的风险 / Risks
验证执行矩阵
- changed_invariant_positive(PASS):
npm run typecheck:control-plane(exit 0);TS 测试host_failure_policy.test.ts+effect_runtime_handlers.test.ts+turn_loop_controller.test.ts+turn_journal.test.ts共 81/81 通过;Python 测试test_host_failure_policy_transport.py、test_loopx_turn_host_failure.py、test_external_scheduler_worker.py、test_turn_failure_readback.py共 163/163 通过;test_loopx_turn_managed_step.py、test_loop_turn_loop_controller.py、test_loop_turn_controller_contract.py、test_external_scheduler_replan_retry.py共 349/349 通过;test_dsh_goal_mode.py+test_rfc_bilingual.py共 412/412 通过;test_loopx_turn_executor.py -k "host_failure or retry"6/6 通过。 - material_negative_or_failure(PASS):实测基线
1a3ab75a53与当前头94ba859f4f对照:44 组build/retry组合、7 组normalize_host_failure_record拒绝优先序反例、7 组_host_result成功/跨会话/损坏/超时对照结果完全一致(0 差异);跨会话成功与跨会话 capacity 伪造回执在真实 worker 入口下均保持原会话并降级为unknown。 - repository_required_checks(PASS):
ruff check通过;mypy --follow-imports=silent loopx/control_plane/turn_driver/host_failure.py无错误;scripts/check_rfc_bilingual.py通过;examples/semantic-vocabulary-drift-smoke.py通过;loopx check --scan-path扫描 10 个变更文件 0 错误 0 警告;loopx canary premerge(19 项检查:4 项直接检查、1 项编译、10 项目录金丝雀、8 项风险特征 smoke、1 项公开边界扫描)全部通过(0 失败)。
发现的边界问题(非阻塞 follow-up)
- [P2 · 非阻塞]
host_failure.retryRPC 缺少与record/observe对等的超大整数(≥ 10^309)传输通道:在host_failure_retry_available与explicit_host_failure_retry_available中,Python 侧直接把含原始int的failure传给host_failure.retry(hostFailureRetryRequest亦未接收attempt_text/integerAttempt)。经本地实测:当attempt为1..10**40(含2**53+1、2**60+1)时全部正常返回;但对可重试类型传入10**400时 JSJSON.parse溢出为Infinity抛出EffectRuntimeRejected,传入10**5000时 Pythonjson.dumps触发 4300 位整数序列化上限抛出ValueError(test_host_failure_policy_transport.py中仅测了非重试类型auth_failed,在 Python 早返回路径绕开了该 RPC)。在基线1a3ab75a53上,build_host_failure_record("provider_capacity", attempt=10**400)原先本就会在2 ** (10**400)处卡死,explicit_host_failure_retry_available也已拒绝>= 10**309,且正常唤醒链在max_attempts <= 3处即进入host_failure_hold,因此本问题不构成对基线的回归。建议后续在host_failure.retry补齐attempt_text/integerAttempt以对齐全链路编解码。
语义与 CI 对齐
本次仅复用既有 loopx_turn_host_failure_v0 与共享失败词汇,semantic-vocabulary-drift-smoke.py 通过,未引入新持久化 schema、权限模型或默认开启门控。远端 GitHub Actions 状态按当前评审契约未作为充分或阻塞依据(其中 merge_state 为 BEHIND,但 mergeable=MERGEABLE 且 10 个变更文件在基线到最新 origin/main 间均无冲突改动);真实安装态 worker 与端到端模型采用仍属后续独立验证边界。
我的整体评价 / Verdict
本 PR 属于 justified_increment:在不增加任何执行器、状态存储或默认激活面的前提下,干净地消除了 Python 重试表与外部 worker 本地成功/失败分叉,实现了控制面单一规则源的实质收敛。
- 长程与用户体验影响:
long_horizon为preserved(保留有限重试次数、退避 ceiling、原会话绑定与一次结算,无额外重放或漂移);user_experience为preserved,其中visual_review为not_applicable(已核对全部 10 个变更文件均属控制面 TS/Python、外部 worker 脚本、测试与 RFC,无前端或 Lark 渲染界面改动;无额外未核验渲染范围unverified_scope=none)。 - 语义对齐与架构评估:复用既有词汇(
reuse_existing),架构评估结论为retain(策略归 TS、IO 适配与无 RPC 错误构造归 Python),兼容评估为follow_up(host_failure.retry的>= 10^309传输对齐可安全延后)。全部本地单元测试、类型检查、双语 RFC 校验、语义漂移检查及 premerge 金丝雀均已通过,无阻塞项(merge_state=BEHIND与远端 CI 门禁另由合并前流程校验)。
English verdict: APPROVE - exact head 94ba859 centralizes host failure allocation and external worker phase observation in host_failure.ts; local TS/Python/premerge checks and base-head parity controls pass; one non-blocking P2 follow-up noted for attempt >= 10**309 transport on host_failure.retry, with merge readiness and parent RFC adoption remaining separate.
Goal And Delivered Outcome
Native failure records and journal recovery maintained separate retry-allocation tables. The external worker also selected native success and same-session failures in Python. They now use the existing TypeScript failure owner: one allocation produces native records and qualifies journals, and one observation boundary checks the configured Host result.
Python retains native error creation, its Mapping API/refusal order, exact integer IO, session-ID decoding and process/clock execution. The worker retains its pending witness, scope, wake ownership, stop, polling and state persistence. A native success confirms transport; it grants no Turn completion, ACK or spending authority.
main; immutable validation baseline2d302ca189cf3798ca4d53b7e01d37353ffc19d6.Author Declaration
Scope And Continuation
The same-domain refactoring pass reuses
host_failure.tsto allocate records and observe native phases. Existing RPC dispatch carries two cohesive operations; there is no new executor, state store, permission rule, or automatic activation. The internalautomatic_onlyquery distinguishes automatic eligibility from an explicitly repaired retry; omitted/false preserves the prior explicit query. Existing native protocol-kind checks remain local so error construction works without RPC.The exact integer codec preserves original Python values, including values beyond JSON's safe range and the interpreter's decimal-string conversion threshold, without changing global interpreter limits. Mapping refusal order and public record shape remain intact. Retry/observation now adds local RPC traffic on failure paths; no efficiency or outcome benefit is claimed.
The baseline's stale monitor test consumer was repaired during initial qualification. Current main subsequently shipped exact auxiliary Monitor selection and its canary. Rebase adopts that current owner/canary unchanged; this PR now contains 10 files and no monitor test delta. Earlier failures and the scoped baseline results remain recorded.
Validation
npm run typecheck:control-plane; Ruff and focused failure-owner mypy; broader worker mypy retains two unchanged baseline errorsAffected real stores: isolated local File/SQLite authority, Turn journals and worker state files. PostgreSQL authority/routing is unchanged; this does not qualify an ordinary PostgreSQL Host Turn, installed Host adoption or matched-budget model outcomes. No live model or active Goal was used for tests.
Frontend / Visual Evidence
UI impact: none. Existing App/CLI/Host readbacks keep the same native failure/session wire and controls; no caller-facing capability or frontend setting is added. Screenshots/first viewport: N/A.
Type of Change
Boundary Checklist