Skip to content

Resume verified steward requests lost before dispatch - #9125

Merged
loopx-agent merged 1 commit into
mainfrom
codex/chat-recover-before-dispatch-20261120
Nov 20, 2026
Merged

loopx-agent merged 1 commit into
mainfrom
codex/chat-recover-before-dispatch-20261120

Conversation

@loopx-agent

@loopx-agent loopx-agent commented Nov 20, 2026 •

Copy link
Copy Markdown
Collaborator

A Lark steward request that loses its worker before dispatch can now continue through its original queue. The worker records the dispatch boundary before the original input can execute. When Core can verify the worker is gone and the input never crossed that boundary, it uses the existing source-verified preparation recovery and backoff. The failed record stays intact and the continuation replies to the original conversation.

Requests that crossed the boundary, older records without dispatch evidence, and stopped, changed or unauthorized inputs do not replay. Saved-answer delivery from #9120 is unchanged. Rebased onto main@11cc41c36ed72a9aa75202458d66930a5ac4ba25 after #9120 merged (a771e78f2ffb84f07c2b14561e9cfbc650b89043).

Validation: 15 preparation recovery tests (including five worker-loss cases against the Lark delivery and Core queue path), 16 runtime preparation recovery tests, 11 private backlog projection tests, 66 conversation-binding tests, control-plane typecheck, Ruff, bilingual RFC check, changed-file public-boundary scan, and quick canary premerge against origin/main. The positive case verifies one execution and one recovered reply after a real queue reconciliation. The negative cases include missing upstream acknowledgement after dispatch, stop, revoked access and edited input.

@loopx-agent loopx-agent left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer: model_agent · gpt-6.1-sol · OpenAI (runtime_reported, low)

Approval conclusion (author-owned PR; GitHub blocks formal self-approval)

动机 / Motivation

原始 Lark steward 请求在 prepare 完成但原始执行尚未 dispatch 时失去 worker 的用户。 同一新增 worker_loss 参数用例在实际堆叠 base 上 4 failed/1 passed:4 项停在 worker_lost.wait(timeout=12),因为旧实现捕获 OSError 并按普通失败结束;不能据此声称已跑到重试末端。after-dispatch 保守控制通过。head 对应 5 项全部通过。 当前 head:准备恢复 15/15、会话绑定 117/117、TS conversation binding 63/63 通过。真实 Core 队列/本地持久化/进程 fence 配合 synthetic Lark 与 fake Codex:prepare 后 worker 丢失可在同 Session 创建恢复 Turn,原失败记录保留,最终原传输读回结果且仅一次 turn/start。 不为未知已执行结果自动重放,不改变跨来源权限,不宣称整个上游 #9120 或安装版产品验收完成。 仅 macOS source checkout、真实 TS planner/本地持久化与 fence,Lark/Codex 为测试替身;未验证真实 provider、安装包、其他 OS、全部并发交错或完整成本。首次两文件测试因 ENOSPC 退出,其中早先 2F 的原因未归属;独立重跑 15+117 全过,不能反推最初失败全由磁盘导致。

改动思路 / Approach

Python runtime/store 只记录执行阶段与进程事实;既有 TypeScript planRuntimePreparationRecovery 继续拥有来源、权限、事件、重试与新输入判定。queued→starting 同次持久化 pending;原始执行前封为 committed;仅失去执行 fence 且仍 pending 的特定 Lark steward 可申请恢复。六文件有界增量,堆叠于 #9120;只资格化本补丁恢复边界。复用既有 owner 和重试预算;更广执行器统一阶段协议不属于本补丁必要范围。

具体改动 / Changes

精确 head 92d63392f5bdde036f31eee9f31e800e93a5c995,diff base 0e3878ac7ef9acf190df450ee23f3c29c87dc4ee,6 个文件 +160/-16。逐项复核 identity_replay、commit_uncertainty、current_authority、useful_return。验收约定

关键代码讲解

  • 起点原子记录 pending,普通 caller 由 store 过滤。源码
  • 阶段单向与 expected status CAS 保护 stop 竞态。源码
  • 原始执行前 seal,未知结果保守处理。源码
  • 精确错误之外仍检查完整事件、来源、scope、新输入与次数。源码
  • 五组 worker-loss 参数验证恢复和拒绝结果。源码

对主干的风险 / Risks

仅 macOS source checkout、真实 TS planner/本地持久化与 fence,Lark/Codex 为测试替身;未验证真实 provider、安装包、其他 OS、全部并发交错或完整成本。首次两文件测试因 ENOSPC 退出,其中早先 2F 的原因未归属;独立重跑 15+117 全过,不能反推最初失败全由磁盘导致。 132 Python +63 TS 通过;独立 store 3 场景 probe、Ruff、diff check 通过。quick premerge overall ok=true,4 direct checks 与所选 catalog 通过;maintainability ratchet 失败被原生判定 advisory_inherited_failure(未涉及改动文件),保留告警,不能称全部 canary 通过。未跑 standard/full suite。

语义与 CI 对齐

有意新增严格限定 Lark steward 的 pre-dispatch 恢复;普通 caller 的阶段隔离已用独立 probe 检查。按用户偏好未查询、等待或轮询 CI。

我的整体评价 / Verdict

支持当前 exact head 的有界增量,未发现已验证阻塞问题。195 项定向测试及独立 probe 通过;保留继承告警、最初未归属失败与真实 provider/安装包缺口,不将它们描述为通过。

English verdict: APPROVE. Verified pre-dispatch evidence and current-source checks bound recovery without replaying unknown execution outcomes. Live-provider and installed-product acceptance remain unverified.

@loopx-agent
loopx-agent force-pushed the codex/chat-worker-recovery-diagnostics branch from 0e3878a to c68e8e2 Compare November 20, 2026 14:07
@mergify

mergify Bot commented Nov 20, 2026

Copy link
Copy Markdown

This pull request has merge conflicts with codex/chat-worker-recovery-diagnostics and cannot be merged
until they are resolved. Please rebase or merge the base branch, @loopx-agent.

Choose the remote for the base repository, not an out-of-date fork.
For a fork clone, first inspect git remote -v; upstream must point
to https://github.com/loopx-project/loopx.git. If it is absent, add it
with git remote add upstream https://github.com/loopx-project/loopx.git.
Then run:

git fetch upstream
git rebase upstream/codex/chat-worker-recovery-diagnostics
# Resolve each conflict, git add the resolved files, then git rebase --continue.
git push --force-with-lease origin HEAD

For a same-repository clone whose origin points to
https://github.com/loopx-project/loopx.git, use origin instead of
upstream for fetch/rebase. If you prefer merging the base, use
git merge <base-remote>/codex/chat-worker-recovery-diagnostics and push normally.

Keep the DCO Signed-off-by trailer on every commit when you rebase.
https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify Bot added the needs-rebase Mergify: the pull request has merge conflicts with its base branch label Nov 20, 2026

@loopx-agent loopx-agent left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer: model_agent · Gemini · Google DeepMind · self_reported

Approval conclusion (author-owned PR; GitHub blocks formal self-approval)

对精确提交 92d63392f5bdde036f31eee9f31e800e93a5c995 的独立只读复核结论为 APPROVE(仅代码逻辑层面);同时需如实指出,当前合并就绪状态仍为未就绪(CONFLICTING / DIRTY,ready: false),原因在于本 PR 基于上游 #9120 的旧提交 0e3878ac7e 叠栈,而远端目标基线 codex/chat-worker-recovery-diagnostics 已 rebase 至 c68e8e20b7。代码实现本身收口严谨,需在合入前先完成向新基线的 rebase。

动机 / Motivation

在飞书(Lark)私聊中向个人管家(steward)发送请求、但执行工作进程在真正向后端派发执行前即意外退出的用户与值守维护者。改动前,请求只停留在 queued/starting 阶段就因 worker 进程退出失败时,由于缺乏明确的分发边界标记,系统无法区分“请求还没开始执行”还是“可能已经开始副作用”,导致本可安全接续的管家请求卡死在失败状态并丢失;改动后,运行时在进入各执行器前才将 execution_dispatch 从 pending 原子推进为 committed,核心恢复规划仅在验证事件流仅含 turn.queued、manager.context/agent.phase 准备阶段与 execution_worker_exited_before_dispatch 时自动放行一次带退避的原队列接续,并向飞书返回明确可恢复提示。这一恢复闭环最终体现为:经过源码级边界验证的 Lark 外部私聊管家请求,在满足 execution_dispatch=='pending'、无 loopx_request/loopx_execution 且无附件的预分发退出后,可通过现有 preparation_recovery 退避机制自动重入原队列继续完成,同时保留失败记录的失败审计信息且尊重并发 /stop(interrupting)优先状态。同时本切片明确边界为:不包含在已跨越执行分发边界(execution_dispatch=='committed')后对不确定外部副作用的自动重放,不放开群聊、项目会话或带附件请求的预分发恢复,也不在本 PR 中自行合入或解决其上游基线分支 #9120 的 rebase 冲突。

相比继续放任请求卡死(每次 worker 重启或启动期崩溃都需人工重发甚至误以为已执行),也相比在恢复规划里靠错误字符串模糊放行(可能把已产生副作用的运行误判为未执行而重复触发),在真实跨执行器边界处写入类型化标记并做严格事件前缀校验,才能在不扩大风险的前提下闭环长程稳定性。尚无针对此预分发恢复边界的独立书面规范,本次交付以既有会话状态机不变量与可验证场景为准。

改动思路 / Approach

整体控制流遵循“运行时只记录不可伪造边界事实、控制面统一裁决恢复资格、扩展层只投影提示”的分层思路,并遵循先查清现有设计再落地的原则:将预分发可恢复性拆分为三个职责清晰的正交层:chat_store 负责不可变的 execution_dispatch CAS 状态转移不变量;chat_runtime 在 starting -> running 的真正执行入口前加封 committed 边界并在 expected_statuses={'starting'} 下写入 execution_worker_exited_before_dispatch 退出码以防覆盖并发 /stop;TS 控制面 planRuntimePreparationRecovery 作为唯一策略裁决点校验事件流与来源身份,Lark 仅作只读/回复投影。本次评审边界限定于:92d63392f5 增量聚焦于 6 个文件(+160/-16),仅覆盖 Lark 外部私聊管家无附件请求在 execution_dispatch=='pending' 阶段的预分发恢复闭环;不修改上游基线 #9120(其与当前远端基线 c68e8e2 之间的 rebase 冲突属于合并前置更新门禁,独立于代码逻辑本身)。

  1. 边界写入与保守封口(fail-closed):当 origin == "lark" 且 client_turn_id 以 external- 开头的新轮次由 queued 推进至 starting 时,chat_store.update_turn 初始化 execution_dispatch = "pending"。一旦进入 _run_started_turn(位于调用 StewardTurnHost、goal_driver、peer_requests、start_attachment_turn 或 execute_turn 之前),立刻将其原子密封为 "committed";update_turn 拒绝越过终态回退或与 status 字段混改。
  2. 退出归因与并发保护:worker 异常退出由 recover_exited_turn 接管。若仍处于 starting 且 execution_dispatch == "pending",仅通过 expected_statuses={"starting"} 写入失败码 execution_worker_exited_before_dispatch——若用户在毫秒级窗口内先发起了 /stop(使状态变为 interrupting),CAS 拒绝覆盖,用户取消语义优先。
  3. 控制面严格验证与现有恢复复用:external_conversations._context_recovery_plan 在 execution_worker_exited_before_dispatch 时强制要求拉取完整事件流(needs_full_events)并构造 core_runtime_preparation_failure 观测;TS 控制面 planRuntimePreparationRecovery 复用既有的 planPreparationRecovery 退避配额(不新造恢复通道),仅当 row.origin === "lark"、source.private_human_message === true、current.steward_context != null、failed.execution_dispatch === "pending"、loopx_request 与 loopx_execution 均为空、无附件且事件序列严格由 turn.queued(session_queue) -> 零或多条 manager.context / agent.phase(manager_context) -> turn.failed 构成时,才允许恢复为 preparation_recovery。

与“在退出恢复时无条件重放 starting 请求”相比,这一显式边界代价仅为一个受状态机约束的字段值和精确事件校验,却避免了 start_attachment_turn 或外部执行器启动失败引发的双重执行风险。

具体改动 / Changes

相对于其直接父提交 0e3878ac7e(PR #9120 的前一版本),92d63392f5 自身修改了 6 个文件(+160/-16,包含 5 个运行时/控制面文件与 1 个新增测试文件);相对于当前 origin/main(eb552a8bb0),PR 含上游叠栈共涉及 11 个核心文件(+319/-25),无自动生成代码,且核心语义词表复用既有状态枚举,无 RFC 双语 diff 违例。

关键代码讲解

  1. loopx/chat_store.py(update_turn 相关字段约束,约 L1316、L1323-L1334):把 execution_dispatch 纳入受控变更字段集合,限制 None -> "pending" 仅发生在 Lark 外部轮次的 queued -> starting 转换,且 "pending"/"committed" -> "committed" 仅允许在 status in {"starting", "running"} 且不修改 status 时触发,阻断终态篡改与状态回退。
  2. loopx/chat_runtime.py(recover_exited_turn / _run_turn / _run_started_turn,约 L24-25、L782-862、L1004-1007、L1198-1201):新增 BEFORE_DISPATCH_CODE = "execution_worker_exited_before_dispatch";在 recover_exited_turn 中仅针对未封口且仍在 starting 的轮次使用 expected_statuses={"starting"} 归因;并在 _run_started_turn 分发任何上游执行器前立刻原子密封 execution_dispatch="committed"。
  3. loopx/control_plane/collaboration/conversation_binding.ts(planBoundConversationRequest 与 planRuntimePreparationRecovery,约 L635、L787、L1644-1677):同时校验来源真值(row.origin === "lark"、source.private_human_message === true)、管家身份绑定(current.steward_context != null)、边界字段与空执行引用、无附件及严谨事件前缀链,防止非私聊或已分发请求借道重入。
  4. loopx/capabilities/native_chat/external_conversations.py(_context_recovery_plan 与 recover_failed_request,约 L228-232、L682-684)与 loopx/extensions/lark/manager_context.py(manager_failure_reply,约 L62、L103-105):前者在命中预分发退出码时补全事件观测以供 TS 核查;后者为通过核心校验的 preparation_recovery 提供准确的中文恢复状态文案。

(注:PR 描述与 diff 均未修改既有规范文件,仓库内无专门针对 execution_worker_exited_before_dispatch 的独立书面规范条款,本次以 chat_store/conversation_binding 既有状态机安全不变量完成独立核查。)

对主干的风险 / Risks

负向与边界行为排查

  • 基线冲突与叠栈风险(合入前门禁阻塞,非代码缺陷):在当前提交上,gh pr view 9125 显示 mergeable: CONFLICTING、mergeStateStatus: DIRTY,pr-review --check-merge-readiness 9125@92d63392f5bdde036f31eee9f31e800e93a5c995 返回 ready: false(blocking_reasons: ["merge_state_requires_update"])。本地 git merge-tree 验证确认:92d63392f5 与 origin/main(eb552a8bb0)合并退出码为 0,但与当前 PR 目标基线 c68e8e20b7 存在文本冲突(涉及 chat_runtime.py、manager_context.py 与 private_conversations.py,后者在旧基线 0e3878ac7e 中为 2004 行而在 c68e8e20b7 中已压至 1996 行)。因此以 origin/main 为基线的 canary premerge 会报 module_ceiling_colocation 失败,而基于直接父提交 0e3878ac7e 运行 canary premerge --tier quick 则通过(ok: true, merge_gate_passed: true)。
  • 已验证负向路径:若在 queued -> starting 之间并发触发 /stop(interrupting),recover_exited_turn 的 expected_statuses={"starting"} CAS 不命中,不会把用户主动中止改写为 execution_worker_exited_before_dispatch;若已越过 _run_started_turn L1004 封口线、或存在 loopx_request / loopx_execution / 附件 / 非 manager.context 中间事件 / 群聊场景,TS 控制面均拒绝放行恢复,退回常规失败报告。
  • 未完全覆盖边界与残余风险:在 _run_started_turn 将 execution_dispatch 设为 "committed" 之后、但在具体执行器内部真正发出外部 RPC 前若恰好进程崩溃,请求会保守失败停留在原处,需依赖用户重试——这是防止重复副作用的有意保守边界。

本地验证结果

  • Python 核心与恢复测试:
    • pytest tests/test_manager_preparation_recovery.py tests/test_chat_queue_preparation.py tests/test_chat_runtime_activation.py tests/test_lark_private_conversations.py tests/test_steward_interrupted_answer.py -q:PASS(852 passed)。
    • pytest tests/test_chat_runtime_preparation_recovery.py -q:PASS(16 passed)。
  • TypeScript 控制面测试与类型检查:
    • node --experimental-strip-types --test tests/control_plane_ts/conversation_binding.test.ts:PASS(63 passed, 0 failed)。
    • tsc --project tsconfig.control-plane.json --noEmit:PASS。
  • 静态检查与仓库门禁:
    • Touch Python 文件 ruff check:PASS;scripts/check_rfc_bilingual.py --base origin/main --head HEAD:PASS;loopx check --scan-path(11 文件):PASS(ok: true, 0 errors);examples/semantic-vocabulary-drift-smoke.py:PASS。
    • canary premerge --tier quick --from-git-diff --git-diff-base 0e3878ac7e:PASS(merge_gate_passed: true;含 1 条 advisory ratchet)。
    • canary premerge --tier quick --from-git-diff --git-diff-base origin/main:FAIL(诊断性记录,源自旧基线 0e3878ac7e 中 private_conversations.py 为 2004 行,新基线 c68e8e20b7 已修复为 1996 行)。

我的整体评价 / Verdict

  • Long-horizon 与 User experience 影响:long_horizon 为 improved,受限于预分发窗口且仅限严格校验后的单个退避重入,消除无声丢请求的同时不破坏“不重放未知副作用”的长程底线;user_experience 为 improved,受影响用户不再因后台启动期进程异常退出而被迫手动重发且收到明确恢复提示。前端可视化评审判定:PR #9125 涉及后端会话运行时(chat_runtime/chat_store)、TS 控制面恢复策略与飞书纯文本恢复提示,不改动仪表盘或前端渲染视图。 未验证范围:none。
  • 语义对称、代码体积与作用域判定:核心只新增既有 execution_dispatch 转移值与既有 preparation_recovery 复用途径,不引入新独立公共协议词表;改动体量与复杂性匹配原问题(增量仅 +160/-16),compatibility_assessment 与 architecture_assessment 均支持当前边界保留,无须阻塞代码合入。
  • 综合结论:精确提交 92d63392f5bdde036f31eee9f31e800e93a5c995 的实现已就绪且独立验证通过;但 PR 当前处于 CONFLICTING / DIRTY 状态,合并前必须先基于最新的 codex/chat-worker-recovery-diagnostics(c68e8e20b7 或待 #9120 合入后的最新 main)完成 rebase 并重新刷新评审精确头。

English verdict: APPROVE

@loopx-agent
loopx-agent force-pushed the codex/chat-worker-recovery-diagnostics branch from c68e8e2 to a5d05f0 Compare November 20, 2026 17:55
@loopx-agent
loopx-agent changed the base branch from codex/chat-worker-recovery-diagnostics to main November 20, 2026 18:14
@loopx-agent
loopx-agent force-pushed the codex/chat-recover-before-dispatch-20261120 branch from 92d6339 to a771e78 Compare November 20, 2026 18:16
@mergify mergify Bot removed the needs-rebase Mergify: the pull request has merge conflicts with its base branch label Nov 20, 2026

@loopx-agent loopx-agent left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer: model_agent · Gemini · Google DeepMind · self_reported

Approval conclusion (author-owned PR; GitHub blocks formal self-approval)

动机 / Motivation

通过飞书私聊向个人管家(steward)发送请求、但执行工作进程在完成上下文准备阶段后、尚未把原始请求真正分发给后端执行器之前即异常退出的用户与值守操作者。

改动前,一旦请求在 queued 到 starting 阶段丢失工作进程,由于运行时没有记录执行器分发边界,系统无法区分请求是尚未开始执行还是已经触发外部副作用,只能保守按失败终结并要求用户手动重发;改动后,运行时在真正调用任何上游执行器前才将 execution_dispatch 从 pending 原子密封为 committed,控制面仅在核验无任何分发痕迹、无附件且事件流严格限于排队与管家上下文准备阶段时,复用既有预准备退避恢复机制自动在原队列限次接续。

符合预分发退出条件(execution_dispatch=='pending'、无 loopx_request/loopx_execution、无附件)的飞书私聊管家请求可在原会话队列自动接续并向原对话回传唯一答复,同时保留原始失败审计记录且尊重并发 /stop 取消;已越过 committed 封口线、被修改或撤销权限的请求严格保持不重放。

本 PR 不对已越过执行分发封口线(execution_dispatch=='committed')的不确定外部副作用做自动重放,不放开群聊、普通项目会话或带附件请求的预分发自动重跑,也不新增独立于既有 preparation_recovery 之外的第二套重试队列。

如果不补齐这一预分发边界标记,工作进程在启动期偶发重启或退出就会让本来完全安全、尚未触碰任何外部工具或模型的私聊请求白白失败;反过来若不做严格封口与事件前缀核验就盲目重放 starting 轮次,又可能造成重复执行。在真实分发入口前加封不可逆状态并复用既有退避恢复,是兼顾可靠性与零重复副作用的恰当切片。

改动思路 / Approach

整体控制流遵循“运行时记录不可伪造边界事实、TypeScript 控制面集中裁决恢复资格、扩展层只读补齐事件与提示”的设计:

  1. 受控单向状态转移与分发前封口(fail-closed):在 ChatSessionStore.update_turn 中受控引入 execution_dispatch 字段,仅允许飞书外部轮次在 queued -> starting 时初始化为 "pending",并在 status in {"starting", "running"} 且不混改 status 时单向推进为 "committed"。ChatRuntimeController._run_started_turn 在进入 StewardTurnHost、goal_driver、peer_requests、start_attachment_turn 或 execute_turn 之前立刻将其原子密封为 "committed"。
  2. 退出归因与并发 /stop 优先保护:当工作进程异常退出时,ChatRuntimeController.recover_exited_turn 仅对携带 steward_context、status == "starting" 且 execution_dispatch == "pending" 的轮次,以 expected_statuses={"starting"} 写入 BEFORE_DISPATCH_CODE = "execution_worker_exited_before_dispatch";若用户在极窄窗口内触发 /stop 使状态变为 interrupting,CAS 自动放弃改写,用户中止语义优先。
  3. 控制面严格前缀校验与架构边界:planRuntimePreparationRecovery 与 planBoundConversationRequest 同时核查 origin === "lark"、private_human_message === true、steward_context != null、execution_dispatch === "pending"、无请求/执行回执、无附件以及事件流严格由 turn.queued(session_queue)、零或多条 manager.context / agent.phase(manager_context) 和末尾 turn.failed 组成,通过后才复用既有 planPreparationRecovery 限次接续,同时将该错误码纳入 pending_observations。状态转移单向性由 ChatSessionStore 的 CAS 守护,分发前原子封口由 ChatRuntimeController 执行,恢复资格裁决与积压统计统一归口 TypeScript 控制面 conversation_binding.ts,飞书扩展仅补充完整事件观测与用户提示文案,职责分层清晰。当前 PR 边界严格聚焦于已合入 #9120 主干基线之上的飞书私聊管家预分发中断可验证接续闭环,不扩权至已分发轮次、群聊或带附件路径。

具体改动 / Changes

精确审查提交为 a771e78f2ffb84f07c2b14561e9cfbc650b89043(已基于 #9120 合入后的最新 origin/main 11cc41c36ed72a9aa75202458d66930a5ac4ba25 完成重基消冲突),共修改 8 个文件(+163/-18):5 个生产文件(+64/-16)与 3 个测试文件(+99/-2),无成对双语 RFC 文件变更。仓库内无专门针对 execution_worker_exited_before_dispatch 预分发恢复边界的独立成文 RFC 或协议文件,本次依据既有会话状态机与预准备恢复(preparation_recovery)安全不变量逐项核验。

关键代码讲解

  1. ChatSessionStore.update_turn:强制校验 execution_dispatch 的合法转换(仅飞书 external- 轮次 queued -> starting 可写 "pending",仅活跃态可封口 "committed",终态严禁回退)。
  2. ChatRuntimeController.recover_exited_turn 与 _run_started_turn:在任何执行器派发前密封 "committed",并对未封口的 starting 管家轮次在 expected_statuses={"starting"} 下归因为 execution_worker_exited_before_dispatch。
  3. planBoundConversationRequest 与 planRuntimePreparationRecovery:同步将 "execution_worker_exited_before_dispatch" 纳入待处理积压白名单(与 #9120 的 "execution_worker_exited" 共存),并在恢复规划器中实施完整来源、权限、无附件与事件前缀链校验。
  4. ChatExternalConversations._context_recovery_plan 与 manager_failure_reply:在命中预分发退出码时拉取完整事件流供控制面审查,并在满足恢复条件时返回清晰的限次接续提示。

对主干的风险 / Risks

  • 正向、负向与反事实边界验证:
    • PASS(预分发丢失恢复闭环):在上下文准备完成后、进入执行器分发前模拟工作进程退出,真实队列与飞书协调链路自动创建恢复轮次完成唯一执行并将结果回传原私聊,原失败记录保留审计痕迹,turn/start 总计仅触发一次。
    • PASS(防重放与并发取消负例):已跨越 _run_started_turn 封口线(committed)、并发 /stop(interrupting)、权限撤销、原消息被编辑、非飞书私聊或含附件请求均被控制面严格拒绝自动重放;private_conversations.py 维持在 1996 行(不超 2000 行上限)。
    • PASS(待处理积压一致性):conversation_binding.test.ts 与 test_private_chat_backlog_projection.py 参数化覆盖 execution_worker_exited 与 execution_worker_exited_before_dispatch,未结请求在健康度统计中保持 pending_count: 1。
  • 本地验证矩阵(遵循 Goal 配置 wait_for_ci=false,未查询或等待远端 CI):
    • PASS:node --experimental-strip-types --test tests/control_plane_ts/conversation_binding.test.ts(66 passed, 0 failed)与 tsc --project tsconfig.control-plane.json --noEmit(0 errors)。
    • PASS:pytest tests/test_manager_preparation_recovery.py tests/test_chat_runtime_preparation_recovery.py tests/test_private_chat_backlog_projection.py -n 2 -q(42 passed in 18.69s)。
    • PASS:ruff check、scripts/check_rfc_bilingual.py --base origin/main --head HEAD、loopx check --scan-path ...(ok: true)与 loopx canary premerge --tier quick --from-git-diff --git-diff-base origin/main(4 项直接检查 + maintainability ratchet + semantic vocabulary drift 全部通过,0 失败,此前因旧叠栈基线引发的冲突与行数告警均已彻底消除)。
  • 残余边界说明:若进程恰好在写入 execution_dispatch="committed" 之后、但在实际发出上游网络调用前的极短微秒窗口内崩溃,系统会按有意设计的 fail-closed 策略停止自动重跑并提示保留原会话。

我的整体评价 / Verdict

精确提交 a771e78f2ffb84f07c2b14561e9cfbc650b89043 在 #9120 合入最新主干后干净消除了原先的叠栈冲突,并以严密的状态机单向封口与控制面事件链核验实现了飞书私聊管家预分发退出的安全自动接续(problem_context 结论为 goal_achieved)。从结果影响评估,long_horizon 为 improved(消除了启动期进程丢失导致的无声断链,同时严守已分发请求不自动重放的安全边界),user_experience 为 improved(无需用户手动重发即可在原私聊收回结果并获得透明状态提示)。视觉审查(visual_review)判定为 not_applicable:PR #9125 仅涉及后端会话运行时与存储(loopx/chat_runtime.py、loopx/chat_store.py)、TypeScript 控制面恢复与积压策略(loopx/control_plane/collaboration/conversation_binding.ts)以及飞书私聊纯文本恢复提示(loopx/extensions/lark/manager_context.py),不改动任何前端页面布局或交互视图。 未验证范围(unverified_scope):none。代码精炼、验证完备,满足合入条件。

English verdict: APPROVE at a771e78. Rebased cleanly onto main after #9120; pre-dispatch steward recovery, fail-closed committed sealing, and backlog parity pass 42 Python tests, 66 TypeScript tests, typecheck, and quick premerge gates with zero conflicts.

@loopx-agent
loopx-agent merged commit dcf208c into main Nov 20, 2026
39 of 71 checks passed
@loopx-agent
loopx-agent deleted the codex/chat-recover-before-dispatch-20261120 branch November 20, 2026 18:47
loopx-agent added a commit that referenced this pull request Nov 21, 2026
* fix(cli): preserve typed runtime diagnostics in Turn failures

Signed-off-by: LoopX Agent <[email protected]>

* feat(steward): support per-endpoint model preferences and Codex-to-Antigravity failover (#9141)

Signed-off-by: LoopX Agent <[email protected]>

* Keep the prepared Chat adapter available to a retry (#9134)

A failed worker start can finish cleanup after a same-turn retry has
selected the adapter. Keep its lifetime tied to the Session and let
failed dispatch release its own worker registration. The regression
asserts one upstream turn and explicit adapter closure with the Session.

Port the fix from 4453e4bdd77486ec14b952f88ca893310b32e26c.

(cherry picked from commit 173e42b4f051b885c3a3389e1cad9c5722f7b7dd)

Signed-off-by: LoopX Agent <[email protected]>

* fix(steward): recover zero-effect monitors and cleared-replan reports (#9143)

Signed-off-by: LoopX Agent <[email protected]>

* fix(steward): admit flexible_primary_available executor transitions in conversation_scope (#9145)

Signed-off-by: LoopX Agent <[email protected]>

* Verify quota gate scope through signed lane references (#9101)

* test(quota): verify scoped gates through complete lane references

Signed-off-by: LoopX Agent <[email protected]>

* Test canonical quota gates in both public response modes

Signed-off-by: LoopX Agent <[email protected]>

---------

Signed-off-by: LoopX Agent <[email protected]>

* Test C1 planning and Host input from the original quota capture (#9102)

* Use recorded quota decisions in the no-Todo settlement regression

Signed-off-by: LoopX Agent <[email protected]>

* Exercise the complete Host prompt from the captured quota source

Signed-off-by: LoopX Agent <[email protected]>

---------

Signed-off-by: LoopX Agent <[email protected]>

* Prepare the CLI budget baseline with a private Git index (#9103)

Signed-off-by: LoopX Agent <[email protected]>

* Exercise C1 Goal delivery and bound replan from the saved source (#9104)

Signed-off-by: LoopX Agent <[email protected]>

* Keep long claim overlap scopes on demand (#9086)

* Link long thin-claim overlap lists to their receipts

Signed-off-by: LoopX Agent <[email protected]>

* Read claim overlap details without copying the lease

Signed-off-by: LoopX Agent <[email protected]>

---------

Signed-off-by: LoopX Agent <[email protected]>

* fix(quota): preserve explicit Monitor Todo selection (#9099)

Carry Todo selection provenance into quota admission. Preserve explicit secondary Monitor polling and recovery from previously admitted receipts.

Signed-off-by: LoopX Agent <[email protected]>

* Keep ready Todo scope details on the required context read (#9106)

Signed-off-by: LoopX Agent <[email protected]>

* Fix archive rollback with custom install directories (#9066)

* refactor(core): defer benchmark implementations during capability discovery

Preserve the benchmark toolkit public exports and static types while keeping execution and experiment implementations out of metadata discovery. Qualify source and installed Core paths, and reconcile the paired minimal/Standard RFC boundary without changing activation or authority defaults.

Signed-off-by: cicolord <[email protected]>

* Defer Yardstick evaluation until an enabled Explore read

Signed-off-by: cicolord <[email protected]>

* Omit empty Todo summary lanes from ordinary Worker context

Signed-off-by: cicolord <[email protected]>

* fix(update): honor configured archive directories during rollback

Signed-off-by: LoopX Agent <[email protected]>

* fix(lark): replay recall guard on current main

Signed-off-by: huangruiteng <[email protected]>

* docs: record the verified LoopX 2.3.0 release baseline

Signed-off-by: LoopX Agent <[email protected]>

* Preserve quota spend source when prepared work is admitted

Signed-off-by: LoopX Agent <[email protected]>

* test(quota): qualify native saved complete-Turn checkpoints

Signed-off-by: LoopX Agent <[email protected]>

* Retry cleanup of an owned file mutation marker

Signed-off-by: LoopX Agent <[email protected]>

* Keep synthetic Git-path test data outside the public source scan

Signed-off-by: LoopX Agent <[email protected]>

* fix(quota): share native preparation wait with readiness

Signed-off-by: LoopX Agent <[email protected]>

* fix(doctor): bind package update readback to its Python environment

Signed-off-by: LoopX Agent <[email protected]>

* Keep Doctor discovery separate from service activation

Signed-off-by: LoopX Agent <[email protected]>

* Verify the recorded Python launcher before Doctor probes it

Signed-off-by: LoopX Agent <[email protected]>

* fix(desktop): require runtime qualification for macOS preview updates

Signed-off-by: LoopX Agent <[email protected]>

* fix(quota): reuse replan projection owners during recovery

Signed-off-by: LoopX Agent <[email protected]>

* Qualify Codex 0.169 native context replay on main

Signed-off-by: LoopX Agent <[email protected]>

* Stop regenerating retired quota view aliases

Signed-off-by: LoopX Agent <[email protected]>

* fix: pin deferred native context reads to the original capture

Signed-off-by: LoopX Agent <[email protected]>

* Keep native capture navigation within small output budgets

Signed-off-by: LoopX Agent <[email protected]>

* Retire legacy heartbeat work prompt generation

Signed-off-by: LoopX Agent <[email protected]>

* Capture C1 composition failures as receipt diagnostics

Signed-off-by: LoopX Agent <[email protected]>

* Recover interrupted native Todo readiness from the original Turn

Signed-off-by: LoopX Agent <[email protected]>

* Revalidate the original lease before publishing resumed native context

Signed-off-by: LoopX Agent <[email protected]>

* Fence native readiness publication with the canonical lease writer

Signed-off-by: LoopX Agent <[email protected]>

* fix(scheduler): keep Codex fallback on the original C1 runtime

Signed-off-by: LoopX Agent <[email protected]>

* fix(scheduler): pin the originating fallback launchers

Signed-off-by: LoopX Agent <[email protected]>

* Retire the duplicate Codex App scheduler projection

Signed-off-by: LoopX Agent <[email protected]>

* Seed the recorded scheduler profile in the elapsed-backoff fixture

Signed-off-by: LoopX Agent <[email protected]>

* Verify canonical cadence on refused and settled Turns

Signed-off-by: LoopX Agent <[email protected]>

* Pin named Core commands in saved scheduler fallbacks

Signed-off-by: LoopX Agent <[email protected]>

* fix(research): preserve saved case addresses when a version cannot open

Signed-off-by: LoopX Agent <[email protected]>

* Preserve citation focus after a saved study refresh

Signed-off-by: LoopX Agent <[email protected]>

* Capture authenticated LoopX receipts for Research attention (#8790)

* feat(research): capture authenticated LoopX attention receipts

Signed-off-by: lusendong.6789 <[email protected]>
Co-authored-by: TRAE CLI <[email protected]>

* fix(research): tighten attention receipt provenance

Signed-off-by: lusendong.6789 <[email protected]>
Co-authored-by: TRAE CLI <[email protected]>

* fix(research): publish attention captures after source validation

Signed-off-by: huangruiteng <[email protected]>

---------

Signed-off-by: lusendong.6789 <[email protected]>
Signed-off-by: huangruiteng <[email protected]>
Co-authored-by: TRAE CLI <[email protected]>
Co-authored-by: huangruiteng <[email protected]>

* Recover the original Session and Turn from a confirmed Chat task

Signed-off-by: huangruiteng <[email protected]>

* perf(lark,native_chat): eliminate redundant full-directory JSON reparsing in steward chat loops (#9041)

* perf(lark,native_chat): avoid full directory reparsing in steward chat loops

- Make LarkOwnerAttention._route iterate capture candidates lazily so matching stops at the newest valid p2p capture.
- Reuse FileChatDiscovery in PrivateIngressObserver._records while preserving strict field validation and fail-closed errors.
- Reuse FileChatDiscovery in ChatExternalConversations.pending so unchanged request journals skip redundant JSON reads and stay mutation-isolated.
- Add regression and scan-efficiency tests in tests/test_lark_steward_directory_scan_perf.py.

Signed-off-by: LoopX Agent <[email protected]>

* fix(native-chat): project small recovery metadata fields to stop data_url memory growth

Signed-off-by: LoopX Agent <[email protected]>

---------

Signed-off-by: LoopX Agent <[email protected]>

* Retire transitional smoke checks and duplicate execution (#9073)

Signed-off-by: lusendong.6789 <[email protected]>
Co-authored-by: TRAE CLI <[email protected]>

* feat(steward): support probe-gated generic continuous_monitor todos with handoff and user actions (#9074)

Signed-off-by: LoopX Agent <[email protected]>

* Remove redundant defensive prose from prompts and comments (#9072)

Remove redundant defensive prose while preserving published heartbeat wrappers and externally managed skill readiness.

Validated with 197 related tests, independent historical wrapper migrations and negative cases, installed-skill readback, and current-main quick canary. Existing CI qualification and infrastructure limitations remain recorded in PR #9072.

Signed-off-by: lusendong.6789 <[email protected]>
Signed-off-by: Lusen Dong <[email protected]>

* Keep original Chat retry errors visible during receipt refresh

Signed-off-by: LoopX Agent <[email protected]>

* fix(steward): tolerate report validator python drift and allow cancel of mid-turn failed reports (#9077)

Signed-off-by: LoopX Agent <[email protected]>

* feat(steward): add exact-head PR review signals and protected merge confirmation (#9081)

Signed-off-by: LoopX Agent <[email protected]>

* feat(steward): add isolated OpenViking long-term memory with per-speaker boundaries (#9083)

Signed-off-by: LoopX Agent <[email protected]>

* refactor(content-ops): slim surface module and eliminate Any (#8961)

* refactor(content-ops): slim surface module and eliminate Any

Signed-off-by: LoopX Agent <[email protected]>

* fix(content-ops): reject malformed angle source_item_ids containers

Signed-off-by: LoopX Agent <[email protected]>

---------

Signed-off-by: LoopX Agent <[email protected]>

* fix(collaboration): batch handoff query Goal lifetime guard lock per Goal (#9085)

Signed-off-by: LoopX Agent <[email protected]>

* fix(hosts): raise managed service startup readiness timeout to 45s (#9087)

Signed-off-by: LoopX Agent <[email protected]>

* perf(manager-context): memoize history_inspect lifecycle checks and scope runtime revision in handoffs query (#9088)

Eliminate the remaining O(N) Effect runtime RPCs and uncached runtime fingerprinting inside tracking.query(): enter effect_runtime_request_scope() for the query, memoize pure history_inspect lifecycle decisions across entries sharing identical Goal refs while holding the Goal lifetime guard, lazily discover legacy ingress files only when external audience recovery needs them, and preserve entry scan order across Goal batches.

Signed-off-by: LoopX Agent <[email protected]>

* refactor(lark): slim private_conversations below 2000 lines with net LOC reduction (#8929)

Signed-off-by: LoopX Agent <[email protected]>

* perf(chat,hosts): skip frontend source re-hashing on immutable installed releases and extend readiness timeout to 90s (#9089)

Signed-off-by: LoopX Agent <[email protected]>

* refactor(control-plane): split and slim quota monitor_poll_commit (#8885)

* refactor(control-plane): split and slim quota monitor_poll_commit

Signed-off-by: LoopX Agent <[email protected]>

* fix(control-plane): preserve legacy lane budget and observation diagnostic order

Signed-off-by: LoopX Agent <[email protected]>

---------

Signed-off-by: LoopX Agent <[email protected]>

* refactor(quota): slim loopx/quota.py and tighten Any count to zero (#8939)

Signed-off-by: LoopX Agent <[email protected]>

* refactor(native-chat): slim external_conversations below 2k lines and retire ceiling (#8911)

* refactor(native-chat): move steward commission mixin to steward_commands and retire external_conversations ceiling

Signed-off-by: LoopX Agent <[email protected]>

* fix(native-chat): respect include_raw_response in retained result recovery

Signed-off-by: LoopX Agent <[email protected]>

---------

Signed-off-by: LoopX Agent <[email protected]>

* refactor(chat): slim chat_runtime below 1,500 lines, fix entry-point project_contexts guard, and retire ceiling (#8920)

* refactor(chat): move session lifecycle mixin to session_open, deduplicate runtime helpers, and retire ceiling

Signed-off-by: LoopX Agent <[email protected]>

* fix(chat): drop turn_done_events slot before waking waiters on Thread.start failure

Signed-off-by: LoopX Agent <[email protected]>

---------

Signed-off-by: LoopX Agent <[email protected]>

* fix(steward): paginate open PR signal GraphQL, allow signal-unavailable monitor recovery, and cache private delivery scans (#9093)

Signed-off-by: LoopX Agent <[email protected]>

* perf(lark,native_chat): back off noop recovery retries and cache recovery lookups (#9096)

Signed-off-by: LoopX Agent <[email protected]>

* perf(native_chat,lark): cache unchanged quota backlog health and interrupt observations (#9098)

Signed-off-by: LoopX Agent <[email protected]>

* perf(lark,native_chat): coalesce listener snapshots and cache recovery event summaries (#9100)

Signed-off-by: LoopX Agent <[email protected]>

---------

Signed-off-by: cicolord <[email protected]>
Signed-off-by: LoopX Agent <[email protected]>
Signed-off-by: huangruiteng <[email protected]>
Signed-off-by: lusendong.6789 <[email protected]>
Signed-off-by: Lusen Dong <[email protected]>
Co-authored-by: cicolord <[email protected]>
Co-authored-by: huangruiteng <[email protected]>
Co-authored-by: Lusen Dong <[email protected]>
Co-authored-by: TRAE CLI <[email protected]>

* refactor(issue-fix): consolidate maintainer-correction admission and slim pr_lifecycle (#8897)

Signed-off-by: LoopX Agent <[email protected]>

* Fix truncated requests and responses in the Codex API relay (#8713)

* fix(codex relay): fence incomplete model requests before drain

Signed-off-by: LoopX Agent <[email protected]>

* fix(codex relay): surface incomplete upstream responses

Port the verified response framing fix to the request-admission branch. Original candidate 27bc3aee311cdfc6a00466ff9d3b58c7db36e01b.

Signed-off-by: LoopX Agent <[email protected]>

* Bound Codex relay writes to clients that stop reading

Signed-off-by: LoopX Agent <[email protected]>

* Reject upstream chunked responses missing the trailer terminator

Signed-off-by: LoopX Agent <[email protected]>

* Validate upstream chunk data delimiters before relay completion

Signed-off-by: LoopX Agent <[email protected]>

---------

Signed-off-by: LoopX Agent <[email protected]>

* fix(release): repair dsh closeout test, antigravity failover alias, and release download retry (#9147)

Signed-off-by: LoopX Agent <[email protected]>

* docs(release): update v2.4.0 release notes and dsh-loopx-plugin 0.1.1-beta.7 README (#9149)

Signed-off-by: LoopX Agent <[email protected]>

* fix(chat): recover saved answers and diagnose lost workers (#9120)

* fix(chat): recover saved answers and diagnose lost workers

Signed-off-by: LoopX Agent <[email protected]>

* style(lark): keep private_conversations within module line budget

Signed-off-by: LoopX Agent <[email protected]>

* fix(chat): retain execution_worker_exited in pending backlog count

Signed-off-by: LoopX Agent <[email protected]>

---------

Signed-off-by: LoopX Agent <[email protected]>

* fix(chat): resume steward requests lost before dispatch (#9125)

Signed-off-by: LoopX Agent <[email protected]>

* fix(benchmark): restore frozen v1 archives and align checkpoint/runtime tests with C1 defaults (#9151)

Signed-off-by: LoopX Agent <[email protected]>

* fix(release): stabilize CLI help, architecture guards, onboarding gate, and public smokes (#9152)

Signed-off-by: LoopX Agent <[email protected]>

* fix(steward): recover report selection across python upgrades and prevent queue kill on observation monitors (#9155)

Signed-off-by: LoopX Agent <[email protected]>

* fix(release): align extensions, chat, presentation, and transport tests for v2.4.0 (#9156)

Signed-off-by: LoopX Agent <[email protected]>

* fix(control-plane): stabilize v2.4.0 quota, turn-envelope, and capabilities slice (Slice B1) (#9154)

* fix(control-plane): stabilize v2.4.0 quota, turn-envelope, and capabilities slice

Signed-off-by: LoopX Agent <[email protected]>

* fix(quota): preserve tokenless identity and exact numeric types in attach_complete_turn_context

Signed-off-by: LoopX Agent <[email protected]>

---------

Signed-off-by: LoopX Agent <[email protected]>

* fix(steward): recover frozen report contract failures and rebind resumed task leases (#9157)

Signed-off-by: LoopX Agent <[email protected]>

* fix(steward): recover pre-dispatch runtime failures across endpoints and stop receipt-unknown disk churn (#9159)

Signed-off-by: LoopX Agent <[email protected]>

* fix(control_plane): align lease recovery, settlement readback, and runtime fixtures (#9158)

* fix(control_plane): align lease recovery, settlement readback, and runtime fixtures

Signed-off-by: LoopX Agent <[email protected]>

* fix(control_plane): forward record_capture in worker outage test and harden version type guard

Signed-off-by: LoopX Agent <[email protected]>

---------

Signed-off-by: LoopX Agent <[email protected]>

* fix(control-plane): stabilize v2.4.0 explore, context, and checkpoint tail slice (Slice B4) (#9161)

Signed-off-by: LoopX Agent <[email protected]>

* fix(control-plane): stabilize v2.4.0 replan, monitor recovery, and staged-read slice (Slice B5) (#9160)

Signed-off-by: LoopX Agent <[email protected]>

* docs(release): reconcile final v2.4.0 stabilization PRs and boundary wording (#9162)

* docs(release): reconcile final v2.4.0 stabilization PRs and boundary wording

Signed-off-by: LoopX Agent <[email protected]>

* docs(release): align #9158 and #9161 descriptions with verified commits

Signed-off-by: LoopX Agent <[email protected]>

---------

Signed-off-by: LoopX Agent <[email protected]>

* fix(steward): tolerate minor JSON delimiter glitches and recover saved report candidates (#9164)

Signed-off-by: LoopX Agent <[email protected]>

* fix(control-plane): preserve bound yardstick reversals in native replan openings (#9165)

Signed-off-by: LoopX Agent <[email protected]>

* fix(native-chat): recover loopback relay 401 failures and isolate test auth (#9167)

Signed-off-by: LoopX Agent <[email protected]>

---------

Signed-off-by: LoopX Agent <[email protected]>
Signed-off-by: cicolord <[email protected]>
Signed-off-by: huangruiteng <[email protected]>
Signed-off-by: lusendong.6789 <[email protected]>
Signed-off-by: Lusen Dong <[email protected]>
Co-authored-by: cicolord <[email protected]>
Co-authored-by: huangruiteng <[email protected]>
Co-authored-by: Lusen Dong <[email protected]>
Co-authored-by: TRAE CLI <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant