diff --git a/docs/architecture/rfcs/long-horizon-harness-benchmark-research-program-v0.md b/docs/architecture/rfcs/long-horizon-harness-benchmark-research-program-v0.md index 2703ae9548..bb41873b00 100644 --- a/docs/architecture/rfcs/long-horizon-harness-benchmark-research-program-v0.md +++ b/docs/architecture/rfcs/long-horizon-harness-benchmark-research-program-v0.md @@ -721,6 +721,22 @@ Every experiment report keeps these layers separate: An internal control score may diagnose behavior. It must not be added to the native task score or used to turn a failed task into a success. +### 5.5 Four-tier verification funnel + +Full-horizon replicated runs (for example, `18h–20h` on long-horizon scalar +studies such as EdgeBench) are too slow and expensive to act as the first +filter for runtime regressions, uncached KV-prefill overhead, or local +proxy-to-native inversions. Every new control mechanism, runtime optimization, +and benchmark study advances sequentially through a fail-closed four-tier funnel, +calibrated to each study's declared native full budget (§3.8): + +| Tier | Typical scope & horizon | Required evidence & gating metrics | Gate & non-substitution boundary | +|---|---|---|---| +| **Tier 0 · Offline invariants & calibration** | Zero live model spend: unit/regression suites, replay fixtures, `WarmStart` / evaluator `/context` preflight, container/auth/network boundary audits, and offline [`benchmark/proxy_order.py`](../../../benchmark/proxy_order.py) common-panel checks | Deterministic pass/fail contracts, exact artifact/source digest binding, zero judge leakage, and common-panel Kendall $\tau_b$ + tail-cliff characterization | Fails closed before any live token spend; passing static/replay checks proves only E0 foundation readiness (E1 and E2 still require live runner receipts and repeated campaign transitions per §11.2) | +| **Tier 1 · Fast efficiency-parity gate** | Short matched controls shorter than the study's native full budget (for example `--timeout 18000` `5h` controls on `12h–20h` tracks, or shorter proportional windows on `3h–5h` tracks) contrasting `Raw-Artifact Reset Baseline`, quiet single-session execution, and `Controller–Worker` + `EpochReset` against an anchored baseline under §5.1–§5.3 | Native `score` parity, `time-to-threshold` parity, compression of inter-turn protocol latency gaps (such as training-to-next-submission gaps), `cached_prefix_tokens` vs. `uncached_input_tokens` stability, and bounded total token/time ratio | Prunes protocol overhead or execution regressions early; short-horizon parity never certifies native full-horizon net uplift or absence of late-horizon context drift | +| **Tier 2 · Bounded counterfactual calibration** | Short matched forks within the study's horizon (for example `~1h–3h` remaining cap) from one quiescent sealed `research-checkpoint` ($s_0$) with identical pinned source, evaluator conditions (`/context`), and submission cooldowns | Decision-neighborhood Kendall $\tau_b(s; a, a_0, \pi)$, false-improvement rate over the native incumbent, proxy-cliff (`Yardstick`) detection, and conditional $\Delta J(B \mid s_0)$ charging complete marginal control/evaluation cost | Isolates late-horizon decision and proxy fidelity before launching multi-arm full-horizon suites; conditioned on $s_0$ and does not replace cold-start to terminal full-horizon comparisons | +| **Tier 3 · Full-horizon replicated promotion** | Full-budget replicated studies at each benchmark's declared native full horizon (for example `3h` on `LHTB 2.0` and `5h` on `FrontierCS-2` per §3.8, or `18h–20h` on EdgeBench; `N >= 5` where promoted), including the matched `Raw-Artifact Reset Baseline` on episodic RSI / scalar optimization tracks (§5.2), full cost accounting (§6.2), and independent integrity audits (§6.5) | Benchmark-native outcome uplift or cost-normalized Pareto superiority with uncertainty bounds on the pinned study (plus cross-family replication where claiming C3) | Tier 3 single-study native full-horizon replication supports C2, cross-family replication supports C3, and C4 remains separately gated per §4; regressions at Tier 0–2 trigger diagnosis rather than full-matrix padding | + ## 6. Measurement Model ### 6.1 Native outcomes @@ -1884,6 +1900,7 @@ This is a living research RFC, not a frozen benchmark snapshot. | 2026-08-25 | Add StartupBench, LoopsBench, and WideSearch to the benchmark primer and integration plan as a second, complementary trio; require native metrics per benchmark and keep benchmark material out of reusable LoopX state. | | 2026-10-16 | Archive [feedback-first observations](https://github.com/loopx-project/loopx/blob/1d60e7edb2a05fe048767444f872934c3adef992/deprecate/benchmark-legacy/edgebench/feedback-first-observations-20261016.md) separately from active practice. Runner source-subset and configuration-projection corrections remain E0/E1 work; mixed-profile, interrupted and running observations do not close E2/E3 or isolate a planning effect. General guard re-entry relaxation is deferred pending affected-caller validation. | | 2026-11-15 | Codify the Bitter-Lesson boundary and five governed control-plane failure regimes (§2.2), require a matched, fully costed Raw-Artifact Reset Baseline (newly run or qualified historical reuse) for C2/C3 claims on episodic RSI / scalar optimization tracks (§5.2), and adopt the 2026-11 benchmark portfolio ranking (§3.8): **P0** (`EdgeBench`, `FrontierCS-2`, `LHTB / LHTB 2.0`, `LoopsBench`), **P1** (`SWE-Marathon`, `PaperBenchX` over `PaperBench`, `ALE` distinct from `ALE-Bench`, `WideSearch`, `KernelBench` / `AlgoTune`), **P2** (`ALE-Bench`, `StartupBench`, `MLE-bench`, `tb-science` episodic transfer samples only—excluded from RSI due to CI-locked `28800s` budget and 100% binary `reward.txt`), and **Control-Only / Deprioritized** (`DeepSWE`, `SWE-bench Pro`, `CORE-Bench`, `ScienceAgentBench`, `BixBench`, `FrontierScience`, deprecated `RE-Bench`), gating multi-arm expansion on unified `benchmark/runtime/` + Stop-hook E0–E2 readiness and Linux x86\_64 / GPU hosts. | +| 2026-11-22 | Codify the fail-closed **Four-Tier Verification Funnel** (§5.5: Tier 0 offline invariants/`benchmark/proxy_order.py` proving E0 only -> Tier 1 sub-horizon fast efficiency-parity gate -> Tier 2 quiescent checkpoint counterfactual calibration -> Tier 3 native full-budget replicated promotion such as `3h` LHTB 2.0, `5h` FrontierCS-2, or `18h–20h` EdgeBench, separating single-study `C2` from cross-family `C3`) so runtime/protocol optimizations prune regressions cheaply without weakening §4 C2–C4 claim thresholds or §11.2 readiness gates. | ## 18. References diff --git a/docs/architecture/rfcs/long-horizon-harness-benchmark-research-program-v0.zh-CN.md b/docs/architecture/rfcs/long-horizon-harness-benchmark-research-program-v0.zh-CN.md index e979e01299..69482709ab 100644 --- a/docs/architecture/rfcs/long-horizon-harness-benchmark-research-program-v0.zh-CN.md +++ b/docs/architecture/rfcs/long-horizon-harness-benchmark-research-program-v0.zh-CN.md @@ -568,6 +568,17 @@ goal-mode baseline。benchmark-reference scaffold reproduction 是额外 fidelit 内部 control score 可以诊断行为,但不得加到原生 task score 上,也不能把失败任务变成成功。 +### 5.5 四层漏斗验证协议 + +完整原生全预算长跑(例如 EdgeBench 等长程标量研究的 `18h–20h` 单元)的时钟与算力成本过高,不应作为排查基础设施缺陷、未命中 KV-cache 开销或局部代理指标反序的第一道过滤器。每项新控制机制、运行时优化及 benchmark 实验均按各基准已声明的原生全预算(§3.8)顺序通过闭环阻塞的**四层漏斗验证协议**推进: + +| 层级 | 典型范围与时长 | 必需证据与核心指标 | 门禁与不可替代边界 | +|---|---|---|---| +| **Tier 0 · 离线不变量与标尺校准** | 零在线模型开销:单测与回归套件、确定性重放与 checkpoint 恢复夹具、`WarmStart` 与评测器 `/context` 预检、容器/认证/网络隔离审计,以及 [`benchmark/proxy_order.py`](../../../benchmark/proxy_order.py) 离线共同面板诊断 | 确定性通过/失败契约、精确产物与源码 digest 绑定、零隐藏评测泄漏,以及共同面板 Kendall $\tau_b$ 与尾部断崖刻画 | 任一项失败即阻止实跑消耗 token;静态与重放验证仅证明 E0 基础就绪度(E1 与 E2 仍须按 §11.2 给出实 runner 实跑回执与重复转换恢复证据) | +| **Tier 1 · 快速效率对齐门** | 低于研究原生全预算的短周期匹配实验(如 `12h–20h` 长程赛道上采用 `--timeout 18000` 的 `5h` 对照,或 `3h–5h` 赛道上的等比例更短窗口),按 §5.1–§5.3 对比 `Raw-Artifact Reset Baseline`、静默单会话执行与 `Controller–Worker` + `EpochReset` 相对于锚定基线的表现 | 原生 `score` 对齐度、`time-to-threshold`、轮间协议延迟缺口压缩(如训练结束至下次提交的等待窗口)、`cached_prefix_tokens` 与 `uncached_input_tokens` 稳定性,以及总 token/耗时比上限 | 尽早拦截协议税膨胀或执行退化;短周期效率对齐不能替代原生全周期净收益或证明长程无熵增漂移 | +| **Tier 2 · 有界反事实标定** | 处于研究预算内的短窗口匹配分叉(如剩余上限 `~1h–3h`),从同一静止封存的 `research-checkpoint`($s_0$)发起,严格固定源码、评测条件(`/context`)与提交冷却 | 决策邻域 Kendall $\tau_b(s; a, a_0, \pi)$、相对原生 incumbent 的代理假提升率、前沿断崖(`Yardstick`)检出,以及计入完整边际控制/评测成本后的条件净收益 $\Delta J(B \mid s_0)$ | 在启动全周期多臂组前隔离后期决策与局部标尺保真度;结果以起点 $s_0$ 为条件,不替代从冷启动到终局的全周期对照 | +| **Tier 3 · 全周期复现与晋升实验** | 各基准已声明原生全预算下的复现实验(如 §3.8 下 `LHTB 2.0` 为 `3h`、`FrontierCS-2` 为 `5h`、EdgeBench 为 `18h–20h`;准备晋升时满足 `N >= 5`),含 episode 型 RSI / 标量赛道必配的 `Raw-Artifact Reset Baseline`(§5.2)、完整成本核算(§6.2)及独立完整性审计(§6.5) | 在锚定研究上达成含不确定性界限的 benchmark 原生收益提升或成本归一化帕累托占优(若主张 C3 则还需跨族复现) | Tier 3 单研究原生全周期复现支撑 C2,跨族复现支撑 C3,C4 产品晋升仍按 §4 单独把关;Tier 0–2 出现退化时须停止归因与重设计,严禁靠堆砌全矩阵掩盖 | + ## 6. 测量模型 ### 6.1 原生结果 @@ -1503,6 +1514,7 @@ promotion 或修改 LoopX default。 | 2026-08-25 | 把 StartupBench、LoopsBench 与 WideSearch 加入 benchmark primer 与集成计划,作为第二组互补 trio;每个 benchmark 必须在原生 metric space 报告,且 benchmark 材料不得进入可复用的 LoopX state。 | | 2026-10-16 | 将[反馈优先观察](https://github.com/loopx-project/loopx/blob/1d60e7edb2a05fe048767444f872934c3adef992/deprecate/benchmark-legacy/edgebench/feedback-first-observations-20261016.md)归入历史报告;运行器源码子集与配置投影修复仍属 E0/E1,混合处理、中断和进行中的观察不关闭 E2/E3,也不分离单一规划效应。通用 guard 重入放宽待受影响调用方独立验证。 | | 2026-11-15 | 确立 Bitter-Lesson 边界与五类受治理控制面失败机制(§2.2),要求 episode 型 RSI / 标量优化赛道的 C2/C3 主张纳入匹配且保留完整成本记账的 Raw-Artifact Reset Baseline(新实跑或合格历史复用,§5.2),并采纳 2026-11 Benchmark 组合优先级排序(§3.8):**P0**(`EdgeBench`、`FrontierCS-2`、`LHTB / LHTB 2.0`、`LoopsBench`)、**P1**(`SWE-Marathon`、`PaperBenchX` 优先于 `PaperBench`、`ALE` 区别于 `ALE-Bench`、`WideSearch`、`KernelBench` / `AlgoTune`)、**P2**(`ALE-Bench`、`StartupBench`、`MLE-bench`、`tb-science` 仅保留单次迁移样本——因 CI 锁定 `28800s` 预算与 100% 二值 `reward.txt` 排除出 RSI),以及 **仅作对照 / 降级**(`DeepSWE`、`SWE-bench Pro`、`CORE-Bench`、`ScienceAgentBench`、`BixBench`、`FrontierScience`、已废弃 `RE-Bench`);多臂扩展以统一 `benchmark/runtime/` + Stop-hook E0–E2 就绪及 Linux x86\_64 / GPU 宿主机为前置门槛。 | +| 2026-11-22 | 确立闭环阻塞的**四层漏斗验证协议**(§5.5:Tier 0 离线不变量与 `benchmark/proxy_order.py` 校准仅证 E0 -> Tier 1 低于原生全预算的快速效率对齐门 -> Tier 2 静止检查点反事实标定 -> Tier 3 各基准原生全预算复现晋升,如 LHTB 2.0 `3h`、FrontierCS-2 `5h`、EdgeBench `18h–20h`,区分单研究 `C2` 与跨族 `C3`),在不降低 §4 C2–C4 主张门槛与 §11.2 就绪门禁的前提下提前剪枝运行时与协议退化。 | ## 18. 参考资料 diff --git a/docs/architecture/rfcs/loopx-overall-roadmap-v0.md b/docs/architecture/rfcs/loopx-overall-roadmap-v0.md index 3d7bdde2f8..7c7d1c33a6 100644 --- a/docs/architecture/rfcs/loopx-overall-roadmap-v0.md +++ b/docs/architecture/rfcs/loopx-overall-roadmap-v0.md @@ -164,6 +164,44 @@ moats** outside a single model forward pass: delivery) so sandbox resets or failures never corrupt control authority or expose host credentials. +### Harness-neutral kernel and three-level harness co-design + +LoopX keeps its core authority and decision substrate (`AuthorityStore`, `CAS`, +`claim/lease`, `Turn` fences, once-only `receipt` settlement, and +`Target/Yardstick/Stride` records) **harness-neutral**: the kernel must never +embed a single vendor CLI's prompt syntax, private loop driver, or transcript +layout. Because attached and managed hosts expose very different control +seams, LoopX pairs that single kernel with a **three-level harness co-design +surface** rather than forcing one generic wrapper across every environment: + +| Co-design level | Target host surface | Primary optimization & boundary mechanism | Source ownership & convergence direction | +|---|---|---|---| +| **Level 1 · `BoundaryCLI`** | Boundary-isolated external CLI hosts (`codex-cli`, `claude-code`, `opencode`, `pi`, `zcode`, `kiro-cli`, `kunluncode`, skill-facade CLIs) where LoopX observes only turn boundaries | Keeps routine uncontested steps zero-ceremony (**`Quiet-Turn`**, zero per-Turn schema tax) and preserves native `KeepRaw` prefix KV-cache stability within a phase, deferring prefix rebasing and compact delta/snapshot resets to explicit quiescent **`EpochReset`** boundaries | [`loopx/host_loop_activation.py`](../../../loopx/host_loop_activation.py), [`loopx/hosts/skill_facade.py`](../../../loopx/hosts/skill_facade.py), `context_delivery.ts` ([History working sets](history-working-set-context-policy-v0.md#51-one-projection-owner)) | +| **Level 2 · `OrchestratedApp`** | Multi-unit app, IDE, IM, and `Controller–Worker` orchestrators coordinating concurrent lanes and handoffs | Enforces **`Subagent Scope`** isolation (`collaboration_brief_v0`, bounded `remaining_work`, per-worker workspace/lease fences, and async wake predicates) so long worker execution neither blocks controller/user interaction nor leaks parent prompt gravity across slices | [`controller-executor-unified-contract-v0.md`](controller-executor-unified-contract-v0.md#router-controller-worker), [`capable-manager-semantic-handoff-v0.md`](capable-manager-semantic-handoff-v0.md) | +| **Level 3 · `InTree`** | Runtime-embedded harnesses running inside or directly wired to LoopX's process transport (such as `loopx turn run-once --host dsh` for DeepSeek Harness) | Pairs receipt-gated `FoldToReceipt` / `EpochReset` queue compaction with deterministic turn-start context/artifact prefetch (**`FoldAndPrefetch`**) and typed failure-class recovery (`loopx_turn_host_failure_v0`) inside one process boundary | [`loopx/dsh_goal_mode/turn_host_adapter.py`](../../../loopx/dsh_goal_mode/turn_host_adapter.py), [`harness-selection-dsh-pi-v0.md`](harness-selection-dsh-pi-v0.md) | + +To prevent host divergence from duplicating control-plane logic across +top-level packages: + +- **Unified `HostHarnessCapabilities` descriptor:** every host binding declares + a single typed `HostHarnessCapabilities` contract covering loop-driver + ownership, native goal/automation/cron/subagent-scope primitives, `EpochReset` + versus `FoldAndPrefetch` support, stdio/encoding requirements, and stop/pause + fence fidelity—never claiming autonomous execution or budget enforcement a host + cannot verify. +- **Consolidating `loopx/*_goal_mode/` into `loopx/hosts/`:** migrate the `9` + remaining top-level `loopx/*_goal_mode/` packages (`agy_goal_mode`, + `claude_goal_mode`, `dsh_goal_mode`, `kiro_cli_goal_mode`, + `kunluncode_goal_mode`, `opencode2_goal_mode`, `opencode_goal_mode`, + `pi_goal_mode`, `zcode_goal_mode`) into `loopx/hosts/` under shared + `HostHarnessCapabilities` projections per + [the unified contract's retirement gate](controller-executor-unified-contract-v0.md#3-normative-retirement-backlog-and-ratchet-rules) + and [`top_level_module_budget.json`](../../../tests/architecture/top_level_module_budget.json), + keeping CLI compatibility aliases strictly until caller cutover completes. +- **Funnel-gated validation:** co-design optimizations across `BoundaryCLI`, + `OrchestratedApp`, and `InTree` remain architecture targets until verified + through the [Four-Tier Verification Funnel](long-horizon-harness-benchmark-research-program-v0.md#55-four-tier-verification-funnel). + ### App-first conversation delivery The installed App is the primary interaction and qualification surface; Lark diff --git a/docs/architecture/rfcs/loopx-overall-roadmap-v0.zh-CN.md b/docs/architecture/rfcs/loopx-overall-roadmap-v0.zh-CN.md index 846df20ecb..e9c23de1f5 100644 --- a/docs/architecture/rfcs/loopx-overall-roadmap-v0.zh-CN.md +++ b/docs/architecture/rfcs/loopx-overall-roadmap-v0.zh-CN.md @@ -122,6 +122,36 @@ LoopX 内部保持严格、唯一 owner 的类型化内核(`AuthorityStore`、 一次性 worker 沙盒及 worktree(`--worker turn-resume`、隔离容器、归一化补丁交付)严格分离, 确保沙盒重置或故障不会破坏控制权威或暴露宿主凭据。 +### 与 Harness 无关的内核与三层协同设计 + +LoopX 的权威与决策基座(`AuthorityStore`、`CAS`、`claim/lease`、`Turn` 围栏、 +仅一次幂等 `receipt` 结算与 `Target/Yardstick/Stride` 记录)保持**与 Harness 无关**: +内核绝不把单一厂商 CLI 的提示词语法、私有循环驱动或会话文件布局写死在核心规则里。 +同时,不同宿主向 LoopX 暴露的控制接缝差异很大,因此 LoopX 以**三层 Harness 协同设计** +加上统一能力描述符承接宿主差异,而不是强行套用同一种重型包装器: + +| 协同层级 | 适用宿主表面 | 核心优化与边界机制 | 源码归属与收敛方向 | +|---|---|---|---| +| **Level 1 · `BoundaryCLI`** | 进程边界隔离的外部 CLI 宿主(`codex-cli`、`claude-code`、`opencode`、`pi`、`zcode`、`kiro-cli`、`kunluncode` 及 skill-facade CLI),LoopX 仅在 Turn 边界接入 | 保持无竞争日常轮次零仪式感(**`Quiet-Turn`**,零每 Turn 必填表单税),在阶段内保持原生 `KeepRaw` 前缀 KV-cache 稳定,并把前缀重定基与精简增量/快照重置推迟到显式静止的 **`EpochReset`** 边界 | [`loopx/host_loop_activation.py`](../../../loopx/host_loop_activation.py)、[`loopx/hosts/skill_facade.py`](../../../loopx/hosts/skill_facade.py)、`context_delivery.ts`([历史工作集策略](history-working-set-context-policy-v0.zh-CN.md#51-单一投影-owner)) | +| **Level 2 · `OrchestratedApp`** | 协调并发 lane 与交接的多单元 App、IDE、IM 及 `Controller–Worker` 编排器 | 强化 **`Subagent Scope`** 隔离(`collaboration_brief_v0`、有界 `remaining_work` 投影、按 worker 隔离的 workspace/lease 围栏与异步唤醒谓词),使长程 worker 执行不阻塞 controller 与用户交互,也不跨切片泄漏父上下文重力 | [`controller-executor-unified-contract-v0.zh-CN.md`](controller-executor-unified-contract-v0.zh-CN.md#router-controller-worker)、[`capable-manager-semantic-handoff-v0.zh-CN.md`](capable-manager-semantic-handoff-v0.zh-CN.md) | +| **Level 3 · `InTree`** | 运行在 LoopX 进程内或直连进程传输的原生协同宿主(如 DeepSeek Harness 的 `loopx turn run-once --host dsh`) | 把受回执门控的 `FoldToReceipt` / `EpochReset` 队列压缩与回合启动时的确定性上下文/产物预取(**`FoldAndPrefetch`**)及类型化故障恢复(`loopx_turn_host_failure_v0`)融合在同一受控生命周期内 | [`loopx/dsh_goal_mode/turn_host_adapter.py`](../../../loopx/dsh_goal_mode/turn_host_adapter.py)、[`harness-selection-dsh-pi-v0.zh-CN.md`](harness-selection-dsh-pi-v0.zh-CN.md) | + +为避免各宿主适配器在顶层包中重复复制控制面逻辑: + +- **统一 `HostHarnessCapabilities` 描述符:** 每个宿主绑定通过单一类型化的 + `HostHarnessCapabilities` 契约声明事实能力(循环驱动归属、原生 goal/automation/cron/subagent-scope + 原语、支持 `EpochReset` 还是 `FoldAndPrefetch`、stdio/编码边界以及 stop/pause + 围栏保真度),绝不伪造宿主无法执行的自治承诺或预算强约束。 +- **把 `loopx/*_goal_mode/` 收敛至 `loopx/hosts/`:** 按 + [统一契约的退役门禁](controller-executor-unified-contract-v0.zh-CN.md#3-具名退役清单与棘轮规则)与 + [`top_level_module_budget.json`](../../../tests/architecture/top_level_module_budget.json),将现存 + `9` 个顶层 `loopx/*_goal_mode/` 包(`agy_goal_mode`、`claude_goal_mode`、`dsh_goal_mode`、 + `kiro_cli_goal_mode`、`kunluncode_goal_mode`、`opencode2_goal_mode`、`opencode_goal_mode`、 + `pi_goal_mode`、`zcode_goal_mode`)迁入 `loopx/hosts/` 并收敛至共享 `HostHarnessCapabilities` + 投影,仅在调用方切换完成前保留 CLI 兼容别名。 +- **漏斗验证门禁:** `BoundaryCLI`、`OrchestratedApp` 与 `InTree` 的协同优化(`Quiet-Turn`、 + `EpochReset`、`Subagent Scope`、`FoldAndPrefetch`)在通过[四层漏斗验证协议](long-horizon-harness-benchmark-research-program-v0.zh-CN.md#55-四层漏斗验证协议)前均为架构与设计目标,不直接等同已测收益或默认切换。 + ### 总纲与现有文档的关系 - 本页是跨产品、工程、研究和采用的路线总纲;第 2 节定义 S1–S13 工作流,第 3 节定义 G0–G5 组合里程碑,第 4 节映射全部 RFC,第 5–6 节细化多 Agent 核心路径,第 8 节保存可复核审计。