Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -721,6 +721,22 @@ Every experiment report keeps these layers separate:
An internal control score may diagnose behavior. It must not be added to the
native task score or used to turn a failed task into a success.

### 5.5 Four-tier verification funnel

Full-horizon replicated runs (for example, `18h–20h` on long-horizon scalar
studies such as EdgeBench) are too slow and expensive to act as the first
filter for runtime regressions, uncached KV-prefill overhead, or local
proxy-to-native inversions. Every new control mechanism, runtime optimization,
and benchmark study advances sequentially through a fail-closed four-tier funnel,
calibrated to each study's declared native full budget (§3.8):

| Tier | Typical scope & horizon | Required evidence & gating metrics | Gate & non-substitution boundary |
|---|---|---|---|
| **Tier 0 · Offline invariants & calibration** | Zero live model spend: unit/regression suites, replay fixtures, `WarmStart` / evaluator `/context` preflight, container/auth/network boundary audits, and offline [`benchmark/proxy_order.py`](../../../benchmark/proxy_order.py) common-panel checks | Deterministic pass/fail contracts, exact artifact/source digest binding, zero judge leakage, and common-panel Kendall $\tau_b$ + tail-cliff characterization | Fails closed before any live token spend; passing static/replay checks proves only E0 foundation readiness (E1 and E2 still require live runner receipts and repeated campaign transitions per §11.2) |
| **Tier 1 · Fast efficiency-parity gate** | Short matched controls shorter than the study's native full budget (for example `--timeout 18000` `5h` controls on `12h–20h` tracks, or shorter proportional windows on `3h–5h` tracks) contrasting `Raw-Artifact Reset Baseline`, quiet single-session execution, and `Controller–Worker` + `EpochReset` against an anchored baseline under §5.1–§5.3 | Native `score` parity, `time-to-threshold` parity, compression of inter-turn protocol latency gaps (such as training-to-next-submission gaps), `cached_prefix_tokens` vs. `uncached_input_tokens` stability, and bounded total token/time ratio | Prunes protocol overhead or execution regressions early; short-horizon parity never certifies native full-horizon net uplift or absence of late-horizon context drift |
| **Tier 2 · Bounded counterfactual calibration** | Short matched forks within the study's horizon (for example `~1h–3h` remaining cap) from one quiescent sealed `research-checkpoint` ($s_0$) with identical pinned source, evaluator conditions (`/context`), and submission cooldowns | Decision-neighborhood Kendall $\tau_b(s; a, a_0, \pi)$, false-improvement rate over the native incumbent, proxy-cliff (`Yardstick`) detection, and conditional $\Delta J(B \mid s_0)$ charging complete marginal control/evaluation cost | Isolates late-horizon decision and proxy fidelity before launching multi-arm full-horizon suites; conditioned on $s_0$ and does not replace cold-start to terminal full-horizon comparisons |
| **Tier 3 · Full-horizon replicated promotion** | Full-budget replicated studies at each benchmark's declared native full horizon (for example `3h` on `LHTB 2.0` and `5h` on `FrontierCS-2` per §3.8, or `18h–20h` on EdgeBench; `N >= 5` where promoted), including the matched `Raw-Artifact Reset Baseline` on episodic RSI / scalar optimization tracks (§5.2), full cost accounting (§6.2), and independent integrity audits (§6.5) | Benchmark-native outcome uplift or cost-normalized Pareto superiority with uncertainty bounds on the pinned study (plus cross-family replication where claiming C3) | Tier 3 single-study native full-horizon replication supports C2, cross-family replication supports C3, and C4 remains separately gated per §4; regressions at Tier 0–2 trigger diagnosis rather than full-matrix padding |

## 6. Measurement Model

### 6.1 Native outcomes
Expand Down Expand Up @@ -1884,6 +1900,7 @@ This is a living research RFC, not a frozen benchmark snapshot.
| 2026-08-25 | Add StartupBench, LoopsBench, and WideSearch to the benchmark primer and integration plan as a second, complementary trio; require native metrics per benchmark and keep benchmark material out of reusable LoopX state. |
| 2026-10-16 | Archive [feedback-first observations](https://github.com/loopx-project/loopx/blob/1d60e7edb2a05fe048767444f872934c3adef992/deprecate/benchmark-legacy/edgebench/feedback-first-observations-20261016.md) separately from active practice. Runner source-subset and configuration-projection corrections remain E0/E1 work; mixed-profile, interrupted and running observations do not close E2/E3 or isolate a planning effect. General guard re-entry relaxation is deferred pending affected-caller validation. |
| 2026-11-15 | Codify the Bitter-Lesson boundary and five governed control-plane failure regimes (§2.2), require a matched, fully costed Raw-Artifact Reset Baseline (newly run or qualified historical reuse) for C2/C3 claims on episodic RSI / scalar optimization tracks (§5.2), and adopt the 2026-11 benchmark portfolio ranking (§3.8): **P0** (`EdgeBench`, `FrontierCS-2`, `LHTB / LHTB 2.0`, `LoopsBench`), **P1** (`SWE-Marathon`, `PaperBenchX` over `PaperBench`, `ALE` distinct from `ALE-Bench`, `WideSearch`, `KernelBench` / `AlgoTune`), **P2** (`ALE-Bench`, `StartupBench`, `MLE-bench`, `tb-science` episodic transfer samples only—excluded from RSI due to CI-locked `28800s` budget and 100% binary `reward.txt`), and **Control-Only / Deprioritized** (`DeepSWE`, `SWE-bench Pro`, `CORE-Bench`, `ScienceAgentBench`, `BixBench`, `FrontierScience`, deprecated `RE-Bench`), gating multi-arm expansion on unified `benchmark/runtime/` + Stop-hook E0–E2 readiness and Linux x86\_64 / GPU hosts. |
| 2026-11-22 | Codify the fail-closed **Four-Tier Verification Funnel** (§5.5: Tier 0 offline invariants/`benchmark/proxy_order.py` proving E0 only -> Tier 1 sub-horizon fast efficiency-parity gate -> Tier 2 quiescent checkpoint counterfactual calibration -> Tier 3 native full-budget replicated promotion such as `3h` LHTB 2.0, `5h` FrontierCS-2, or `18h–20h` EdgeBench, separating single-study `C2` from cross-family `C3`) so runtime/protocol optimizations prune regressions cheaply without weakening §4 C2–C4 claim thresholds or §11.2 readiness gates. |

## 18. References

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -568,6 +568,17 @@ goal-mode baseline。benchmark-reference scaffold reproduction 是额外 fidelit

内部 control score 可以诊断行为,但不得加到原生 task score 上,也不能把失败任务变成成功。

### 5.5 四层漏斗验证协议

完整原生全预算长跑(例如 EdgeBench 等长程标量研究的 `18h–20h` 单元)的时钟与算力成本过高,不应作为排查基础设施缺陷、未命中 KV-cache 开销或局部代理指标反序的第一道过滤器。每项新控制机制、运行时优化及 benchmark 实验均按各基准已声明的原生全预算(§3.8)顺序通过闭环阻塞的**四层漏斗验证协议**推进:

| 层级 | 典型范围与时长 | 必需证据与核心指标 | 门禁与不可替代边界 |
|---|---|---|---|
| **Tier 0 · 离线不变量与标尺校准** | 零在线模型开销:单测与回归套件、确定性重放与 checkpoint 恢复夹具、`WarmStart` 与评测器 `/context` 预检、容器/认证/网络隔离审计,以及 [`benchmark/proxy_order.py`](../../../benchmark/proxy_order.py) 离线共同面板诊断 | 确定性通过/失败契约、精确产物与源码 digest 绑定、零隐藏评测泄漏,以及共同面板 Kendall $\tau_b$ 与尾部断崖刻画 | 任一项失败即阻止实跑消耗 token;静态与重放验证仅证明 E0 基础就绪度(E1 与 E2 仍须按 §11.2 给出实 runner 实跑回执与重复转换恢复证据) |
| **Tier 1 · 快速效率对齐门** | 低于研究原生全预算的短周期匹配实验(如 `12h–20h` 长程赛道上采用 `--timeout 18000` 的 `5h` 对照,或 `3h–5h` 赛道上的等比例更短窗口),按 §5.1–§5.3 对比 `Raw-Artifact Reset Baseline`、静默单会话执行与 `Controller–Worker` + `EpochReset` 相对于锚定基线的表现 | 原生 `score` 对齐度、`time-to-threshold`、轮间协议延迟缺口压缩(如训练结束至下次提交的等待窗口)、`cached_prefix_tokens` 与 `uncached_input_tokens` 稳定性,以及总 token/耗时比上限 | 尽早拦截协议税膨胀或执行退化;短周期效率对齐不能替代原生全周期净收益或证明长程无熵增漂移 |
| **Tier 2 · 有界反事实标定** | 处于研究预算内的短窗口匹配分叉(如剩余上限 `~1h–3h`),从同一静止封存的 `research-checkpoint`($s_0$)发起,严格固定源码、评测条件(`/context`)与提交冷却 | 决策邻域 Kendall $\tau_b(s; a, a_0, \pi)$、相对原生 incumbent 的代理假提升率、前沿断崖(`Yardstick`)检出,以及计入完整边际控制/评测成本后的条件净收益 $\Delta J(B \mid s_0)$ | 在启动全周期多臂组前隔离后期决策与局部标尺保真度;结果以起点 $s_0$ 为条件,不替代从冷启动到终局的全周期对照 |
| **Tier 3 · 全周期复现与晋升实验** | 各基准已声明原生全预算下的复现实验(如 §3.8 下 `LHTB 2.0` 为 `3h`、`FrontierCS-2` 为 `5h`、EdgeBench 为 `18h–20h`;准备晋升时满足 `N >= 5`),含 episode 型 RSI / 标量赛道必配的 `Raw-Artifact Reset Baseline`(§5.2)、完整成本核算(§6.2)及独立完整性审计(§6.5) | 在锚定研究上达成含不确定性界限的 benchmark 原生收益提升或成本归一化帕累托占优(若主张 C3 则还需跨族复现) | Tier 3 单研究原生全周期复现支撑 C2,跨族复现支撑 C3,C4 产品晋升仍按 §4 单独把关;Tier 0–2 出现退化时须停止归因与重设计,严禁靠堆砌全矩阵掩盖 |

## 6. 测量模型

### 6.1 原生结果
Expand Down Expand Up @@ -1503,6 +1514,7 @@ promotion 或修改 LoopX default。
| 2026-08-25 | 把 StartupBench、LoopsBench 与 WideSearch 加入 benchmark primer 与集成计划,作为第二组互补 trio;每个 benchmark 必须在原生 metric space 报告,且 benchmark 材料不得进入可复用的 LoopX state。 |
| 2026-10-16 | 将[反馈优先观察](https://github.com/loopx-project/loopx/blob/1d60e7edb2a05fe048767444f872934c3adef992/deprecate/benchmark-legacy/edgebench/feedback-first-observations-20261016.md)归入历史报告;运行器源码子集与配置投影修复仍属 E0/E1,混合处理、中断和进行中的观察不关闭 E2/E3,也不分离单一规划效应。通用 guard 重入放宽待受影响调用方独立验证。 |
| 2026-11-15 | 确立 Bitter-Lesson 边界与五类受治理控制面失败机制(§2.2),要求 episode 型 RSI / 标量优化赛道的 C2/C3 主张纳入匹配且保留完整成本记账的 Raw-Artifact Reset Baseline(新实跑或合格历史复用,§5.2),并采纳 2026-11 Benchmark 组合优先级排序(§3.8):**P0**(`EdgeBench`、`FrontierCS-2`、`LHTB / LHTB 2.0`、`LoopsBench`)、**P1**(`SWE-Marathon`、`PaperBenchX` 优先于 `PaperBench`、`ALE` 区别于 `ALE-Bench`、`WideSearch`、`KernelBench` / `AlgoTune`)、**P2**(`ALE-Bench`、`StartupBench`、`MLE-bench`、`tb-science` 仅保留单次迁移样本——因 CI 锁定 `28800s` 预算与 100% 二值 `reward.txt` 排除出 RSI),以及 **仅作对照 / 降级**(`DeepSWE`、`SWE-bench Pro`、`CORE-Bench`、`ScienceAgentBench`、`BixBench`、`FrontierScience`、已废弃 `RE-Bench`);多臂扩展以统一 `benchmark/runtime/` + Stop-hook E0–E2 就绪及 Linux x86\_64 / GPU 宿主机为前置门槛。 |
| 2026-11-22 | 确立闭环阻塞的**四层漏斗验证协议**(§5.5:Tier 0 离线不变量与 `benchmark/proxy_order.py` 校准仅证 E0 -> Tier 1 低于原生全预算的快速效率对齐门 -> Tier 2 静止检查点反事实标定 -> Tier 3 各基准原生全预算复现晋升,如 LHTB 2.0 `3h`、FrontierCS-2 `5h`、EdgeBench `18h–20h`,区分单研究 `C2` 与跨族 `C3`),在不降低 §4 C2–C4 主张门槛与 §11.2 就绪门禁的前提下提前剪枝运行时与协议退化。 |

## 18. 参考资料

Expand Down
38 changes: 38 additions & 0 deletions docs/architecture/rfcs/loopx-overall-roadmap-v0.md
Original file line number Diff line number Diff line change
Expand Up @@ -164,6 +164,44 @@ moats** outside a single model forward pass:
delivery) so sandbox resets or failures never corrupt control authority or
expose host credentials.

### Harness-neutral kernel and three-level harness co-design

LoopX keeps its core authority and decision substrate (`AuthorityStore`, `CAS`,
`claim/lease`, `Turn` fences, once-only `receipt` settlement, and
`Target/Yardstick/Stride` records) **harness-neutral**: the kernel must never
embed a single vendor CLI's prompt syntax, private loop driver, or transcript
layout. Because attached and managed hosts expose very different control
seams, LoopX pairs that single kernel with a **three-level harness co-design
surface** rather than forcing one generic wrapper across every environment:

| Co-design level | Target host surface | Primary optimization & boundary mechanism | Source ownership & convergence direction |
|---|---|---|---|
| **Level 1 · `BoundaryCLI`** | Boundary-isolated external CLI hosts (`codex-cli`, `claude-code`, `opencode`, `pi`, `zcode`, `kiro-cli`, `kunluncode`, skill-facade CLIs) where LoopX observes only turn boundaries | Keeps routine uncontested steps zero-ceremony (**`Quiet-Turn`**, zero per-Turn schema tax) and preserves native `KeepRaw` prefix KV-cache stability within a phase, deferring prefix rebasing and compact delta/snapshot resets to explicit quiescent **`EpochReset`** boundaries | [`loopx/host_loop_activation.py`](../../../loopx/host_loop_activation.py), [`loopx/hosts/skill_facade.py`](../../../loopx/hosts/skill_facade.py), `context_delivery.ts` ([History working sets](history-working-set-context-policy-v0.md#51-one-projection-owner)) |
| **Level 2 · `OrchestratedApp`** | Multi-unit app, IDE, IM, and `Controller–Worker` orchestrators coordinating concurrent lanes and handoffs | Enforces **`Subagent Scope`** isolation (`collaboration_brief_v0`, bounded `remaining_work`, per-worker workspace/lease fences, and async wake predicates) so long worker execution neither blocks controller/user interaction nor leaks parent prompt gravity across slices | [`controller-executor-unified-contract-v0.md`](controller-executor-unified-contract-v0.md#router-controller-worker), [`capable-manager-semantic-handoff-v0.md`](capable-manager-semantic-handoff-v0.md) |
| **Level 3 · `InTree`** | Runtime-embedded harnesses running inside or directly wired to LoopX's process transport (such as `loopx turn run-once --host dsh` for DeepSeek Harness) | Pairs receipt-gated `FoldToReceipt` / `EpochReset` queue compaction with deterministic turn-start context/artifact prefetch (**`FoldAndPrefetch`**) and typed failure-class recovery (`loopx_turn_host_failure_v0`) inside one process boundary | [`loopx/dsh_goal_mode/turn_host_adapter.py`](../../../loopx/dsh_goal_mode/turn_host_adapter.py), [`harness-selection-dsh-pi-v0.md`](harness-selection-dsh-pi-v0.md) |

To prevent host divergence from duplicating control-plane logic across
top-level packages:

- **Unified `HostHarnessCapabilities` descriptor:** every host binding declares
a single typed `HostHarnessCapabilities` contract covering loop-driver
ownership, native goal/automation/cron/subagent-scope primitives, `EpochReset`
versus `FoldAndPrefetch` support, stdio/encoding requirements, and stop/pause
fence fidelity—never claiming autonomous execution or budget enforcement a host
cannot verify.
- **Consolidating `loopx/*_goal_mode/` into `loopx/hosts/`:** migrate the `9`
remaining top-level `loopx/*_goal_mode/` packages (`agy_goal_mode`,
`claude_goal_mode`, `dsh_goal_mode`, `kiro_cli_goal_mode`,
`kunluncode_goal_mode`, `opencode2_goal_mode`, `opencode_goal_mode`,
`pi_goal_mode`, `zcode_goal_mode`) into `loopx/hosts/` under shared
`HostHarnessCapabilities` projections per
[the unified contract's retirement gate](controller-executor-unified-contract-v0.md#3-normative-retirement-backlog-and-ratchet-rules)
and [`top_level_module_budget.json`](../../../tests/architecture/top_level_module_budget.json),
keeping CLI compatibility aliases strictly until caller cutover completes.
- **Funnel-gated validation:** co-design optimizations across `BoundaryCLI`,
`OrchestratedApp`, and `InTree` remain architecture targets until verified
through the [Four-Tier Verification Funnel](long-horizon-harness-benchmark-research-program-v0.md#55-four-tier-verification-funnel).

### App-first conversation delivery

The installed App is the primary interaction and qualification surface; Lark
Expand Down
Loading