Repository navigation
feat(benchmark): add Kendall tau-b and frontier cliff diagnostics to proxy_order - #9177
loopx-agent wants to merge 1 commit into
Conversation
…proxy_order Signed-off-by: LoopX Agent <[email protected]>
loopx-agent
left a comment
There was a problem hiding this comment.
Reviewer: model_agent · Gemini · Google · self_reported
Approval conclusion (author-owned PR; GitHub blocks formal self-approval)
本次对 exact head 9d4872bf83aa9932d5530498bf7cc149c86cc352(基于 64353773dc8b1ca82ff531cc7653dec6c3d41c0b)完成了独立代码审阅与本地验证:在保留原有全部输出键与 Decimal 比较精度的前提下,benchmark/proxy_order.py 补齐了 tie-adjusted Kendall tau-b 与顶端前沿反例诊断,有效覆盖了“全局成对一致率与相关系数仍为正、但顶端候选出现 proxy cliff 回归”的盲区。
动机 / Motivation
离线评估本地 proxy(代理指标)排序一致性的 benchmark 研究人员与诊断脚本调用方,在运行 python -m benchmark.proxy_order 或调用 benchmark.proxy_order.compare() 对比固定 cohort 的本地分数与官方 native 分数时遇到该问题。
改前当低分段候选保持单调一致、仅顶端出现高分代理悬崖(如超时崩溃或 reward hack 使最高 proxy 候选的 native 分数回退)时,脚本仅输出成对一致/反转计数,研究者会看到 agrees > reverses 甚至正相关而漏判顶端失真;改后同一组输入同时输出 tie-adjusted Kendall tau-b 与 frontier 摘要,能立刻暴露最高 proxy 候选未命中 native 最优且存在非最优候选在 proxy 上严格超越 native 最优的假改进。
在保留 cohort、coverage、native_tied_pairs 与 rules 全部原有字段和 Decimal 比较精度的前提下,compare() 与 CLI JSON 输出新增 kendall_tau_b 与 frontier(含 best_native_count、best_proxy_matches_best_native、false_improvements_over_best_native)两个诊断字段。 本 PR 仅扩展离线面板比较脚本 benchmark/proxy_order.py 及其单测和参考文档,不修改控制面 online yardstick 判定逻辑(yardstick_decision.ts)、不触发真实模型实验,也不新增持续运行守护进程或持久化存储。
需要说明的边界缺口:benchmark/proxy_order.py 仍属按指定 cohort 离线分析的辅助脚本,尚不自动接入 online search_comparison 运行时门控;在线运行时针对单个 candidate pair 的 proxy_order_reversed 护栏由既有控制面路径独立负责。若不做此改动,离线研究在只看 agrees / reverses 时容易被低分段单调性掩盖顶端代理失效风险;而在现有 benchmark/proxy_order.py 内直接增补这两个汇总指标是消除该盲区的最小闭环方案。
改动思路 / Approach
本次设计沿着既有 compare() 数据流就地扩展,避免引入外部统计库依赖或重构无关模块。在成对遍历阶段同时记录所有代理平局对数 proxy_ties(包含“代理与原生同时平局”的联合平局对),再结合既有的 native_ties 计算精确 Kendall tau-b;在候选匹配完成后,通过纯函数 _frontier_summary() 计算最优原生集合与顶端假改进计数。
matched ──┬──> pairwise loop (combinations) ──> rules + proxy_ties + native_ties ──> kendall_tau_b
└──> _frontier_summary() ──> best_native_indices / best_proxy_indices ──> frontier
以上结构示意基于 exact head 9d4872bf83aa9932d5530498bf7cc149c86cc352 源码整理,仅用于说明数据流向,不替代实际运行证据。在架构边界判断上,本次改动属于 benchmark 离线分析脚本的局部计算与投影增量,复用既有的 Decimal/int 面板加载、方向比较与成对循环,通过独立无副作用的 _frontier_summary() 隔离顶端悬崖统计逻辑,不新增控制面状态、持久化契约或运行时策略。 当前边界明确收口为:仅限 benchmark/proxy_order.py、benchmark/tests/test_proxy_order.py 与 docs/reference/benchmark-behavior-findings.md 的离线诊断输出,不跨越到 online 控制面决策器。
具体改动 / Changes
针对规格文档 docs/architecture/rfcs/long-horizon-harness-benchmark-research-program-v0.md#22-empirical-failure-regimes(固定基线 64353773dc8b1ca82ff531cc7653dec6c3d41c0b)中第 2.2 节第 4 条(Proxy misalignment and tail TLE cliffs)的逐项核对如下:
RFC-2.2-4-KENDALL(implemented):在benchmark/proxy_order.py的compare()中精确计算全匹配对的 Kendall tau-b。RFC-2.2-4-FRONTIER(implemented):新增_frontier_summary()输出顶端最优对齐与非原生最优假改进数量。RFC-2.2-4-CLI-DOC(implemented):在benchmark/tests/test_proxy_order.py增加 CLI 子进程及边界测试,并在docs/reference/benchmark-behavior-findings.md同步契约说明。
关键代码讲解
_frontier_summary()(benchmark/proxy_order.py:64-116):当matched为空时返回默认安全结构(best_native_count=0、best_proxy_matches_best_native=None、false_improvements_over_best_native=0)。非空时按native_direction与proxy_direction分别求取精确best_native_score和best_proxy_score,并用enumerate(matched)下标集合区分不同候选,防止相同分数元组混叠。判断“非原生最优候选是否在 proxy 上严格超越至少一个原生最优候选”时,等价地将其归约为与worst_best_native_proxy的单次order(..., proxy_direction) > 0比较,兼顾可读性与O(m)复杂度。compare()(benchmark/proxy_order.py:119-194):先在主循环中计算proxy_decision = order(left.proxy, right.proxy, proxy_direction)并先于if actual == 0:累加proxy_ties,确保联合平局进入 tau-b 分母(N - T_p) * (N - T_n),同时保持rules["proxy"]["abstains"]原有仅统计原生非平局对的语义不变。分母不大于 0(len(matched) < 2或任一侧全平局)时返回None,避免除零且兼容 JSON 序列化。- 测试与文档(
benchmark/tests/test_proxy_order.py:111-294、docs/reference/benchmark-behavior-findings.md:109-123):保留全部 10 个既有测试用例,新增单调一致/完全反转、代理平局与联合平局、退化输入、5 候选顶端悬崖反例(agrees=6, reverses=4, kendall_tau_b=0.2, best_proxy_matches_best_native=False, false_improvements_over_best_native=1)、多原生最优点与平局边界以及python -m benchmark.proxy_order子进程 JSON 端到端冒烟验证。
对主干的风险 / Risks
本次改动风险处于可控范围,已针对潜在失效模式做逆向核验:
- 联合平局误漏算风险:若将
proxy_ties简单等同于rules["proxy"]["abstains"],在存在(proxy, native)联合平局时分母N - T_p会偏大导致 tau-b 偏小;目前实现先计入proxy_ties再过滤actual == 0,并由test_kendall_tau_b_with_proxy_only_and_joint_ties明确验证abstains == 0且kendall_tau_b == 1.0。 - 顶端平局误报假改进风险:非原生最优候选若仅在 proxy 上与某原生最优候选打平(
order == 0),不应计为严格假改进;_frontier_summary()使用order(...) > 0严格大于判定,且已在单测中覆盖。 - 生产调用方与兼容性:所有新增字段为顶层追加键(
kendall_tau_b、frontier),既有的cohort、coverage、native_tied_pairs、rules结构及Decimal亚浮点精度逻辑均未破坏。离线脚本不涉及控制面default_off门控或运行时 actor 权限扩展,文档与代码中的词表均为局部 JSON 输出键,未修改共享协议词表。 - 本地验证摘要:
pytest benchmark/tests/test_proxy_order.py(19 passed — PASS)、pytest tests/control_plane/test_work_yardstick_cli.py(50 passed — PASS)、ruff check(PASS)、mypy(17 files — PASS)、check_rfc_bilingual.py(PASS)、loopx check公开边界扫描(3 files clean — PASS)以及loopx canary premerge(direct+compile 5/5 PASS,catalog 6/6 PASS,risk_profile 8/8 PASS,boundary 1/1 PASS,change-qualityreceiptcqr_e90f9a3287922c8e79daPASS)。远程 CI 状态依据 Goal 配置未轮询等待。
我的整体评价 / Verdict
综合长程任务(long_horizon:保持离线实验评估可续跑、防止被均值/全局一致率误导选择退化候选)与使用者体验(user_experience:单条 CLI 命令即可同时获得秩相关与悬崖诊断,无新增交互负担),本次改动属于精准且比例恰当的增强。就视觉审查而言:本次改动仅涉及离线 Python CLI(python -m benchmark.proxy_order)JSON 输出与参考文档片段,不涉及前端页面、桌面端或富交互渲染视图,已检查改动文件清单确认无 UI 表面变更。(未验证范围:无前端渲染界面受影响,未执行浏览器截屏检查。)
代码量与复杂度克制,未新增持久化状态或共享词表,完全复用既有面板读取与精确序比较原语,向后兼容原有全部输出键,未发现阻塞合并的代码或文档缺陷(canary premerge 提示的 benchmark_sensitive manual hold 属于仓库策略要求的合并前人工把关,不影响代码审阅通过)。
English verdict: APPROVE — exact head 9d4872b cleanly adds exact tie-adjusted kendall_tau_b and frontier cliff diagnostics to benchmark/proxy_order.py with preserved Decimal precision, full backward compatibility, clear documentation, and verified passing local unit, CLI subprocess, lint, type, boundary, and premerge checks (19/19 proxy_order tests passed).
Summary
Implements tie-adjusted Kendall rank correlation (
kendall_tau_b) and top-frontier / native-incumbent reversal diagnostics (frontier) inbenchmark/proxy_order.py, addressing the Proxy Misalignment & Cliff gap documented in #8835 §2.2(4) under #3243.Gap & Consumer
benchmark/proxy_order.pypreviously reported only pairwiseagrees/reverses/abstainscounts andnative_tied_pairs. A cohort where the lower bulk agrees monotonically can showagrees > reversesand even positive rank correlation while an over-optimizing candidate wins the proxy frontier and regresses below the native best (e.g., reward hacking or a tail timeout cliff).python -m benchmark.proxy_order proxy.json native.jsonor callingbenchmark.proxy_order.compare()(documented indocs/reference/benchmark-behavior-findings.md#compare-a-local-proxy-with-released-scores).Changes
benchmark/proxy_order.py:kendall_tau_bacross all matched pairs:(C - D) / math.sqrt((N - T_p) * (N - T_n)), counting every matched pair whereorder(left.proxy, right.proxy, proxy_direction) == 0inT_p(including joint ties) and returningNonewhen(N - T_p) * (N - T_n) == 0(<2matched candidates or all tied on either metric)._frontier_summary()returningfrontier:best_native_count: count of native-optimal matched candidates.best_proxy_matches_best_native: whether any proxy-optimal matched candidate is also native-optimal (Nonewhenmatchedis empty).false_improvements_over_best_native: count of non-native-optimal matched candidates that strictly beat at least one native-optimal matched candidate onproxy.compare()keys (cohort,coverage,native_tied_pairs,rules) and exactDecimal/intcomparison semantics.benchmark/tests/test_proxy_order.py:1.0), pure reversal (-1.0), proxy-only and joint ties, degenerate<2and all-tied cohorts (None), a 5-candidate high-end proxy cliff counterexample (agrees=6 > reverses=4,kendall_tau_b=0.2 > 0,best_proxy_matches_best_native=False,false_improvements_over_best_native=1), multiple native optima / minimize direction / strict tie boundary, sub-floatDecimalfrontier precision, and CLI subprocess execution (python -m benchmark.proxy_order).docs/reference/benchmark-behavior-findings.md:kendall_tau_bandfrontierfields and the high-end proxy cliff interpretation in#compare-a-local-proxy-with-released-scores.Verification
uv run --extra test python -m pytest -v benchmark/tests/test_proxy_order.py: 19 passeduv run --extra test python -m pytest -q tests/control_plane/test_work_yardstick_cli.py: 50 passeduv run --extra test python -m ruff check benchmark/proxy_order.py benchmark/tests/test_proxy_order.py: passeduv run --extra test python -m mypy: 17 source files checked, 0 issuesuv run --extra test python scripts/check_rfc_bilingual.py --base origin/main --head HEAD: passedloopx check --scan-path benchmark/proxy_order.py --scan-path benchmark/tests/test_proxy_order.py --scan-path docs/reference/benchmark-behavior-findings.md: clean (0 errors, 0 warnings)change-quality: receiptcqr_e90f9a3287922c8e79da(status: valid,decision: pass)loopx canary premerge --from-git-diff --git-diff-base origin/main --goal-id personal-loopx-maintenance --format json:direct_checks+python_compile5/5 passed,catalog_run6/6 passed,risk_profile_run8/8 passed,boundary_run1/1 passed (failures=0,manual_hold_count=1forbenchmark_sensitiveawaiting maintainer review).