Skip to content

feat(bench): slim the report, shrink the fixtures, rewrite the manual (#379) - #381

Merged
T-Auto merged 10 commits into
T-Auto:mainfrom
Richeir:bench/doc-improvements
Oct 25, 2026
Merged

feat(bench): slim the report, shrink the fixtures, rewrite the manual (#379)#381
T-Auto merged 10 commits into
T-Auto:mainfrom
Richeir:bench/doc-improvements

Conversation

@Richeir

@Richeir Richeir commented Oct 24, 2026 •
edited by T-Auto
Loading

Copy link
Copy Markdown
Collaborator

Closes #379

What was wrong

优化后的 Benchmark Report：

  • 继续加强可读性

  • 增加任务时间对比

  • 优化任务和流程，自动生成 MD 格式报告（目前执行一次全量 Benchmark，时间在 10 分钟以内）

image

The change

bench/ 的操作面与文档一次收口，判定口径一行未动（pass-drop 先行、只比配对成本、A/A 噪声底、不裁时间、--fail-on 全部保持）。

  • bench/run.mjs：封存从默认改为显式 --seal（--no-seal 保留为兼容开关）；算出对比后调用 renderReport 把报告写进组目录，新增 --no-report 关闭，单臂 / 中断 / 无对比时显式记 skipped，而不是写一个看起来完整的文件；ab 的 usage 不再列 3 sealing is not wired yet（EXIT.pending 仍留给未落地的切片）。

  • bench/lib/next-folder.mjs、bench/cli.mjs：默认组目录 benchmark-test-NN → benchmark-YYYY-MM-DD-NN（本机日历日期、当天从 -00 起、跳过任何已存在的名字），planAb 接受注入时钟，套件可断言确切名字。

  • bench/tasks/*.task.json：六个任务 repeat: 2 → 1；gate-repair-loop 24 → 12 个 stage；api-rename 60 → 12 个模块（生成器参数、不变量与判卷不变）。

  • bench/compare.mjs、bench/lib/compare-core.mjs、bench/run/session.mjs：标题 # DSH-OPS Benchmark Report (beta)；cand 字样退出渲染输出（列头 ops (Δ%)、steps: base→ops (Δ%)、(ops only)、Tool calls 表的 ops 列）；首表加 time: base→ops (Δ%) 墙钟列（沿用 --aggregate median|min 的 durationMsMedian / durationMsMin，TOTAL 按时长求和）；新增 ## Per-request overhead (what every request costs before any work)，口径是新算的 metrics.prompt（每步 prompt = input + cacheRead + cacheWrite，首个请求只由 step 1 决定、按 turn 取首），全有才报，旧归档缺数据时打印 Skipped；删掉 Warnings / Conclusion / Evidence / steps-and-tool-calls 表 / 用量拆分表 / footer，renderReport 的 sources 参数随之删除。

  • bench/README.md、bench/llm.patch.example.yml：README 197 → 44 行，收敛成中文「Benchmark Manual」（准备 → 怎么跑 → 结果在哪看 → 任务一览），overlay 模板同步改写。

  • test/bench-{schedule,run-session,compare-core,compare,cli}.test.mjs：同步更新，新增断言 267 行（实现侧非测试插入 397 行）。

刻意没做：不动 run.json 的字段名与 schema 结构；不碰 setup/profile.mjs 的组件行；不做多模型 / 跨平台 / 开关矩阵（#294 / #295 / #296）；不把 bench 放进 CI 或门禁（#205 边界）。一处有意的收窄：test/bench-cli 里「README ↔ run.schema.json 交叉检查」随 README 重写被删除（只留 schema 自检），理由与替代做法写在 #379 的「需要评审确认」一节。默认不写 run.json 之后，要可比证据得显式 --seal——这一点也记在同一节。

Validation


$ npm run verify
image

Environment

  • Windows 11 (10.0.26100) x64；Node v24.11.0；npm 11.12.1

  • 基于上游 main b963fb2（fix(policy): name the shell tool by its published name (#306)）

  • 安装通道：本地 checkout，npm ci --ignore-scripts；lib/ 与 native worker 由 npm run build、npm run build:ptc 本地构建

Richeir added 10 commits October 24, 2026 14:49
@Richeir
feat(bench): make A/B sealing opt-in and wire it to the runner
08c7c3f 
bench/cli.mjs spawns bench/run.mjs as a child and cannot inject a callback, so the runner's requirement of an injected io.seal made every interactive sweep exit 3 before spawning anything unless --no-seal was passed. Sealing is now opt-in with --seal, and every sweep - sealed or not - keeps each arm's results/ and summary.json, which bench report renders the report from.

--seal resolves bench/run/seal.mjs (an injected io.seal still wins), fingerprints both arms by hashing \dsh --profile <name> --dump-config\ into the arms[].profileHash that sealRun requires, and refuses before any session when a fingerprint cannot be read. sealSweep adapts the seal context's {base, cand} directory map to the directory list sealRun folds; that mismatch was invisible to the injected doubles and only a real sweep could expose it.
@Richeir
docs(bench): document the current workflow and rewrite the overlay te…
4500ad8 
…mplate

Add bench/README-NEW.md (preparation, full and single-task sweeps, reports, every task and gate with the ops_* tools they exercise, and where temporary files land) and bench/README-mini.md (the short preparation path).

Rewrite bench/llm.patch.example.yml's header in English and add the credentials-local insert the working copy needs, since the base/headless bundles ship no credentials service.
@Richeir
feat(bench): default the sweep to one repeat per task
b1594ce 
DEFAULT_REPEAT drops 3 -> 1 and every task in bench/tasks/ declares
"repeat": 1, so an ab sweep without --repeat now runs each task once
per arm (12 sessions, 6 interleaved pairs) instead of twice per arm.
--repeat N still overrides everything; the default only covers a task
with no repeat field of its own. The default-value assertion in
test/bench-schedule.test.mjs follows so reverting the constant fails
the suite.
@Richeir
feat(bench): date-stamp the default group folder and auto-write the s…
eaec3e1 
…weep report

The default group folder is now benchmark-YYYY-MM-DD-NN — the local calendar
date, numbered from -00, incrementing past anything already present — instead
of the ever-growing benchmark-test-NN. Allocation still happens exactly once
per sweep, still skips any same-named entry case-insensitively, and the clock
is injectable so suites assert exact names.

A finished sweep now renders its own verdict through compare.mjs's
renderReport and writes "DSH-OPS Benchmark Report <UTC>.md" into the group
folder (or the --out directory) — the bench report artifact without a second
command, re-renderable at any time. --no-report skips it; an interrupted
sweep writes none, because a partial run is not a comparison; a failed write
is a loud note, not a failed sweep. The ab help text and the three bench
READMEs follow, along with the repeat-2-to-1 leftovers in README-NEW's task
table.

api-rename's corpus shrinks to one fifth: symbolTree files 60 -> 12 (seed
unchanged, all five module shapes present, the look-alike-only decoys
included), so the model searches, reads and edits 11 touched files instead
of 51. The task's title and the README-NEW row follow.
@Richeir
feat(bench): show each task's wall-clock in the report's first table
9aa1703 
Durations were already recorded per session; compare-core now aggregates
them like steps — durationMsMedian / durationMsMin under the same
--aggregate median|min, per-task delta.durationPct, and totals summed over
the task medians. The per-task pairs table gains a
"time base→cand (Δ%)" column (seconds, one decimal, — when unmeasured),
and its Δ% is painted like every other percent.

The bench docs' "never judge time" principle statements are gone, and the
report footer now says time is reported like any other metric. The gate
itself is unchanged: it judges pass rate and paired tokens, and duration
stays out of every verdict — the new core assertions pin that gate
outcomes do not move with duration.
@Richeir
docs(bench): fold the bench docs into one short README
a598ee9 
README-NEW.md and README-mini.md are gone: bench/README.md now carries the
short preparation-and-usage guide on its own. No script, manifest or doc
referenced the two removed files.
@Richeir
feat(bench): report title (beta), ops labels, and a 12-stage gate-rep…
f9324b6 
…air-loop

The generated report's heading is now "# DSH-OPS Benchmark Report (beta)";
the file name keeps its old shape ("DSH-OPS Benchmark Report <UTC>.md"),
which the report subcommand, the runner help and the bench README all say.

Every rendered "cand" label becomes "ops" — the first table, the (ops only)
marker, the tool table, the three diagnostic headers and the Evidence line —
matching the bench-ops arm label. JSON keys, CLI flags and warning codes are
unchanged: they are the machine contract.

gate-repair-loop's pipeline is 12 stages instead of 24; every stage is still
broken, so the repair loop halves with it. Seed, probes and linesPerGate are
untouched, and the fixture regenerates with all invariants intact.

test/bench-cli drops the README ↔ run.schema.json cross-check — the bench
docs are now one short README — and keeps the schema self-checks.
@Richeir
feat(bench): add the per-request overhead table and slim the report
646e1bc 
The report gains "Per-request overhead (what every request costs before any
work)": one first-request cell per task (base → ops with the absolute delta
and a painted Δ%). Real sweeps never recorded the data it needs, so the
session fold now derives metrics.prompt exactly as the aggregator defines
it — per-step prompt = input + cacheRead + cacheWrite, the first request set
only by step 1, per-turn firsts, reported all-or-nothing — and new sweeps
light the table up while archives without the data render the section's
Skipped note instead of inventing numbers.

Everything else got smaller: Warnings, Conclusion, Evidence, the steps and
tool calls table, the usage split table and the closing footer are gone, so
the report is four data tables and nothing else. The gate's verdict still
travels through the exit code and the comparison document, unchanged.
Section headings lose their parenthetical qualifiers (Tool calls, Per-task
bucket detail), per-column headers read "steps: base→ops (Δ%)" to separate
the metric from the direction, and renderReport drops its sources parameter,
which only the removed Evidence ever used.
@Richeir
docs(bench): describe every task in the README
7f17d05 
bench/README.md gains a task table: fixture size, what each task measures
(symbol rename vs decoys, paged reads against stale-comment bait, layered
config priorities, shell-side aggregation, the run-fix-rerun repair loop,
exact log extraction) and which ops_* tool each step maps to. The numbers
match the current fixtures — api-rename 12 modules and gate-repair-loop 12
stages included — and the expectations are the generators', recomputed per
run.
@Richeir
docs(bench): retitle the manual and add a where-to-look section
f1dafcc 
The bench README becomes the Benchmark Manual: setup says what it prepares
(the two headless profiles and the locally built plugin), the LLM overlay
section is reworded, and the group-folder naming plus the auto-written
report move into their own "结果在哪看" section.
This was referenced Oct 24, 2026
Closed
Closed
Merged
Merged
Merged
Merged
Closed
Closed
@T-Auto
T-Auto merged commit d14bae8 into T-Auto:main Oct 25, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@T-Auto T-Auto T-Auto approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Benchmark] 操作面与文档瘦身：默认不 seal、夹具瘦身、报告精简、README 重写

2 participants

@Richeir @T-Auto