Repository navigation
[Benchmark] 增加 Benchmark Report 的可读性，修复 LLM 配置项问题 (#262) - #321
Merged
Merged
Conversation
The report's H1 was derived from the two run labels ("bench compare: base ->
cand"); it is now a fixed title. The compared sides stay named in the 证据
section, and the suite pins both the new title and that evidence.
Refs T-Auto#262
The report led with a summary bar and a `结论` section; it now opens with the fixed title and goes straight into the evidence, and the `结论` — pass rate, paired token Δ, noise floor, direction, gate and reasons — ends the document, so a reviewer reads the data before the judgement. Every percent is painted by its sign (a drop or no change green, a rise red, the same palette as `panel.html`) with inline spans; the digits stay the core's, including in the gate reasons and the warnings. A value the core stated as a bare number is printed verbatim and painted the same way instead of gaining an invented sign. Two diagnostics join the report: per-task steps and tool calls, summed in `TOTAL` and read as `min` or `median` by `--aggregate` (one decimal when an even repeat count halves the median, so a cell never contradicts its own Δ%), and the 工具调用 table marks a candidate-side call to a tool outside `ops` in red. bench/README.md documents the reading order and both colour rules; the suite pins the layout, the colours and the counts. Refs T-Auto#262
Retitle the report to "DSH-OPS Benchmark Report" and translate every rendered section, table header, direction label and evidence line, plus the matching CLI notes and comments, so generated report.md and comparison artifacts never carry CJK text. Align bench/README's report walkthrough with the English section names and add a guard assertion that fails if CJK returns to a rendered report. Verified with node test/run.mjs bench- (17 suites, all passing) and npm run check:size.
The default artifact was <cand>/report.md, so a report about a whole sweep was buried inside the arm directory that measured half of it. It now lands in the sweep's group folder as 'DSH-OPS Benchmark Report <UTC>.md': full-sweep-4 keeps its report at bench/runs/full-sweep-4/, a rerun keeps the earlier artifact instead of overwriting it, and a run with no group folder above it keeps the report beside its run.json. --out, --out - and --json are unchanged, and the rendered text stays clock-free and diffable. The optional benchmark smoke uploads the stamped name as well as report.md.
The per-task table now carries the numbers a reviewer scans for: paired tokens, steps, tool calls, tool errors and pass, with each Δ% beside the value it describes. The pairs-usable, excluded, pass, direction and geomean rows are gone, and no table carries an explanatory line under its heading. The tool-calls table moves directly under that table: one row per tool with the sweep-wide base/cand columns kept, then one column per task. Its colour follows adoption rather than the sign: an ops_* tool rising is green, any other tool rising is red, and the reverse for a fall. The excluded-pairs, A/A noise-band and token-attribution sections are gone; the excluded counts survive in the warnings, and the conclusion still states the floor and the one direction that is read. The sweep runner now counts tool results whose status is error, which the new column reads and the metrics schema already declared.
…atch bench/llm.patch.example.yml, the README and setup all promised that every dsh call carries bench/llm.patch.yml; the runner never did. A sweep then ran both arms on the profiles' own routes - identical on both sides, so the report could not show the difference and the run still looked complete. Resolve the overlay before any spawn: default to the file setup creates, accept --llm-patch <path> for another one, --no-llm-patch to state that the profiles' own routes are meant, and refuse a missing overlay with exit 2. The archived argv records the overlay the operator did not type, and the plan prints it.
Richeir
changed the title
Richeir
marked this pull request as ready for review
October 21, 2026 13:37
Thatgfsj
approved these changes
Oct 21, 2026
Thatgfsj
left a comment
Thatgfsj
left a comment
Collaborator
There was a problem hiding this comment.
工作流加了三条注释很神奇啊，别的都是测试区域内的文件，不影响全局，可以合并
This was referenced Oct 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #262
What was wrong
Environment