Repository navigation
feat: make the completion review its own flag - #45
Merged
Merged
Conversation
--benchmark-report and --benchmark-trajectory now only write their files. The completion-audit turn runs only with --completion-review or SC_COMPLETION_REVIEW=1. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Collaborator
|
i think it should be on? not entirely sure |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
--benchmark-reportand--benchmark-trajectoryused to turn on the completion review (benchmark_modeinrc-cli/src/main.rsfedwith_completion_review). Asking for a report file therefore also changed how the agent behaved: after its first stop following tool work, Marathon got one extra completion-audit note that interactive users never get.This splits the two:
--benchmark-reportand--benchmark-trajectoryonly write their files.--completion-review(orSC_COMPLETION_REVIEW=1) turns on the audit turn on its own. It works in headless and interactive runs.Behavior change: benchmark runs no longer get the review unless they pass
--completion-review. By default they now run the way users run Marathon.Why: bench-runner measured the review on
glm-5.3-marathonwith 3 runs each way on a small task. It took 5–6 model requests instead of 3 and about 70–100% more tokens, and it makes Marathon's benchmark runs differ from other agents' runs. Harnesses that want it can still pass the flag.@danielsuit, you added the review in 7eb1b4d. Can you check whether benchmarks were meant to keep it on by default?
Test plan
--completion-reviewturns it on, and both work togetherSC_COMPLETION_REVIEWparses1/true/onand0/false(checked locally)cargo fmt --check, Clippy 1.99-D warnings,cargo test --workspace(573 pass),cargo +1.89.0 check, DLR fmt and Clippy🤖 Generated with Claude Code