Repository navigation
Leaderboard: folded note on provider refusals - #172
Merged
Merged
Conversation
… Argon in Pi (owner) In 10 of its 48 runs the model provider refused a request (finish_reason content_filter) and Pi, which does not retry, ended the run. The note is a fold in the chart foot's small grey type. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_011RFssRzn2zquijjLwvHBG6
…tched the run to Claude Opus 4.8 (owner) The fold is now "Provider refusals" with one line per model. In LWE, Opus 5.5 was refused once and went on as Opus 5.5, so LWE is not listed. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_011RFssRzn2zquijjLwvHBG6
wenhaochai
changed the title
Antigravity retries a refused request; in Structured LWE the retries ran out and the run ended, and in CREST the refused context summaries were kept cut short. In 7 runs the session ended while the agent waited for a background task. The fold is now "Refusals and harness faults". Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_011RFssRzn2zquijjLwvHBG6
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_011RFssRzn2zquijjLwvHBG6
wenhaochai
marked this pull request as ready for review
October 26, 2026 02:32
github-actions Bot
added a commit
that referenced
this pull request
Oct 26, 2026
Leaderboard: folded note on provider refusals [ci skip]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A short folded note in small grey type under the pass-rate bars:
From the run traces:
finish_reason: content_filter; Pi does not retry.model_refusal_fallbacktoclaude-opus-4-8with session scope, and the rest of the session ran on Opus 4.8. In LWE, Opus 5.5 was refused once and continued as Opus 5.5, so LWE is not listed.Draft: the wording may change after the remaining trace audits. Do not merge yet.
🤖 Generated with Claude Code
https://claude.ai/code/session_011RFssRzn2zquijjLwvHBG6