Repository navigation
test(evals): Make the asking evals realistic, and ask for draft facts - #2518
Merged
Merged
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
dcramer
approved these changes
Nov 2, 2026
…e data viewImage and imageGenerate said "unsupported image format; use PNG, JPEG, GIF, or WebP" for a file whose bytes match no image format. For an attached .png with bad bytes, the agent read this as "the attachment did not load" and asked for a re-upload as if no file came through. The error now says the file exists and its content is not PNG, JPEG, GIF, or WebP data. Part of #2488 Co-Authored-By: David Cramer <[email protected]>
Slack delivery splits a reply over the inline budget into several posts. The fixture made each post its own reply, so a long answer failed `replies` count checks, and the judge saw the answer three times: the stored reply and each post. Consecutive posts that make up one stored reply are now one reply. A repeated post is still a second reply. Part of #2488 Co-Authored-By: David Cramer <[email protected]>
…channel Part of #2488 Co-Authored-By: David Cramer <[email protected]>
sentry-junior
Bot
changed the title
…s go The createAutomation description said the automation "delivers to the channel or DM of this conversation". With a task_creator outcome, the messages go to the creator's DM. Guardian read the description, decided that a direct reminder asked for in a public channel would post to that channel, and asked for confirmation (unclear_destination). The description now says that the automation belongs to the conversation and that its outcomes set where its messages go. Part of #2488 Co-Authored-By: David Cramer <[email protected]>
Member
|
evals are failing, is that related? |
…read it When the vision model could not read an image, the turn input said only that the attachment "could not be analyzed". The agent then told the user that the attachment did not come through and asked for a re-upload, as if no file arrived. The error now says the attachment arrived and the vision model could not read it. Part of #2488 Co-Authored-By: David Cramer <[email protected]>
The case named spawnAgent, gave the child task text, and told the parent what to do with the result. Now Maya asks to hand off a deploy command for a region that only she knows. The assertions do not change. Follow-up to #2504, part of #2488 Co-Authored-By: David Cramer <[email protected]>
The asking rule named another person's calendar that returns notFound, which is the Google Calendar eval scenario. policies/evals.md does not allow scenario phrases in product prompts. Co-Authored-By: David Cramer <[email protected]>
The out-of-office asking eval failed on the #2504 head and on this branch. The agent wrote a reply without the vacation dates and asked nothing, because it read the dates as a scope choice. A fact that only the person knows is now a reason to ask, and the agent must not drop it or guess it. Co-Authored-By: David Cramer <[email protected]>
This reverts commit d8027a3. Co-Authored-By: David Cramer <[email protected]>
The agent can write a complete out-of-office reply without dates, so the case asked for a fact that the request did not need. Maya now asks for the day she is back, which only she knows. The assertion does not change. Co-Authored-By: David Cramer <[email protected]>
The drafting rule allowed a placeholder or a question. So for a draft that needs a fact that only the person knows, the agent put in a placeholder and asked nothing. The out-of-office asking eval failed on the #2504 head and on this branch for this reason. Now the agent puts in the placeholder and asks for the facts with one addReviewItem item. Co-Authored-By: David Cramer <[email protected]>
sentry-junior
Bot
force-pushed
the
fix/asking-eval-overfit
branch
from
November 2, 2026 02:30
b984016 to
04cfdb3
Compare
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
When delivery splits a reply inside a code block, it closes the block at the end of the post and opens it again in the next post. The fixture removed only the continuation marker, so the reopen line, such as ```ts, kept the posts from matching the stored reply. Slack output now owns joinSlackReplyPosts, which undoes the markers and the fence close and reopen, and the fixture uses it.
joinSlackReplyPosts removed a fence close at the end of a post and a reopen at the start of the next post. A cut between two code blocks in the same language has the same shape, so it also removed real fences and the posts no longer matched the stored reply. The fixture now drops code fence lines when it compares posts with the stored reply, the same way it already drops other formatting. Both readings of the cut then compare the same, and Slack output does not need to export a join helper.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 0a23d5e. Configure here.
| if (!postTexts[start]) return 0; | ||
| let joined = ""; | ||
| for (let end = start; end < postTexts.length; end += 1) { | ||
| joined = joined ? `${joined} ${postTexts[end]}` : postTexts[end]!; |
There was a problem hiding this comment.
Fence-only posts break reply matching
Low Severity
comparableText now drops every fence line, so a Slack split whose first chunk is only a fence close or reopen becomes an empty postTexts entry. postCount then returns immediately and never joins the later chunks, so slackCallReplies misses the stored reply.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit 0a23d5e. Configure here.
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #2504, part of #2488
Blocked-child case (
asking.eval.ts): the case was scripted. It namedspawnAgent, wrote the child's task text ("The region is not in this task"), and told the parent "get what it needs so it can finish". Now Maya asks to hand off a deploy command for a Cloud Run region that she "picked last week". Only she knows the region. The prompt does not say how to delegate or how to ask, and it names no tool. The assertions do not change: onespawnAgentcall, anaddReviewItemaction item in the result Turn, and no?in the replies of that Turn.Out-of-office case (
asking.eval.ts): this case also failed on the final #2504 head. The agent wrote an out-of-office reply with no dates and asked nothing. That reply is complete, so the request did not need a fact from Maya. Maya now asks for a reply that says the day she is back, which only she knows. The assertion does not change.Drafting rule (
system-prompt-rules.ts): after that request change, the agent put in a[day/date]placeholder and still asked nothing. The cause is the drafting rule, which said "use clear placeholders or ask one focused question", so a placeholder alone satisfied it. Now the agent puts a placeholder where a fact that only the person knows goes, and asks for those facts with oneaddReviewItemitem. This matches the asking rule: deliver the rest of the work and ask about the blocked part.Neutral prompt example (
system-prompt-rules.ts): the asking rule used "another person's calendar that returns notFound" as its example of a gap that nobody can fill. That is the Google Calendar integration eval scenario, andpolicies/evals.mddoes not allow scenario phrases in product prompts. The example is now "data that only someone outside the Conversation controls". The Google Calendar integration cases still pass.I checked the other two cases. The CSV case and the Automation case ("Check with me whether to cancel ... or move it") use requests that a person would write and name no tool. I left them unchanged.
On b984016, all four
asking.eval.tscases pass in behavioral shard 4/4.via @dcramer.
--
View Session [Sentry]