Repository navigation
fix(evals): Fix the integration failures on main's combined prompt - #2520
Merged
Merged
Conversation
…e data viewImage and imageGenerate said "unsupported image format; use PNG, JPEG, GIF, or WebP" for a file whose bytes match no image format. For an attached .png with bad bytes, the agent read this as "the attachment did not load" and asked for a re-upload as if no file came through. The error now says the file exists and its content is not PNG, JPEG, GIF, or WebP data. Part of #2488 Co-Authored-By: David Cramer <[email protected]>
Slack delivery splits a reply over the inline budget into several posts. The fixture made each post its own reply, so a long answer failed `replies` count checks, and the judge saw the answer three times: the stored reply and each post. Consecutive posts that make up one stored reply are now one reply. A repeated post is still a second reply. Part of #2488 Co-Authored-By: David Cramer <[email protected]>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
…channel Part of #2488 Co-Authored-By: David Cramer <[email protected]>
sentry-junior
Bot
force-pushed
the
fix/integration-evals-combined-main
branch
from
November 2, 2026 01:48
24c9a8b to
0d8683e
Compare
…s go The createAutomation description said the automation "delivers to the channel or DM of this conversation". With a task_creator outcome, the messages go to the creator's DM. Guardian read the description, decided that a direct reminder asked for in a public channel would post to that channel, and asked for confirmation (unclear_destination). The description now says that the automation belongs to the conversation and that its outcomes set where its messages go. Part of #2488 Co-Authored-By: David Cramer <[email protected]>
…read it When the vision model could not read an image, the turn input said only that the attachment "could not be analyzed". The agent then told the user that the attachment did not come through and asked for a re-upload, as if no file arrived. The error now says the attachment arrived and the vision model could not read it. Part of #2488 Co-Authored-By: David Cramer <[email protected]>
Member
|
fix the pr title/description |
dcramer
reviewed
Nov 2, 2026
When delivery splits a reply inside a code block, it closes the block at the end of the post and opens it again in the next post. The fixture removed only the continuation marker, so the reopen line, such as ```ts, kept the posts from matching the stored reply. Slack output now owns joinSlackReplyPosts, which undoes the markers and the fence close and reopen, and the fixture uses it. Co-Authored-By: David Cramer <[email protected]>
joinSlackReplyPosts removed a fence close at the end of a post and a reopen at the start of the next post. A cut between two code blocks in the same language has the same shape, so it also removed real fences and the posts no longer matched the stored reply. The fixture now drops code fence lines when it compares posts with the stored reply, the same way it already drops other formatting. Both readings of the cut then compare the same, and Slack output does not need to export a join helper. Co-Authored-By: David Cramer <[email protected]>
dcramer
marked this pull request as ready for review
November 2, 2026 03:05
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 4184e7f. Configure here.
The description repeated rules that other surfaces own. The tool-policy block owns creator credentials and subscribable tool results. The trigger identifier field and searchEventTypes own namespace-wide event reach. The namespace example also named a provider in core. Keep the destination and outcomes wording that the Guardian reminder case depends on. Co-Authored-By: David Cramer <[email protected]>
A stored instruction calls the creator "the Automation creator". The run had to swap in the mention from the `Created by:` line, and the model sometimes copied the phrase or left the mention out. A reminder for its creator then posted to the channel without pinging them. renderTaskInput now writes the creator mention in place of the phrase before the run sees the instructions. The `Created by:` line stays for "me" and "my". Watches have no creator, so their input does not change. Co-Authored-By: David Cramer <[email protected]>
Whether a due reminder post writes the creator's mention, and whether a created instruction says "the Automation creator", is model choice. The behavioral suite owns model choice. The assertions do not change. The runtime wiring stays covered. The creator mention in the run input is checked in heartbeat-scheduled-automations.test.ts. createAutomation rejects "this channel", and only a unit test covers that gate, so the integration creation case keeps that assertion. Co-Authored-By: David Cramer <[email protected]>
…-combined-main # Conflicts: # packages/junior/src/chat/automations/tools/create.ts
dcramer
approved these changes
Nov 2, 2026
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Integration evals on PRs off main with both #2514 and #2504 failed four cases. Each failure has a root cause in a tool contract, a turn input, or the eval fixture. None is a runtime double post.
Attachments: "when the vision model cannot read an image" (0.50 in 4 of 4 runs on main; at the 0.75 floor in 9 of 13 earlier runs)
The turn input said only that the image "could not be analyzed".
viewImagesaid "unsupported image format; use PNG, JPEG, GIF, or WebP" for a.pngwith bad bytes. The agent then told the user that the attachment "didn't come through" and asked for a re-upload, as if no file arrived. Both messages now say that the file arrived but could not be read as an image. The case now scores 1.0.Slack Thread Status and Title, both cases ("expected 2 to have a length of 1")
This is not a double post and not a second title. The failing assertion is
replies(lines 27 and 42). Since #2514, a how-to answer is about 2,200–2,400 characters, which is over the 2,200-character Slack inline budget. Delivery splits it into two posts. #2514's own branch failed this way at 08f9f6a. The fixture counted each post as a reply, and the judge saw the answer three times.The fixture (
slackCallRepliesinsrc/fixture/results.ts) now treats the consecutive posts that make up one stored reply as one reply. A repeated post is still a second reply. The comparison removes the continuation marker and drops code fence lines. A split inside a code block closes the block in one post and reopens it, with its language tag, in the next. Dropping the fence lines also keeps a cut between two real code blocks matching. The fix is fixture-only, and Slack output code does not change. I checked it against the real split posts of the failing runs. Unit tests cover a plain split, a split inside a ```ts block, a cut between two code blocks, and a repeated post.Schedule Creation: "simple one-off reminder" (Guardian
ask, then the agent asked to confirm)The
createAutomationdescription said that the automation "delivers to the channel or DM of this conversation". Guardian believed it over thetask_creatoroutcome and returnedunclear_destination. The new Guardian snapshot reproduces this on the old description (run 41374373955: "schedules a reminder in the active public Slack channel") and allows on the new one (run 41374880826). The description now says that the automation belongs to the conversation and that its outcomes set where its messages go. A later cleanup removes the sentences that other surfaces already own. The<tool-policy>block owns creator credentials and subscribable tool results. The triggeridentifierfield andsearchEventTypesown namespace-wide event reach. The cleanup also removes the provider example from core. The first invalid call (timezonebesideschedule) is a model nesting slip that 8 earlier passing runs also had. The strict schema catches it, and the retry recovers.Scheduled Delivery: the creator mention (run 41382009284 on 60f4efe, all 3 attempts)
createAutomationis in the tool list of every automation run. A change to its description stops the stored recordings from matching, so these cases run live. Live, a reminder for its creator often posted without the mention: "Reminder: The Automation creator, please do healthchecks.", "Reminder: Do your healthchecks.", and "Reminder: Do healthchecks.". Such a reminder posts to the channel without pinging its creator.requireInstructionReferencesmakes a stored instruction call the creator "the Automation creator".renderTaskInputpassed that text through, and only theCreated by:line told the run to write the mention instead.renderTaskInputnow writes the creator mention in place of "the Automation creator" (any case, whole words). Scheduled and event automations pass a creator. Watches do not, so their input does not change. TheCreated by:line stays, because it still covers "me" and "my".heartbeat-scheduled-automations.test.tsnow checks that a stored "the Automation creator" reaches the run as the mention in theInstructions:line. The test fails without the fix.All three failures were on "when a due reminder says me, mention its creator". Its legacy instruction says "me", not "the Automation creator", so this fix does not reach it. That case stays model choice. #2519, stacked on this PR, moves it to the behavioral suite. This PR does not move it.
CI: pending on a35f17f.
Part of #2488
via @dcramer.
--
View Session [Sentry]