Skip to content

fix(evals): Fix the integration failures on main's combined prompt - #2520

Merged
dcramer merged 11 commits into
mainfrom
fix/integration-evals-combined-main
Nov 2, 2026
Merged

dcramer merged 11 commits into
mainfrom
fix/integration-evals-combined-main

Conversation

@sentry-junior

@sentry-junior sentry-junior Bot commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

Integration evals on PRs off main with both #2514 and #2504 failed four cases. Each failure has a root cause in a tool contract, a turn input, or the eval fixture. None is a runtime double post.

Attachments: "when the vision model cannot read an image" (0.50 in 4 of 4 runs on main; at the 0.75 floor in 9 of 13 earlier runs)
The turn input said only that the image "could not be analyzed". viewImage said "unsupported image format; use PNG, JPEG, GIF, or WebP" for a .png with bad bytes. The agent then told the user that the attachment "didn't come through" and asked for a re-upload, as if no file arrived. Both messages now say that the file arrived but could not be read as an image. The case now scores 1.0.

Slack Thread Status and Title, both cases ("expected 2 to have a length of 1")
This is not a double post and not a second title. The failing assertion is replies (lines 27 and 42). Since #2514, a how-to answer is about 2,200–2,400 characters, which is over the 2,200-character Slack inline budget. Delivery splits it into two posts. #2514's own branch failed this way at 08f9f6a. The fixture counted each post as a reply, and the judge saw the answer three times.

The fixture (slackCallReplies in src/fixture/results.ts) now treats the consecutive posts that make up one stored reply as one reply. A repeated post is still a second reply. The comparison removes the continuation marker and drops code fence lines. A split inside a code block closes the block in one post and reopens it, with its language tag, in the next. Dropping the fence lines also keeps a cut between two real code blocks matching. The fix is fixture-only, and Slack output code does not change. I checked it against the real split posts of the failing runs. Unit tests cover a plain split, a split inside a ```ts block, a cut between two code blocks, and a repeated post.

Schedule Creation: "simple one-off reminder" (Guardian ask, then the agent asked to confirm)
The createAutomation description said that the automation "delivers to the channel or DM of this conversation". Guardian believed it over the task_creator outcome and returned unclear_destination. The new Guardian snapshot reproduces this on the old description (run 41374373955: "schedules a reminder in the active public Slack channel") and allows on the new one (run 41374880826). The description now says that the automation belongs to the conversation and that its outcomes set where its messages go. A later cleanup removes the sentences that other surfaces already own. The <tool-policy> block owns creator credentials and subscribable tool results. The trigger identifier field and searchEventTypes own namespace-wide event reach. The cleanup also removes the provider example from core. The first invalid call (timezone beside schedule) is a model nesting slip that 8 earlier passing runs also had. The strict schema catches it, and the retry recovers.

Scheduled Delivery: the creator mention (run 41382009284 on 60f4efe, all 3 attempts)
createAutomation is in the tool list of every automation run. A change to its description stops the stored recordings from matching, so these cases run live. Live, a reminder for its creator often posted without the mention: "Reminder: The Automation creator, please do healthchecks.", "Reminder: Do your healthchecks.", and "Reminder: Do healthchecks.". Such a reminder posts to the channel without pinging its creator.

requireInstructionReferences makes a stored instruction call the creator "the Automation creator". renderTaskInput passed that text through, and only the Created by: line told the run to write the mention instead. renderTaskInput now writes the creator mention in place of "the Automation creator" (any case, whole words). Scheduled and event automations pass a creator. Watches do not, so their input does not change. The Created by: line stays, because it still covers "me" and "my". heartbeat-scheduled-automations.test.ts now checks that a stored "the Automation creator" reaches the run as the mention in the Instructions: line. The test fails without the fix.

All three failures were on "when a due reminder says me, mention its creator". Its legacy instruction says "me", not "the Automation creator", so this fix does not reach it. That case stays model choice. #2519, stacked on this PR, moves it to the behavioral suite. This PR does not move it.

CI: pending on a35f17f.

Part of #2488

via @dcramer.

--

View Session [Sentry]

sentry-junior Bot and others added 2 commits November 2, 2026 01:40
…e data

viewImage and imageGenerate said "unsupported image format; use PNG, JPEG,
GIF, or WebP" for a file whose bytes match no image format. For an
attached .png with bad bytes, the agent read this as "the attachment did
not load" and asked for a re-upload as if no file came through. The error
now says the file exists and its content is not PNG, JPEG, GIF, or WebP
data.

Part of #2488

Co-Authored-By: David Cramer <[email protected]>
Slack delivery splits a reply over the inline budget into several posts.
The fixture made each post its own reply, so a long answer failed
`replies` count checks, and the judge saw the answer three times: the
stored reply and each post. Consecutive posts that make up one stored
reply are now one reply. A repeated post is still a second reply.

Part of #2488

Co-Authored-By: David Cramer <[email protected]>
@vercel

vercel Bot commented Nov 2, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
junior-docs Ready Ready Preview Nov 2, 2026 4:59pm UTC

Request Review

…s go

The createAutomation description said the automation "delivers to the
channel or DM of this conversation". With a task_creator outcome, the
messages go to the creator's DM. Guardian read the description, decided
that a direct reminder asked for in a public channel would post to that
channel, and asked for confirmation (unclear_destination). The
description now says that the automation belongs to the conversation and
that its outcomes set where its messages go.

Part of #2488

Co-Authored-By: David Cramer <[email protected]>
…read it

When the vision model could not read an image, the turn input said only
that the attachment "could not be analyzed". The agent then told the
user that the attachment did not come through and asked for a re-upload,
as if no file arrived. The error now says the attachment arrived and the
vision model could not read it.

Part of #2488

Co-Authored-By: David Cramer <[email protected]>
@dcramer

dcramer commented Nov 2, 2026

Copy link
Copy Markdown
Member

fix the pr title/description

@sentry-junior sentry-junior Bot changed the title fix(evals): Fix the three integration failures on main's combined prompt fix(evals): Fix the integration failures on main's combined prompt Nov 2, 2026
When delivery splits a reply inside a code block, it closes the block at
the end of the post and opens it again in the next post. The fixture
removed only the continuation marker, so the reopen line, such as
```ts, kept the posts from matching the stored reply. Slack output now
owns joinSlackReplyPosts, which undoes the markers and the fence close
and reopen, and the fixture uses it.

Co-Authored-By: David Cramer <[email protected]>
joinSlackReplyPosts removed a fence close at the end of a post and a
reopen at the start of the next post. A cut between two code blocks in
the same language has the same shape, so it also removed real fences
and the posts no longer matched the stored reply.

The fixture now drops code fence lines when it compares posts with the
stored reply, the same way it already drops other formatting. Both
readings of the cut then compare the same, and Slack output does not
need to export a join helper.

Co-Authored-By: David Cramer <[email protected]>
@dcramer
dcramer marked this pull request as ready for review November 2, 2026 03:05
@github-actions github-actions Bot added the risk: high PR risk score: high label Nov 2, 2026

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 4184e7f. Configure here.

The description repeated rules that other surfaces own. The tool-policy
block owns creator credentials and subscribable tool results. The
trigger identifier field and searchEventTypes own namespace-wide event
reach. The namespace example also named a provider in core.

Keep the destination and outcomes wording that the Guardian reminder
case depends on.

Co-Authored-By: David Cramer <[email protected]>
@github-actions github-actions Bot added risk: medium PR risk score: medium and removed risk: high PR risk score: high labels Nov 2, 2026
sentry-junior Bot and others added 2 commits November 2, 2026 04:27
A stored instruction calls the creator "the Automation creator". The run
had to swap in the mention from the `Created by:` line, and the model
sometimes copied the phrase or left the mention out. A reminder for its
creator then posted to the channel without pinging them.

renderTaskInput now writes the creator mention in place of the phrase
before the run sees the instructions. The `Created by:` line stays for
"me" and "my". Watches have no creator, so their input does not change.

Co-Authored-By: David Cramer <[email protected]>
Whether a due reminder post writes the creator's mention, and whether a
created instruction says "the Automation creator", is model choice.
The behavioral suite owns model choice. The assertions do not change.

The runtime wiring stays covered. The creator mention in the run input
is checked in heartbeat-scheduled-automations.test.ts. createAutomation
rejects "this channel", and only a unit test covers that gate, so the
integration creation case keeps that assertion.

Co-Authored-By: David Cramer <[email protected]>
@github-actions github-actions Bot added risk: high PR risk score: high and removed risk: medium PR risk score: medium labels Nov 2, 2026
…-combined-main

# Conflicts:
#	packages/junior/src/chat/automations/tools/create.ts
@github-actions github-actions Bot added risk: medium PR risk score: medium and removed risk: high PR risk score: high labels Nov 2, 2026
@dcramer
dcramer merged commit ed194f8 into main Nov 2, 2026
51 checks passed
@dcramer
dcramer deleted the fix/integration-evals-combined-main branch November 2, 2026 17:23

This branch was successfully deployed

1 active deployment
Preview – junior-docs — 3512ef0a Deployed Nov 2, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

risk: medium PR risk score: medium

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant