Repository navigation
test(evals): Move scheduler model-choice cases to behavioral - #2519
Merged
sentry-junior[bot] merged 1 commit intoNov 2, 2026
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
sentry-junior
Bot
force-pushed
the
test/scheduler-model-choice-behavioral
branch
from
November 2, 2026 02:29
fba1be8 to
68d5644
Compare
Whether a due reminder post writes the creator's mention, and whether a created instruction says "the Automation creator", is model choice. The behavioral suite owns model choice. The assertions do not change. The runtime wiring stays covered. The creator mention in the run input is checked in heartbeat-scheduled-automations.test.ts. createAutomation rejects "this channel", and only a unit test covers that gate, so the integration creation case keeps that assertion. Co-Authored-By: David Cramer <[email protected]>
sentry-junior
Bot
force-pushed
the
test/scheduler-model-choice-behavioral
branch
from
November 2, 2026 04:29
68d5644 to
9041739
Compare
sentry-junior
Bot
changed the base branch from
main
to
fix/integration-evals-combined-main
November 2, 2026 04:30
Contributor
|
5 screenshot changes — 5 changed · 0 added · 0 removed Review screenshots in Frameshift
|
sentry-junior
Bot
merged commit Nov 2, 2026
3512ef0
into
fix/integration-evals-combined-main
50 checks passed
dcramer
added a commit
that referenced
this pull request
Nov 2, 2026
…2520) Integration evals on PRs off main with both #2514 and #2504 failed four cases. Each failure has a root cause in a tool contract, a turn input, or the eval fixture. None is a runtime double post. **Attachments: "when the vision model cannot read an image"** (0.50 in 4 of 4 runs on main; at the 0.75 floor in 9 of 13 earlier runs) The turn input said only that the image "could not be analyzed". `viewImage` said "unsupported image format; use PNG, JPEG, GIF, or WebP" for a `.png` with bad bytes. The agent then told the user that the attachment "didn't come through" and asked for a re-upload, as if no file arrived. Both messages now say that the file arrived but could not be read as an image. The case now scores 1.0. **Slack Thread Status and Title, both cases** ("expected 2 to have a length of 1") This is not a double post and not a second title. The failing assertion is `replies` (lines 27 and 42). Since #2514, a how-to answer is about 2,200–2,400 characters, which is over the 2,200-character Slack inline budget. Delivery splits it into two posts. #2514's own branch failed this way at 08f9f6a. The fixture counted each post as a reply, and the judge saw the answer three times. The fixture (`slackCallReplies` in `src/fixture/results.ts`) now treats the consecutive posts that make up one stored reply as one reply. A repeated post is still a second reply. The comparison removes the continuation marker and drops code fence lines. A split inside a code block closes the block in one post and reopens it, with its language tag, in the next. Dropping the fence lines also keeps a cut between two real code blocks matching. The fix is fixture-only, and Slack output code does not change. I checked it against the real split posts of the failing runs. Unit tests cover a plain split, a split inside a ```ts block, a cut between two code blocks, and a repeated post. **Schedule Creation: "simple one-off reminder"** (Guardian `ask`, then the agent asked to confirm) The `createAutomation` description said that the automation "delivers to the channel or DM of this conversation". Guardian believed it over the `task_creator` outcome and returned `unclear_destination`. The new Guardian snapshot reproduces this on the old description (run 41374373955: "schedules a reminder in the active public Slack channel") and allows on the new one (run 41374880826). The description now says that the automation belongs to the conversation and that its outcomes set where its messages go. A later cleanup removes the sentences that other surfaces already own. The `<tool-policy>` block owns creator credentials and subscribable tool results. The trigger `identifier` field and `searchEventTypes` own namespace-wide event reach. The cleanup also removes the provider example from core. The first invalid call (`timezone` beside `schedule`) is a model nesting slip that 8 earlier passing runs also had. The strict schema catches it, and the retry recovers. **Scheduled Delivery: the creator mention** (run 41382009284 on 60f4efe, all 3 attempts) `createAutomation` is in the tool list of every automation run. A change to its description stops the stored recordings from matching, so these cases run live. Live, a reminder for its creator often posted without the mention: "Reminder: The Automation creator, please do healthchecks.", "Reminder: Do your healthchecks.", and "Reminder: Do healthchecks.". Such a reminder posts to the channel without pinging its creator. `requireInstructionReferences` makes a stored instruction call the creator "the Automation creator". `renderTaskInput` passed that text through, and only the `Created by:` line told the run to write the mention instead. `renderTaskInput` now writes the creator mention in place of "the Automation creator" (any case, whole words). Scheduled and event automations pass a creator. Watches do not, so their input does not change. The `Created by:` line stays, because it still covers "me" and "my". `heartbeat-scheduled-automations.test.ts` now checks that a stored "the Automation creator" reaches the run as the mention in the `Instructions:` line. The test fails without the fix. All three failures were on "when a due reminder says me, mention its creator". Its legacy instruction says "me", not "the Automation creator", so this fix does not reach it. That case stays model choice. #2519, stacked on this PR, moves it to the behavioral suite. This PR does not move it. CI: pending on a35f17f. Part of #2488 <!-- junior-request-attribution:start --> via @dcramer. <!-- junior-request-attribution:end --> <!-- junior-session-footer:start --> <!-- junior-conversation-id:agent%3A7a1dc2757924a78a9104357ce0de2cbf --> -- [View Session](https://junior-prod.sentry.dev/conversations/agent%3A7a1dc2757924a78a9104357ce0de2cbf) [[Sentry]](https://sentry.sentry.io/explore/conversations/agent%3A7a1dc2757924a78a9104357ce0de2cbf/?project=4510944073809921) <!-- junior-session-footer:end --> --------- Co-authored-by: sentry-junior[bot] <264270552+sentry-junior[bot]@users.noreply.github.com> Co-authored-by: David Cramer <[email protected]>
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #2520. Merge #2520 first. GitHub then retargets this PR to
main.This PR moves the scheduler cases that test model choice from the integration suite to the behavioral suite. The assertions do not change.
createAutomationgate.With #2520,
renderTaskInputwrites the creator mention in place of "the Automation creator" before the run sees the instructions.heartbeat-scheduled-automations.test.tsnow checks that wiring: a stored "the Automation creator" reaches the run as the mention in theInstructions:line. So the moved "names the Automation creator" case no longer tests the swap itself. It now tests only that the model copies the mention into the post. Its assertions stay as they are."says me" stays model choice. Legacy instructions say "me", and only the
Created by:line tells the run who that is.Part of #2488
via @dcramer.
--
View Session [Sentry]