Repository navigation
fix(task-execution): Route conversation wakes to the current deployment - #2570
sentry-junior[bot] wants to merge 1 commit into
Conversation
Vercel Queues pins each message to the deployment that sent it. A Turn that started before a deploy keeps yielding and retrying on the old deployment, and a rollout that kills that worker counts toward the lost-worker limit. Send and consume conversation work with an unpinned queue client so wakes are not tied to the old deployment. Other queues stay pinned. Wake payloads hold only ids; durable state already spans versions. Co-Authored-By: BYK <[email protected]>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
2 screenshot changes — 2 changed · 0 added · 0 removed Review screenshots in Frameshift
|
![sentry-junior[bot]](https://avatars.githubusercontent.com/in/2960336?s=60&v=4){kind=small}
There was a problem hiding this comment.
Verify-before-merge check (1) fails: unpinned push messages are not delivered. Do not merge.
I deployed a scratch Vercel project to the Production target with @vercel/[email protected] (the locked version) and two push consumers (queue/v2beta triggers). Results across 4 Production deployments:
| Send | Consumer | Result |
|---|---|---|
| default client (pinned) | default handleCallback |
delivered in <0.5s, acked |
QueueClient({ deploymentId: null }) |
QueueClient({ deploymentId: null }).handleCallback |
never delivered (no consumer invocation after 6 min) |
| unpinned, sent to the pinned-consumer topic | default handleCallback |
never delivered |
| pinned, sent to the unpinned-consumer topic | unpinned handleCallback |
delivered, then ack failed: MessageNotAvailableError: Message is not currently in-flight |
QueueClient({ deploymentId: <newer deployment id> }) from the older deployment |
default handleCallback on the newer one |
delivered to the newer deployment, acked (also with delaySeconds: 20) |
QueueClient({ deploymentId: <older deployment id> }) from the newer deployment |
default handleCallback on the older one |
delivered to the older deployment, acked |
| default client from an older deployment after a newer one was promoted | default handleCallback |
delivered to the older deployment |
So with this change, every conversation wake is sent and then never pushed. Heartbeat re-wakes use the same client, so they would not recover the work. Merging this would stop all conversation work.
What does work: pinning to an explicit deployment id. An old deployment can send a wake that the newer deployment receives and acks with its normal pinned client. If the goal is to move work onto the newest deployment, the sender needs the current Production deployment id (for example, written to shared state by the Production heartbeat, which runs only on the current Production deployment), with the sender's own id as the fallback. Whether that is worth building still depends on the premise. The rollout kills the in-flight worker either way, so the second loss on the old deployment still needs evidence.
Vercel Queues pins each push message to the deployment that sent it. A Turn that starts before a deploy keeps yielding and retrying on the old deployment. A rollout that kills that worker counts as a lost worker (
MAX_LOST_WORKER_RESUMES = 1), so a second loss fails the Turn. We think this is why running agents and sub-agents stop on deploy and do not resume. That rollouts cause these losses is an inference from code and telemetry. Nobody watched it happen.This change sends and consumes conversation work with a
QueueClient({ deploymentId: null }). Wakes are then not tied to the deployment that sent them. The consumer uses the same unpinned client, so ack and lease calls match the message. Plugin tasks, Workspace snapshot jobs, and every other queue stay pinned. BullMQ ignores the flag.Cross-version risk: new code can now get wakes from old code, and the reverse. A wake holds only a signed conversation id and a schema version. Durable work state is in shared Redis/Postgres, and old and new workers can already read it during a deploy.
Verify before merge. The docs do not say how an unpinned push message is delivered. They describe pinned delivery, and Vercel's own unpinned examples consume in poll mode. The SDK supports
deploymentId: nullfor push clients, and the Queues API calls requests withoutVqs-Deployment-Id"deploymentless mode". On a preview deployment, check that:If (1) fails, wakes are not delivered and conversation work stalls until their TTL ends. Do not merge in that case.
Related: #2562 closes stale Turn records from the same failure path (JUNIOR-9M). JUNIOR-NP (multiple active agent invocations) is not fixed here.
Tests: focused component and integration tests pass, and
pnpm typecheckpasses.tests/integration/durable-queue.test.tshas the same 4 failures onmainin this sandbox (Postgres insert errors and timeouts), with and without this change.via @BYK.
--
View Session in Sentry