Skip to content

OUT-4105 | Sweep stale grouped-email windows instead of deleting them on failure - #1438

Closed
priosshrsth wants to merge 6 commits into
mainfrom
anit/out-4105-prismaclientinitializationerror
Closed

OUT-4105 | Sweep stale grouped-email windows instead of deleting them on failure#1438
priosshrsth wants to merge 6 commits into
mainfrom
anit/out-4105-prismaclientinitializationerror

Conversation

@priosshrsth

@priosshrsth priosshrsth commented Aug 26, 2026

Copy link
Copy Markdown
Collaborator

The Sentry PrismaClientInitializationError burst (16 events, flushGroupedEmailRun) was the Supabase pooler being unreachable 14:30–15:17 on 8/24 — the same outage produced single events in task-notifications.service and notification.service. The unreachable DB isn't ours to fix; what it exposed is. OUT-4105

flushGroupedEmailOnFailure was DELETEing the whole window including unsent rows, so a transient outage destroyed buffered grouped emails — or orphaned them at sentAt IS NULL when the delete failed too. Either way nothing re-flushed them: bufferGroupedEmailEvent only reuses windows younger than 5 minutes.

What to look at:

  • flushGroupedEmailOnFailure — the delete is gone entirely. Sentry capture and logging are unchanged.
  • sweep-grouped-email-windows.ts — hourly. Prunes rows past 15 days, releases stale claims, then re-enqueues distinct windows still unsent after 30 minutes. The 24-hour upper bound on re-enqueue is deliberate: without it a genuinely un-sendable window would retry hourly for a fortnight and page every time.
  • claimUnsentWindowEvents is the important one. Reintroducing a recovery path means two runs can target one window, so the read is now an UPDATE ... RETURNING that claims the unsent rows through the existing batchId column. The losing run matches zero rows and no-ops. Worth checking the three ways a claim is released: a failed attempt releases its own (so the next Trigger.dev retry re-sends), a successful send flips sentAt, and the sweeper releases anything older than 30 minutes — well past the job's maxDuration: 60, so a release can never race a live send.
  • flushGroupedEmailRun was split so the claim wraps the dispatch in a try/catch; dispatchClaimedWindow is the old body unchanged.
  • One unrelated line: the flush retry budget went from ~3s (3 attempts, 1s/2s) to ~75s, so short blips self-heal before the sweeper is involved at all.

Verified: yarn tsc, yarn lint:check (0 errors), yarn prettier:check. Unit 181 passed — same 10 pre-existing failures (withErrorHandler, authenticate) as main, no new ones. Integration 19/19 against real Postgres, including a new sends once when two runs race the same window case; I confirmed it fails with 2 sends when the claim is reduced to a plain SELECT, so it isn't a test that passes by construction. Not verified: no staging run, so cron registration and the sweeper's SQL at production scale are untested outside the container.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Y4tF84tzJW7Bo1DehsDX22

…1397)

deleteLabel passed `id: currentLabel?.id` straight into label.delete, so when
findFirst matched nothing Prisma got `{ id: undefined }` and threw
PrismaClientValidationError, failing the whole delete transaction. Return early
instead.
* OUT-4093 | Remove the localStorage→localForage assignee migration
@linear-code

linear-code Bot commented Aug 26, 2026

Copy link
Copy Markdown

OUT-4105

@vercel

vercel Bot commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
tasks-app Ready Ready Preview Aug 26, 2026 5:44am

Request Review

@greptile-apps

greptile-apps Bot commented Aug 26, 2026

Copy link
Copy Markdown

Greptile Summary

This PR preserves grouped-email windows across temporary database outages by adding delayed retry rounds, round-scoped idempotency keys, and a larger in-run retry budget.

  • Re-enqueues failed windows twice at 15-minute intervals before terminal cleanup.
  • Adds the retry round to the task payload, idempotency key, logs, and terminal Sentry tags.
  • Extends unit coverage for retry scheduling and terminal cleanup.

Confidence Score: 4/5

The retry enqueue failure path should be fixed before merging because it can leave buffered emails permanently unscheduled.

The new recovery path improves database-outage handling when re-enqueue succeeds, but its only scheduling handoff can reject inside a non-retried failure hook, leaving no later job to recover the unsent window.

Files Needing Attention: src/jobs/notifications/flush-grouped-email.ts

Important Files Changed

Filename Overview
src/jobs/notifications/flush-grouped-email.ts Adds multi-round recovery for failed grouped-email flushes, but an enqueue rejection inside the failure hook can permanently orphan the window.
src/jobs/notifications/flush-grouped-email.test.ts Covers successful retry scheduling and terminal cleanup, but not rejection of the retry enqueue operation.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Flush exhausts task attempts] --> B{retryRound below maximum?}
  B -->|Yes| C[Trigger delayed retry round]
  C -->|Scheduled| D[Future flush reads unsent rows]
  C -->|Trigger request rejects| E[Failure hook exits]
  E --> F[Window remains unsent and unscheduled]
  B -->|No| G[Report terminal failure]
  G --> H[Delete window rows]
Loading

Reviews (1): Last reviewed commit: "OUT-4105 | Stop dropping buffered groupe..." | Re-trigger Greptile

retryRound,
error: serializeError(error),
})
await enqueueGroupedEmailFlush({ workspaceId, windowKey, retryRound: retryRound + 1 })

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Failed enqueue orphans the window

If the Trigger.dev enqueue request rejects, the failure hook exits before scheduling the retry or reaching terminal cleanup. Trigger.dev does not retry this hook, and no reconciliation job schedules old unsent windows, so the buffered emails remain permanently unsent.

Knowledge Base Used: Notification delivery workflows

… on failure

When the Supabase pooler went unreachable on 8/24 every flush window burned its
~3s retry budget on the first $queryRaw, and onFailure then DELETEd the whole
window — unsent rows included — so those grouped emails were destroyed, or
orphaned when the delete failed too. Nothing ever re-flushed them.

onFailure now leaves the rows alone, and an hourly sweeper re-enqueues any
window still unsent after 30 minutes. Rows are pruned after 15 days rather than
at the first sign of trouble. The flush run is already idempotent, so a
re-enqueue only sends what is still outstanding.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01Y4tF84tzJW7Bo1DehsDX22
@priosshrsth
priosshrsth force-pushed the anit/out-4105-prismaclientinitializationerror branch from dda4edc to 1c096b7 Compare August 26, 2026 03:40
@priosshrsth priosshrsth changed the title OUT-4105 | Stop dropping buffered grouped emails when the DB is briefly unreachable OUT-4105 | Sweep stale grouped-email windows instead of deleting them on failure Aug 26, 2026
@vercel

vercel Bot commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

Deployment failed for project tasks-app with the following error:

Deploying Serverless Functions to multiple regions is restricted to the Pro and Enterprise plans.

Learn More: https://vercel.link/multiple-function-regions

…er send twice

The sweeper can enqueue a window whose original flush is still pending, and two
runs reading "sentAt" IS NULL would both send. Replace the read with an
UPDATE ... RETURNING that claims the unsent rows via the existing batchId
column: the losing run matches nothing and no-ops.

A failed attempt releases its claim so the next Trigger.dev retry re-sends, and
the sweeper releases any claim older than 30 minutes — well past the job's 60s
maxDuration, so a release can never race a live send.

Covered by an integration test that runs two flushes concurrently against real
Postgres; it fails with two sends if the claim is reduced to a plain SELECT.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01Y4tF84tzJW7Bo1DehsDX22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant