Skip to content

feat(email): connect a user's Outlook mailbox and poll Inbox and Sent (EA-1) - #635

Open
privatejoel wants to merge 1 commit into
fix/secrets-per-value-ivfrom
claude/email-ea1-outlook-sync
Open

privatejoel wants to merge 1 commit into
fix/secrets-per-value-ivfrom
claude/email-ea1-outlook-sync

Conversation

@privatejoel

Copy link
Copy Markdown
Collaborator

Summary

EA-1 of the email accountability plan (#634, sections 5.1 and 6.1). A user can connect their own Outlook mailbox. The 5-minute tick then polls Inbox and Sent Items with Graph delta queries. This PR only counts messages. It makes no action and stores no mail. EA-3 adds the rules engine through the onPage hook.

Stacked on #619 (secrets v2). The tokens use its one-value format. Merge #619 first. GitHub then moves this PR to main.

  • Migration 267 adds mailbox_connections. Each row holds one mailbox, its encrypted tokens, the checkpoint for each folder, health counters and cost limits. RLS is on and has no policy for tenant users, so only the service role can read the tokens. The migration also adds three tenant_settings columns: email_accountability_enabled (default false), and google_oauth_client_id and google_oauth_client_secret_enc for EA-2.
  • _lib/mail/provider.js is the part of the sync that works for any provider:
    • typed MailError codes;
    • mailFetch, which counts each request against the mailbox's daily budget (a retry counts too), retries a 429, a 5xx or a network error at most 2 times, waits for Retry-After inside the run only up to 5 s, and keeps provider error bodies out of messages and logs.
  • _lib/mail/outlook.js is the Outlook adapter:
    • delegated scopes offline_access User.Read Mail.Read, on the tenant's existing app registration (graph_*, migration 028);
    • delta on inbox and sentitems with changeType=created, IdType="ImmutableId" and a receivedDateTime filter on the first request, so no round reads the full history;
    • In-Reply-To and References from MAPI properties 0x1042 and 0x1039;
    • a stored checkpoint that is not a Graph mail-folder URL is never fetched.
  • _lib/mail/sync.js is the runner. The rules are below.
    • A checkpoint moves only after its page is processed.
    • Inbox runs before Sent. If Inbox does not finish, Sent waits.
    • Each folder reads at most 3 pages of 50 messages. A run that stops at the page cap or at the deadline is not a failure: the mailbox is due again at the next tick, so a backlog drains.
    • After a failure, the next sync waits 2, 4, 8, then 16 poll intervals. After 10 failures in a row, the mailbox pauses until a person resumes it.
    • On a throttle, the mailbox pauses for Retry-After, from 1 to 60 minutes.
    • When the daily budget (default 1,500 requests) or a provider quota runs out, the mailbox pauses until 00:00 UTC.
    • An invalid checkpoint is cleared. The next round starts 1 hour before the last success, and at most 7 days back.
    • A refused token marks the mailbox reauth_required.
    • An admin alert (notifyAdmins) is sent when the state changes: a new sign-in is needed, the budget or quota runs out, the failure count reaches 3, or the mailbox pauses after failures. A repeated run sends no second alert.
  • /api/email/sync is the cron handler, in the tick ALWAYS group as email/sync. It reads only tenants with email_accountability_enabled. It syncs at most 8 due mailboxes in each run. A lease on each mailbox stops two overlapping ticks from syncing the same mailbox.
  • /api/email/connections:
    • GET: the caller's own mailboxes. An admin sees every mailbox of the tenant. The response has only safe columns: no token and no checkpoint.
    • POST: returns the authorize URL.
    • PATCH: the owner or an admin resumes a paused mailbox. Only an admin changes the limits.
    • DELETE: disconnects the mailbox and deletes its tokens and checkpoint immediately.
  • /api/email/connect/callback is the public OAuth callback. The state is bound to the tenant, the user, the provider and a purpose. Before the callback stores the mailbox, it does these checks:
    • The user must still be an approved member of the tenant.
    • The callback refuses an address that another user already connected.
    • The callback refuses a grant that has no refresh token.
  • graph-client.js: signState and verifyState accept a purpose. A mailbox state cannot finish the tenant connect, and a tenant state cannot finish a mailbox connect. buildAuthorizeUrl, exchangeCode and refreshTokens accept the scopes as an argument. The defaults do not change, so the tenant send connection works as before.

No new npm package and no paid service. Graph mail reads are not metered.

Deploy notes

  1. Merge fix(secrets): own IV per stored credential, no plaintext fallback, https Tally bridge #619 first.
  2. Apply supabase/migrations/267_mailbox_connections.sql by hand. Then run notify pgrst, 'reload schema';. The migration has VERIFY queries.
  3. In the tenant's Entra app registration (the app that graph_client_id names), do these steps:
    • Add a Web redirect URI: {PUBLIC_APP_URL}/api/email/connect/callback.
    • Add the delegated Microsoft Graph permissions User.Read, Mail.Read and offline_access.
    • If the tenant blocks user consent, grant admin consent.
  4. Nothing polls until an admin runs this SQL for the tenant: update tenant_settings set email_accountability_enabled = true where tenant_id = '<id>';. Each user then connects through POST /api/email/connections. The UI comes in EA-7.
  5. Polling needs a caller for /api/cron/tick every 5 minutes: cron-job.org, fix(cron): give the 5-minute tick a scheduler, behind an allow-list that defaults to extraction jobs only #566 or pg_cron. If fix(cron): give the 5-minute tick a scheduler, behind an allow-list that defaults to extraction jobs only #566 is merged, add email/sync to CRON_TICK_HANDLERS.
  6. No new environment variable. The code uses PUBLIC_APP_URL, ANVIL_SECRETS_KEY and CRON_SECRET.

Tests

New tests use test-memory-supabase and a mocked fetch. They use no live credentials.

src/v3-app/api-mail-outlook-sync.test.js (22 tests). It covers request tests [13], [14], [16] and [17]:

  • Inbox runs before Sent, from a bounded start, with changeType=created, immutable IDs and no isRead field. The test checks that both delta links are saved.
  • [13] An outage keeps the checkpoint and the next sync is 10 minutes later. The next run resumes from the saved delta link.
  • [13] An invalid checkpoint (HTTP 410) is cleared and is not counted as a failure. The next round starts 1 hour before the last success, and never more than 7 days back.
  • [13] A failure on page 2 keeps the checkpoint of page 1. The 3-page cap leaves Sent for the next run.
  • [14] An expired access token is refreshed with the read-only scopes, and the new tokens are stored encrypted.
  • [14] A revoked grant marks the mailbox reauth_required. No mail is read, and one alert is sent, also on later runs. A 401 from Graph does the same.
  • [16] A short Retry-After is retried at most 2 times, then the mailbox pauses for at least 1 minute. A long Retry-After is not waited for inside the run. A 4xx is not retried.
  • [17] A spent budget makes 0 requests, pauses the mailbox until 00:00 UTC and sends one alert. The count starts again on the next UTC day. A budget that runs out during a run stops at the limit and keeps the folder that finished.
  • The alert is sent when failures reach 3, and not again at 4. After 10 failures the mailbox pauses.
  • A run that is out of time makes no request. It is not a failure, and the mailbox is due again at once. The runner claims no mailbox in the last 3 s of its run.
  • A tenant that has not turned the module on is never polled. A mailbox with a lease is skipped. A checkpoint URL that is not a Graph mail-folder URL is never fetched.

src/v3-app/api-email-connections.test.js (14 tests):

  • The authorize URL has the read-only scopes and a mailbox state. The tenant callback refuses that state.
  • A viewer is refused (403), and an unknown provider is refused (400).
  • A user lists only their own mailboxes. An admin lists all mailboxes. No token or checkpoint is in a response.
  • Another user gets 404 for a resume and for a disconnect. The owner resumes. Only an admin changes the limits, and the limits are checked against their ranges. A mailbox in reauth_required cannot be resumed (409). A disconnect deletes the tokens and the checkpoint.
  • The callback stores encrypted tokens and a lower-case address, and writes an audit row. It refuses these cases: a tenant state, an address that another user connected, a user who left the tenant, and a grant with no refresh token. When the same owner connects again, the callback keeps the checkpoint and clears reauth_required.

Mutation check: I broke each rule, ran the two files, saw a failure, and restored the code. All 21 mutations failed the tests:

# Mutation Result
1 Sent runs when Inbox is unfinished killed
2 5 retries, not 2 killed
3 no budget check killed
4 any checkpoint URL accepted killed
5 state purpose not checked killed
6 list without the owner filter killed
7 no "address taken" check killed
8 a checkpoint reset counts as a failure killed
9 no 7-day floor killed
10 no lease condition killed
11 an alert on every failure killed
12 refresh with the send scopes killed
13 disconnect keeps the tokens killed
14 pause not clamped to 1 minute killed
15 tenant gate ignored killed
16 plaintext refresh token killed
17 a user who left the tenant can connect killed
18 no backoff on failures killed
19 no deadline check for each request killed
20 a deadline stop counts as a failure killed
21 no claim cut-off before the deadline killed

Gates:

  • npm run check: pass.
  • npm run lint: 0 errors. The new files add no warnings.
  • npm run build: pass.
  • npm run verify: pass. It rewrote docs/RBAC_AUDIT.md and docs/WRITE_PATH_AUDIT.md with changes that are not from this PR, so I restored the two files.
  • npm run eval:golden: pass (aggregate 1.000).
  • NODE_OPTIONS=--no-experimental-webstorage npx vitest run: 626 files, 7,484 tests, all pass.
  • I did not apply migration 267 to a Postgres. This machine has none. The CI migrations job applies it.

Not covered

  • Not tested on a live mailbox. I did not prove on a real mailbox that the MAPI properties 0x1042 and 0x1039 are present on Sent Items. EA-8 adds an onboarding check.
  • No action is made and no mail is stored. EA-3 does that.
  • Gmail is EA-2.
  • No UI. EA-7 adds the mailbox card. Until then a user connects through the API.
  • The owner of a mailbox gets no notice when it needs a new sign-in. Only admins get one, through notifyAdmins. Notices to one person need feat(notifications): a notice per person and a bell for every role, on desktop and phone #624.
  • shortcut: the runner syncs one mailbox after the other. About 8 mailboxes fit in one tick. Add bounded concurrency when a tenant has more mailboxes than one tick can sync.
  • shortcut: the token refresh is not counted against the request budget. It is a call to the login endpoint, not to the mail API.
  • The test fake test-memory-supabase.ts now accepts lt, and or() terms with lt, lte, gt and gte.

🤖 Generated with Claude Code

… (EA-1)

Migration 267 mailbox_connections (tokens as secrets v2, checkpoint per
folder, health and cost limits). Provider-independent mailFetch with a
daily request budget, bounded retries and a run deadline. Outlook adapter
on Graph delta with immutable IDs and a bounded first sync. Sync runner in
the tick ALWAYS group behind email_accountability_enabled. Connect, list,
resume and disconnect endpoints, and a purpose-bound OAuth state.

Counts messages only; EA-3 adds the rules engine.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@vercel

vercel Bot commented Oct 22, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
anvil Ready Ready Preview Oct 22, 2026 3:02am UTC

This branch was successfully deployed

1 active deployment
Preview — 1c7bde6f Deployed Oct 22, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant