[Silthus]

Moved from Silthus#565. Part of the code-first rebuild of the email send stack; replaces parts of #121272 (part 2, the cdp-api half).

Problem

• An app that renders its own emails, such as a password reset built with React Email, can't send them through PostHog. The send API only sends templates stored in PostHog.
• The Django half, feat(messaging): send an email with inline html from code #127123, accepts the caller's content. cdp-api has to deliver exactly that content, or deliver nothing.
• In the earlier stack, the content rode on the template route as an extra send field. A cdp-api without the change dropped the field and answered 202. It then sent nothing, or the stored template under the same key.
• Map: Silthus/posthog#461. Contract: Transactional email SDK contract for posthog-python and posthog-node #120921.
• Impact: Grow Workflows, through billed workflow email actions once the SDKs send code-rendered email. This PR only adds the delivery path; the route that opens it is behind the workflows-email-from-code flag.

Changes

Warning

🔴 Merge risk: high · two-way door once the email queue drains: reverting removes the route, and Django then answers 503 for code sends

• Behavior changes on the CDP email delivery path, where a wrong send is silent harm, so the grade is high even behind the flag.
• The worker now lays a queued run's state.send over the email step: content, sender, reply-to and capture choice. A wrong overlay sends the wrong content for a code send.
• Rollout order: turn workflows-email-from-code on only after cdp-api and both CDP workers (hogflow, email) run this code. A worker without it ignores state.send. At this head, an old email worker refuses a new key's code-send job and sends nothing, after Django answered 202. An earlier mixed-rollout case sent the stored template under a stored template's key.
• Revert: follow the rollback runbook below. It applies to any cdp rollback while the flag is on: flag off, drain the jobs that carry send, then roll back.
• The flag covers every way in. Only this route sets state.send, it needs a Django token with kind: code_send, and Django mints one only behind workflows-email-from-code. A rerun skips any stored run that carries send unless the request sets replay_code_sends, which feat(messaging): send an email with inline html from code #127123 sets from the flag. A code send through a template's workflow is covered too.
• Merge order: the Django half sits on master in its own chain and may land before or after this PR. Until cdp-api runs this PR, a code send answers 503 and sends nothing.
• Only the new route sets state.send. A template send takes the unchanged path, and the template route ignores a send field.
• The capture rule changes from "the send's choice, else the team switch" to "both must allow it". A send without a choice reads the team switch as before.

Kind	Files	Lines
Production	12	+417 −165
Tests	9	+759 −210

• A code send now goes out with the caller's content or not at all. cdp-api takes it on its own route, POST /api/projects/:team_id/hog_flows/:id/transactional_email/send, which requires send.
• A cdp-api without this route answers 404 (401 when an internal secret is set). Django maps both to 503, nothing sent.
• Each route takes only its own token. The code send route requires the claim kind: code_send, which feat(messaging): send an email with inline html from code #127123 mints only for a code send. The template route refuses any token with a kind. A token on the other route answers 401.
• A first send under a new key finds its workflow. Django creates the key's hidden workflow and posts at once. Both transactional routes and the hogflow and email workers now read a flow the cache misses from the primary database, so replica lag can neither refuse that first send nor drop it after the 202.
• Bounces and SES logs of that first send reach its workflow. Email feedback reads the primary for an id that is neither a workflow nor a live hog function on the replica. A failed lookup counts hog_flow_lookup_failed and keeps the workflows already found.
• Jobs of a workflow the database no longer has share one primary read per workflow, so a hard-deleted workflow's backlog costs one query, not one per job.
• The batched lookup, getHogFlowsCheckingPrimaryOnMiss, stays in this PR rather than its own. The hogflow worker and email feedback need it for a first send's workflow, and a split would ship this route without it.
• A reused idempotency key can't swap the content. On the send route the key also binds the send payload and the route. A retry with changed content, or the same key on the other route, answers 422 idempotency_key_reused. The template route keeps its fingerprint, so its existing keys still replay.
• The template route .../transactional_email keeps its schema and drops any send field, so a stored template send never picks up code content.
• A send field this cdp-api does not know is refused, not dropped. The send route answers 400 invalid_request for a key outside its schema, at the top of send (such as cc or attachments) or inside content, from or tracking, and queues nothing. feat(messaging): send cc, bcc and attachments on a code send #127988, the Django layer that adds copies and attachments, maps that 400 to invalid_request naming the field. A newer Django ahead of this cdp-api then fails loudly instead of sending a bare email.
• The internal API auth middleware exempts the new route by exact path, as it does the template route. Its handler checks the scoped JWT.
• The sendEmail hand-off lays state.send over the email step's rendered inputs: content, sender, reply-to. Recipients stay as the step renders them. The overlay is typed as the email queue parameters, so a renamed field fails to compile.
• Email tracking reads the capture choice from state.send.tracking. Nothing writes it into the hog function metadata any more.
• A step that resumes keeps the send's choices. The executor lays the run's state.send over a saved hog function state too, so a paused, throttled or retried step still honors capture_engagement: false.
• capture_engagement: false turns off the send's $workflows_email_sent, _failed, open, click and bounce events. The tracking code then carries no distinct_id, so feedback has no person to capture against. true never turns capture on over a team that turned it off.
• A rerun of a code send restores state.send only when the request sets replay_code_sends; otherwise it skips the run. A sensitive run's lifecycle snapshot drops it, as it drops the variables, so no rendered reset link is stored.
• A rerun that skips a sensitive run now says why, on that run's log. The warning reads "Not re-run: this run's message is sensitive, so its content wasn't stored and can't be sent again." Before, the workflow's log said only skipped=1. feat(messaging): send an email with inline html from code #127123 makes every code send under a key without a template sensitive.
• A recipient with =, ~, # or another symbol email allows before the @ now gets the email. Zod's default check refused [email protected], while the send path and the provider accept it.
• Both routes now check to with RECIPIENT_ADDRESS_PATTERN: a dot-atom local part and an ASCII domain with at least one dot. feat(messaging): send an email with inline html from code #127123 applies the same pattern, so Django refuses anything cdp-api would refuse.
• A malformed to, such as .jane@, an IP literal or a non-ASCII domain, still answers 422 recipient_unreachable, as fix(cdp): treat malformed transactional email recipients as unreachable #125908 set it.
• The code-send route shares the template route's handler, so it inherits master's behavior: it answers before the Kafka flush (fix(cdp): answer transactional email sends without waiting on kafka #125444), times each stage (feat(cdp): add transactional email latency and SES send metrics #125066), and refuses what the worker would skip with master's codes (feat(messaging): refuse transactional sends the email worker would skip #123019, fix(messaging): refuse a transactional email to a recipient that can't receive email #123878, fix(messaging): stop suppression refusals dropping transactional email #123492, fix(messaging): send transactional workflow email unless the address hard-bounced #124638). A refused code send never falls back to another provider (fix(email): stop falling back to customer.io for transactional emails #125322).
• Mechanical: names such as resolveEmailTrackingDistinctId and replay_code_sends carry the reasons; the code has no comments.

Rollback runbook

For the rollout skill. It applies to any rollback of cdp-api or the CDP hogflow or email workers while workflows-email-from-code is on for any organization, not only to a revert of this PR.

1. Turn workflows-email-from-code off. Django then answers 404 to a new code send, and a rerun skips every stored run that carries a code send, whichever workflow it went through.

2. Wait until this query returns 0 on the job queue database of every region:

   SELECT count(*) FROM cyclotron_jobs
   WHERE status IN ('available', 'running')
     AND (
       (queue_name IN ('hogflow', 'email') AND position('"send":{'::bytea IN state) > 0)
       OR (queue_name = 'rerun' AND position('"replay_code_sends":true'::bytea IN state) > 0)
     )

   The second condition counts reruns that started while the flag was on. They still replay code sends until they finish.

3. Roll back. A code-send job that an older worker picks up fails without sending, and keeps its send for a rerun after the roll forward.

How did you test this code?

Test seams: the cdp-api routes through supertest (cdp-api.serial.test.ts), the auth middleware (internal-api-auth.test.ts), the executor's email hand-off (hogflow-executor.service.test.ts), and the capture decision in email.service and email-tracking.service.

Test rationale:

• stamps %s on the queued run on the new route, and queues a template send without an accepted send, even when the request carries one: content leaking onto the template route, or not reaching the run.
• refuses a send on the accepted send route without the send: the new route silently sending a stored template.
• refuses a code send with a field this version does not know, and its content, sender and tracking variants: a key a newer Django adds, such as cc, dropped after a 202. Each fails without its .strict().
• authenticates every send on %s: the new route skipping the scoped JWT check.
• refuses a template send token on the code send route and refuses a code send token on the template send route: a token minted for one route queueing a send on the other.
• The middleware rows for both transactional email routes: the new route answering 401 in any deploy with an internal secret. The live run found this one; Jest runs without an internal secret.
• The capture tables: a send's true overriding a team that turned capture off.
• rejects the key for a request with changed content and ... with the other route: a reused key replaying a different email.
• runs a flow the replica does not have yet, and fails one missing from both: a first send under a new key refused by a lagging replica.
• reads the primary once per batch of workflows the database no longer has, and again once the miss expires: a deleted workflow's backlog sending one primary query per job.
• writes a hog_flow log entry for a bounce that resolves to a flow only on the primary, with and without a failing hog function lookup: feedback for a workflow in replica lag dropped.
• counts a failed primary lookup as a lookup failure for a bounce on a flow only the primary has: a failed primary read going uncounted.
• The capture table's row for a saved state without send: a resumed step capturing engagement after the caller turned it off.
• skips a run with %s: a rerun replaying a code send while the flag is off, or skipping a sensitive run with no log line on it.
• The executor's email routing tests send through a mocked SES, not the shared maildev. A real send there raced email.service.test.ts clearing the same inbox in one CI shard, and maildev restarted.
• answers a send to %s gains 13 address rows, the same table feat(messaging): send an email with inline html from code #127123 tests: a valid address refused by the recipient check, or a dot that leads, trails or repeats let through.
• replays a keyed send whose workflow stopped being active: the test stores status = 'draft' and runs the reload, so a manager that stops reading the stored status fails it.

At the head eb26545, with Django at #127123's 213fad6: a code send delivers once, and its status reads sent. Master's reader parses a capture-off tracking code and captures no event for it. The recipient check accepts and refuses the same 23 sample addresses as Django's.

Addresses, live at the final head 3639e5d with Django at #127123's 7571429: code sends to [email protected] (Reply-To o'[email protected]) and to an address with every symbol email allows, and a template send to [email protected], each deliver once. An IP-literal to and a leading-dot reply_to answer 400 in Django, and cdp-api gets no request.

The other proofs ran at 2eac306, with Django at #127123's 31dc832 to 5ec28fe. Since 2eac306 only the recipient pattern and its test rows changed:

• Live run: cdp-api, both workers and a second reader at this head, maildev and an SES recorder as sinks. An idempotent replay, a template send, capture on and off on both senders, all three from fallbacks, and two environments under one key each deliver as sent.
• Capture off: the tracking code carries no distinct_id, and redirect, pixel and SES callbacks to both cdp-apis capture 0 events.
• A resumed step: a capture-off code send whose step state was saved without send completes, delivers the caller's content, and captures 0 events.
• Rerun gate: a code send through a stored template's workflow fails at its email step. A rerun with replay_code_sends false skips it and queues nothing. With it true, the rerun sends it once, with the caller's content.
• Cross-version: with this cdp-api and master's workers, master's email worker refuses code-send jobs and sends nothing. With master's cdp-api and these workers, a code send answers 503 and queues nothing. Template sends deliver in both orders, and both versions read their tracking codes.
• Shadow diff against the merge base f4ee0e5: 50 tests move from fail to pass, each a behavior above. No test that passes on base fails here.
• Sensitive rerun, live at ccdd9ff: a code send under a sensitive workflow delivers once and stores no content. A rerun skips it with the warning on that run's log and queues nothing. A control send that is not sensitive replays once. 20fb623 changes only the warning's wording.
• Prod shape: the primary reads stay rare, and the runbook's drain query, reruns included, runs on the production job queue database.

Proofs at earlier heads

• 34347b2: the strict schema, live. Every send form master's Django and feat(messaging): send an email with inline html from code #127123's c4b7fb2 make answers 202 and delivers once. A send carrying cc or attachments answers 400 naming the field.
• abbf035: a first code send under a new key during replica lag delivers one mail, and the primary-read shadow diff and prod shape.
• b5889c9 and the pre-rebase 4182790: the first live runs, cross-version runs and the mixed-rollout run that set the deploy order in the merge-risk callout.

Release status

• This change is behind a feature flag and is not available to users

Docs update

• None. The docs land with the Django route.

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

Agent: Claude Code, Claude Opus 5.5 (claude-opus-5-5)

Skills: /tdd, /writing-tests, /writing-code-comments, /writing-pr-descriptions, /pr-description, /stacking-prs, /recreating-fork-prs, qa-swarm

Question	Answer	Who	Where
How does a code send fail on an old cdp-api?	Own route, so an old cdp-api answers 404 and Django answers 503 with nothing sent	👤 Human	map Silthus#461, 2026-10-25
What does capture_engagement do against the team switch?	A send can turn capture off, never on; default true	👤 Human	map Silthus#461, 2026-10-25
Which flag gates the code-first send?	A new workflows-email-from-code, per organization; master's surfaces keep workflows-transactional-email-api	👤 Human	map Silthus#461, 2026-10-25
May the code carry comments?	No; names carry the reasons	👤 Human	map Silthus#461, 2026-10-25
Carry the capture choice by leaving distinctId out instead of tracking code v3?	Yes; a capture-off send stops its opens, clicks and bounces at once, with no deferred flip	🤖 Agent	qa-swarm round 2, 2026-11-01
Should SES feedback read the primary on a cache miss too?	Yes, in this PR; a final read found a first send's bounces dropped during replica lag	🤖 Agent	final read 6, 2026-11-03
Refuse a send field the schema does not declare?	Yes, .strict(); a cross-version run found cc, bcc and attachments dropped after a 202; nested objects too, per final read 9	🤖 Agent	map conductor, 2026-11-03
Rollback while the flag is on?	Accepted with a runbook: flag off, drain send jobs, then roll back; the flag must gate every entry point	👤 Human	map Silthus#461, 2026-11-03
Clean up the overlay?	Yes: move it into the email path, type it as the email queue parameters, read capture from state.send, explain the batched lookup	👤 Human	map Silthus#461, 2026-11-03
Make a code send under a key without a template sensitive?	Yes; its content stays out of the saved run history, and a rerun skips it with a log line saying why	👤 Human	map Silthus#461, 2026-11-03
Gate the rerun of a code send on the flag?	Yes; the rerun request carries replay_code_sends from the flag, so a code send through a template's workflow is covered too	🤖 Agent	map conductor, 2026-11-03
Keep send on a resumed step?	Yes; the executor merges the root state.send into a saved state, so the root state stays the source	🤖 Agent	map conductor, 2026-11-03
Keep the batched lookup in this PR?	Yes; a split ships the route without the first-send read and resets the proofs	🤖 Agent	map conductor, 2026-11-03
Keep the looser to pattern after the rebase?	No; #125908 refuses a malformed to as recipient_unreachable, and that holds	🤖 Agent	map restack, 2026-10-31
Loosen to for valid addresses Zod refused?	Yes, to a dot-atom pattern Django shares; only Zod refused them, and a dot that leads, trails or repeats stays refused	🤖 Agent	map conductor, 2026-11-03; final read 13

Involvement: one brief from Michael with decisions; QA swarm rounds 1 and 2, seventeen final reads, a full two-family panel and a narrow round (GPT-6.1 Sol reviews, Opus 5.5 fresh eyes); shadow diff, live, cross-version, prod-shape and mixed-rollout proof captured.

More context for AI agents

• Session: map Map: send code-managed email templates with one PostHog call Silthus/posthog#461, Claude Code; the reviewed fork SHA 920e7c9 was pushed here unchanged, signed by its author.
• Decisions: rebased onto master 43333f8 before the move, then merged master 2468be9, d8cbc31, b8095bf, 47f64f0 and 99e1a37; the node diff of the reviewed fork stack moved over as is, and the own route is new.
• StampHog at eb26545, via feat(messaging): send an email with inline html from code #127123's review: a code send token still passed the template route; 6f4136c makes the template route refuse any kind, with tests both ways.
• Follow-ups: the distinctId simplification, before flag-on.
• Read closely: nodejs/src/cdp/services/hogflows/hogflow-manager.service.ts, the overlay in async-functions/send-email.ts, and the code_send claim on the send route.

🤖 Generated with Claude Code

[trunk-io]

😎 Stack merged successfully - details.

[github-actions]

🤖 CI report

✅ Trunk lane — non-backend lane

This PR is assigned to the non-backend lane. It does not run backend Python tests and may merge in parallel with PRs in other lanes.

✅ Duplication (Python) — clean

New Python code duplication introduced by this branch. Fails at 70+ tokens in app code, or 150+ tokens when both copies live in test files. Advisory while the gate proves itself: extract a shared helper instead of copying.

✅ Duplication (TypeScript) — clean

New TypeScript code duplication introduced by this branch. Fails at 70+ tokens in app code, or 150+ tokens when both copies live in test files. Advisory while the gate proves itself: extract a shared helper instead of copying.

✅ Hogbox preview — ready, open the preview

▶ Open the preview

🔑 Login	[email protected] / 12345678 (demo data)
🧩 Running	this PR's backend on the PostHog :master base (frontend unchanged by this PR)
🔗 Link	stable across rebuilds: a re-push swaps the box underneath, the URL stays
🔒 Access	tailnet only (PostHog VPN)
🛠️ Admin	inspect & debug state in hogland
💤 Idle	sleeps after ~30 min idle (snapshot to S3, zero node cost) and wakes on your next visit in ~30s, behind a brief "waking up" screen

_{commit 6f4136c · box box-8715ff5431f5 · ready in 1856s (push → usable) · build log · rebuilds on every push, torn down on close}

[posthog]

PostHog Review · Standard · reviewed 920e7c9: 1 Should fix.

This is the only automatic review while this pull request is a draft. Comment @posthog review to review new changes sooner. Otherwise, a review of the new changes runs when you mark it ready for review.

See this review and its findings in PostHog Review. Comment @posthog review --deep for a Deep review. A Deep review takes longer and looks deeper than a Standard review.

[hosthog]

HostHog preview — posthog-desktop-web

The previews for this PR have been torn down and no longer serve.

[stamphog]

Not approved — this change needs a human reviewer.

Re-adding the stamphog label gives the same result unless the changed files change.

Stamphog can't approve this pull request automatically because two gates refused it. The deny-list gate flagged it for matching the auth category, which fits the changes to internal-api-auth.ts and the new kind: code_send token claim on the transactional email route in cdp-api.ts. The tier gate classified it as T2-never (1620 lines, 24 files, a feature touching the email delivery path), and that tier is never auto-approved.

The size gate passed, so splitting the PR wouldn't clear these refusals. Ask a human reviewer, ideally someone familiar with CDP email delivery and token auth, to review it.

Gate mechanics and policy version

Gate		Result
prerequisites	✓	all clear
deny-list	✗	matches: auth
size	✓	598L, 14F substantive, 1620L/24F incl. docs/generated/snapshots — within ceiling
tier	✗	classified as T2-never: T2-never (1620L, 24F, single-area, feat)
stamphog 2.7.0		.stamphog/policy.yml @ unknown · reviewed head eb26545

[stamphog]

Not approved — this change needs a human reviewer.

Re-adding the stamphog label gives the same result unless the changed files change.

stamphog can't auto-review this pull request because two gates refused it. The deny-list gate flagged it for matching auth, which fits the changes to nodejs/src/common/api/middleware/internal-api-auth.ts and the new token-claim checks in cdp-api.ts. The tier gate classified it as T2-never (1630 lines, 24 files, feat), so it always needs a human reviewer. The size gate passed, so splitting the PR wouldn't change this outcome.

Please ask a human reviewer, preferably someone who owns the CDP email path or the internal API auth, to review it.

Gate mechanics and policy version

Gate		Result
prerequisites	✓	all clear
deny-list	✗	matches: auth
size	✓	602L, 14F substantive, 1630L/24F incl. docs/generated/snapshots — within ceiling
tier	✗	classified as T2-never: T2-never (1630L, 24F, single-area, feat)
stamphog 2.7.0		.stamphog/policy.yml @ unknown · reviewed head 6f4136c

[Silthus]

/trunk merge

[Silthus]

/trunk merge

[deployment-status-posthog]

Deploy status

Environment	Status	Deployed At	Workflow
dev	✅ Deployed	2026-11-04 12:07 UTC	Run
prod-us	✅ Deployed	2026-11-04 12:23 UTC	Run
prod-eu	✅ Deployed	2026-11-04 12:25 UTC	Run