From 452ca6e0a94d1343ba0a1cac6c4c56bab53de683 Mon Sep 17 00:00:00 2001 From: "Reichenbach, Michael" <755327+Silthus@users.noreply.github.com> Date: Wed, 4 Nov 2026 07:26:43 +0000 Subject: [PATCH 01/18] feat(skills): move SMTP sends to PostHog in every SDK language Adds the smtp variant of migrate-email: Nodemailer and the SMTP clients of each language with a PostHog SDK, the hosts that name the provider behind them, and how DNS and suppressions follow that provider. Co-Authored-By: Claude Opus 5.5 --- context/skills/migrate-email/config.yaml | 3 + .../migrate-email/references/smtp/account.md | 60 ++++ .../migrate-email/references/smtp/mapping.md | 162 ++++++++++ .../references/smtp/sdk-reference.md | 281 ++++++++++++++++++ scripts/lib/tests/migrate-email-skill.test.js | 2 +- 5 files changed, 507 insertions(+), 1 deletion(-) create mode 100644 context/skills/migrate-email/references/smtp/account.md create mode 100644 context/skills/migrate-email/references/smtp/mapping.md create mode 100644 context/skills/migrate-email/references/smtp/sdk-reference.md diff --git a/context/skills/migrate-email/config.yaml b/context/skills/migrate-email/config.yaml index 1d949636..c756dfe1 100644 --- a/context/skills/migrate-email/config.yaml +++ b/context/skills/migrate-email/config.yaml @@ -19,3 +19,6 @@ variants: - id: sendgrid display_name: SendGrid tags: [sendgrid] + - id: smtp + display_name: SMTP + tags: [smtp] diff --git a/context/skills/migrate-email/references/smtp/account.md b/context/skills/migrate-email/references/smtp/account.md new file mode 100644 index 00000000..acd8a004 --- /dev/null +++ b/context/skills/migrate-email/references/smtp/account.md @@ -0,0 +1,60 @@ +# SMTP account + +What the migration needs from the account behind the SMTP host. SMTP is only how the app hands mail over: the provider that runs the host, found from the table in `sdk-reference.md`, holds the DNS records and any suppression list. The agent never changes that account: it reads what the person exports or tells it. + +Name that provider in the plan, such as `SMTP host: smtp.mailgun.org (Mailgun)`. When the host comes from a variable with no value in the repository, ask the person which provider runs it. The steps say "the provider" for it. + +## DNS records + +A domain that sends through an SMTP provider carries that provider's records. Keep all of them until the cutover is done, and longer when the domain also gets or sends normal mail. The provider's dashboard shows the exact names, under its domain or sender settings. + +| Record | Usual name | What it does | +|---|---|---| +| `TXT` (SPF) | The domain, or the provider's return-path label | Names the servers that may send for the domain, with an `include:` of the provider | +| `TXT` or `CNAME` (DKIM) | `._domainkey` | Signs the provider's mail. The selector is the provider's own | +| `CNAME` or `MX` | A return-path or bounce label, such as `bounce`, `pm-bounces`, or `em` | The provider's MAIL FROM, where its bounces go | +| `CNAME` | A tracking label | Link and open tracking, only when it is set up | +| `MX` | The domain | Where the domain's own mail arrives, when it has mailboxes | +| `TXT` (DMARC) | `_dmarc` | Shared by every provider on the domain | + +- PostHog's sender uses its own label for its MAIL FROM, `feedback` by default. Never give PostHog a label the provider already uses. Check the domain's records for that label first. +- PostHog's DKIM records have their own selector names, so they sit next to the provider's `_domainkey` records. +- Keep the one `_dmarc` record. Never add a second one. +- A domain whose SPF record names the provider's `include:` keeps it. Merge PostHog's include into that record: a domain has one SPF record. +- **Gmail, Google Workspace, Microsoft 365, and other mailbox providers.** The domain's MX, SPF, and DKIM records also carry the person's normal mail. They stay after the cutover. Never suggest removing one. +- **A server the person runs.** Its records are what the person set up, such as an `A` record for the host, an SPF entry for its address, and its own DKIM selector. Keep them, and ask the person when you can't tell which labels it uses. + +## Read-only key + +A read key is usually not needed. The SMTP password sends mail and reads nothing. The templates are in the app's code, and the suppression list, when the provider keeps one, comes from its dashboard export. Don't ask for a key to the provider's API: this skill has no reference for it, and the steps never need one. Note in the plan `full-access provider key: none created`, so Step 5 and the report have no key to delete. + +The steps call this key the full-access provider key. + +## Template content + +SMTP stores no templates, so there is no template content to read from the provider. The content of each site is in the app's code, or in its template files, such as Handlebars views or `email-templates` folders. Read those in the repository. + +A template the provider renders because a header names it stays with its site, so its content needs no check. + +## Suppression list + +Whether there is a list depends on the provider behind the host. + +**Providers that keep one.** Sending services keep a list of addresses that hard-bounced or complained, and skip them on every send: SendGrid, Mailgun, Postmark, Resend, Brevo, Mailjet, SparkPost, Mandrill, and Amazon SES. Ask the person to export it from the provider's dashboard as a CSV file, and to tell you where they saved each file. Read the header row to find the address column and the column that says why each address is on the list. Don't read it from the provider's API: this skill has no key for it. + +**Providers that keep none.** Gmail, Google Workspace, Microsoft 365, Zoho Mail, and other mailbox providers, and a server the person runs, keep no list to export. Record the list as empty: `- [x] suppressions — empty list: keeps no suppression list`. When the app keeps its own list of bounced addresses, such as a column it sets from bounce mail, that list stays in the app and its sends keep checking it. + +**Map each entry.** + +| Entry on the provider's list | PostHog source | Reason | +|---|---|---| +| Hard bounce, permanent failure, invalid or unknown address | bounce | `Imported from : bounce` | +| Spam complaint or spam report | complaint | `Imported from : spam complaint` | +| An address the person added by hand | manual | `Imported from : manual` | + +`` is the provider behind the host, such as `Imported from Mailgun: bounce`. When the export doesn't say why an address is on the list, ask the person which kind the file holds. + +**Leave out the other entries.** + +- **Soft bounces, deferrals, and blocks** are temporary refusals by a receiving server. The provider retries those addresses, and so does PostHog. +- **Unsubscribes** are marketing preferences, and PostHog's opt-outs apply to marketing messages only. Some providers also stopped transactional mail to them. Once the sends move, these addresses get transactional email again. Count them, and tell the person the count in the report. diff --git a/context/skills/migrate-email/references/smtp/mapping.md b/context/skills/migrate-email/references/smtp/mapping.md new file mode 100644 index 00000000..f4c7652a --- /dev/null +++ b/context/skills/migrate-email/references/smtp/mapping.md @@ -0,0 +1,162 @@ +# SMTP to PostHog mapping + +This file decides, field by field, whether PostHog can send an SMTP call site. A site moves only when every field it sets moves. Otherwise it stays on SMTP, and the reason names the first field or feature that keeps it there. `api.md` has the PostHog side in full, and `posthog-sdks.md` has PostHog's names in each language. + +Fields are written in their Nodemailer names here, such as `replyTo`. A site in another language sets the same part of the message under another name: `msg["Reply-To"]` in Python, `addReplyTo` in PHPMailer, `message.ReplyTo` in MailKit. `sdk-reference.md` lists the part each field sets. PostHog fields are written in their Node.js names: use the name `posthog-sdks.md` gives for the site's language, such as `reply_to` in Python. + +Fields set in the transporter's `defaults`, the second argument of `createTransport`, are fields of every site that sends through that transporter. So are the settings of a `Mail.defaults` block or a shared client object. + +## Fields + +| SMTP | PostHog | A site stays on SMTP when | +|---|---|---| +| `from` | `from` | Never. `{ name, address }` becomes `'Name
'`. It needs a verified sender (Step 4) | +| `to` | `to` | It holds more than one address, including a comma-separated string. A list with one entry moves as that entry. `{ name, address }` or `'Name
'` moves as the bare address: note in the plan row that the To line loses the name | +| `subject` | `subject` | Never | +| `text`, `html` | `text`, `html` | The value is a stream, an `href`, or a `path` that is a URL. A string moves as it is, and a `Buffer` as its UTF-8 string. A `path` to a file in the repository moves as the file's content, read in the moved code | +| `cc`, `bcc` | `cc`, `bcc` | The installed SDK has no `cc` or `bcc` (Step 7 checks), there are more than 10 of either, or the code builds the list at run time and you cannot tell it stays at 10 or fewer. `{ name, address }` moves as the bare address, as for `to` | +| A recipient in the envelope only, such as an address in Python's `sendmail` `to_addrs` that no header names | `bcc` | As for `bcc` | +| `replyTo` | `replyTo`, one address or a list | An entry carries a name, there are more than 10, or the code builds the value at run time and you cannot tell that it holds bare addresses only | +| `attachments` | `attachments` | The installed SDK has no `attachments`, there are more than 10 files, the files can add up to more than 512 KB, such as an upload of the user's or a PDF of unbounded length, or an entry has anything but a `filename`, a `content`, a `path` to a file, a `contentType`, a `cid`, and an `encoding` of its string content. An `href`, a `path` that is a URL or a `data:` URI, a stream `content`, `raw`, `headers`, and an `inline` disposition without a `cid` keep it. See "Attachments by path" below | +| `headers` | `headers` | The installed SDK has no `headers`, or there are more than 10. This includes every header the code sets on a MIME message, such as `msg["X-Entity-Ref-ID"]` or `addCustomHeader` | +| `list` | `headers` | As for `headers`. `List-Unsubscribe` is a header too | +| `priority` | none | Always | +| `sender` | none | Always | +| `inReplyTo`, `references`, `messageId`, `date`, `xMailer`, `otp` | `headers` | As for `headers` | +| `watchHtml`, `amp`, `alternatives` | none | Always | +| `icalEvent` | none | Always. An `.ics` file sent only as an attachment moves as an attachment | +| `envelope` | none | Its `from` or recipients differ from the `from`, `to`, `cc`, and `bcc` of the message, such as a bounce address of its own. An `envelope` with the same addresses moves without it | +| `raw`, a message the code writes by hand, `net/smtp`, `Net::SMTP`, or a `gen_smtp` body that is not `:mimemail.encode` output | none | Always. The fields are in a string the code built, and PostHog takes fields | +| `dsn` | none | Always | +| `dkim` on the message or the transporter | none | Never. PostHog signs with its own DKIM records, which Step 4 sets up | +| `attachDataUrls` | none | The `html` holds a `data:` image | +| `encoding`, `textEncoding`, `disableFileAccess`, `disableUrlAccess`, `pool`, `secure`, and other connection options | none | Never. They say how Nodemailer writes and sends the message, and PostHog's send carries no MIME and opens no connection | +| A `compile` or `stream` plugin on the transporter, `transporter.use(...)` | none | The plugin is not one of the template renderers under "Templates". The moved send would skip it | + +PostHog's `from` takes `'Name
'`. PHPMailer's `setFrom(address, name)`, MailKit's `MailboxAddress(name, address)`, and the other clients' name and address pairs become that form. + +A send through SMTP has no idempotency key, so the moved site sends none either. PostHog's SDK makes one for each call and reuses it on its own retries. + +### Attachments by path + +PostHog takes an attachment's content, not a path. An attachment by `path`, or PHPMailer's `addAttachment($path)`, gomail's `Attach(path)`, go-mail's `AttachFile(path)`, or the `mail` gem's `add_file '/path'`, moves when the path names a file on the app's own disk: the moved code reads the file into `content` before the send. It stays when the path is a URL or a `data:` URI, or when you cannot tell that the files stay at 512 KB or less in all, such as a file the user uploaded. A file in the repository has a size you can check. A file the app writes just before the send, such as a rendered PDF, has the size the code gives it: follow the code that writes it. + +## Calls + +| SMTP call | A site stays on SMTP when | +|---|---| +| Nodemailer `transporter.sendMail(message)`, with a promise or a callback | A field above keeps it | +| Python `send_message(msg)` and `sendmail(from_addr, to_addrs, msg)` with a message built with `EmailMessage` or `email.mime`, and `aiosmtplib.send` | A field above keeps it. A message string the code wrote by hand always stays | +| Ruby `mail` gem `deliver` | A field above keeps it | +| PHPMailer `send()`, and `symfony/mailer` `$mailer->send($email)` used on its own | A field above keeps it | +| Go gomail and go-mail `DialAndSend` | A field above keeps it | +| Go `net/smtp` `SendMail`, Ruby `Net::SMTP` `send_message` | Always: the message is a string the code built | +| MailKit `Send` or `SendAsync`, `System.Net.Mail` `Send` or `SendMailAsync` | A field above keeps it | +| Elixir `:gen_smtp_client.send_blocking` and `send` with a `:mimemail.encode` body | A field above keeps it | +| Rust `lettre` `send` | A field above keeps it | +| A send that names several recipients in `to`, or sends one message to a list in one call | Always. A loop that calls the send once per recipient is one site whose `to` is one address, and it moves | +| A framework mailer: Django `send_mail` or `EmailMessage` with the SMTP backend, Flask-Mail, Flask-Mailman, Rails Action Mailer, Laravel `Mail`, Symfony Mailer through the framework, Swoosh, Bamboo, NestJS `MailerService` | Always. PostHog has no mailer driver | +| A send in a language whose PostHog SDK has no email send (`posthog-sdks.md` lists them), such as Java's Jakarta Mail | Always | + +A Nodemailer transporter with a non-SMTP or third-party transport is not an SMTP site: `sdk-reference.md` says so under "Not SMTP". List its sends among the other calls. + +## Code around the send + +SMTP clients report an error in two ways. Node.js, Python, Ruby, PHPMailer with exceptions on, Symfony Mailer, and .NET throw. PHPMailer with exceptions off answers `false`, and Go, Elixir, and Rust answer an error value. `posthog-sdks.md` says how PostHog's SDK reports an error in each language. Keep the site failing the same way it did, as the Rewrite section says. + +| Code | A site stays on SMTP when | +|---|---| +| A `catch`, `except`, or `rescue` of the client's error, or a check of its error value | It branches on one SMTP code or error code, such as `err.responseCode === 550`, `err.code === 'EENVELOPE'`, `e.smtp_code`, or `SmtpStatusCode.MailboxUnavailable`. SMTP codes have no PostHog counterpart. A branch on any error moves | +| A check of `info.rejected`, `info.accepted`, or Python's dict of refused recipients | Never, when the site sends to one recipient: a refused recipient becomes PostHog's error. Read PostHog's error instead | +| `info.messageId`, `getLastMessageID()`, `message.MessageId`, or the Message-ID stored for a webhook handler of the provider behind the host, or a status read | Always | +| The Message-ID logged or stored for the app's own use | Never. Use PostHog's message id | +| The connection code: `createTransport`, `smtplib.SMTP(...)`, `starttls`, `login`, `ConnectAsync`, `AuthenticateAsync`, `Disconnect` | Never. A moved site needs no connection | + +## Templates + +SMTP stores no templates, so no site here has `template` content. The app renders its HTML in its own code, and the site sends it as `html` or `text`: the code form fits. Classify such a site by its other fields. Step 3's question about templates stored at the provider has nothing to ask for SMTP. + +A site whose send names a template stored at the provider behind the host through a header, such as SendGrid's `X-SMTPAPI` with a template id, stays for its header. + +Two Node.js packages render inside the send. Their sites move when the moved code renders the same content first: + +- **`nodemailer-express-handlebars`.** A site that sends `template` and `context` moves. The moved code renders the view with the plugin's own options, `viewEngine`, `viewPath`, and `extName` (`.handlebars` when unset), and sends the result as `html`, as the Rewrite section shows. A `text_template` renders into `text` the same way. Keep the plugin on the transporter while any site that stays uses it. +- **`email-templates`.** A site moves when the `Email` config sets `send: true` and `preview: false`. The moved code calls `email.renderAll(template, locals, message)` and sends its `subject`, `html`, and `text`. The config's `message` fields, such as `from` and `attachments`, are fields of the site too. When the config leaves `send` or `preview` to their defaults, the site stays, with the reason `email-templates sends only outside development and test`: the moved send would send from every environment. + +Rendered content is classified the same way as any `html`: it moves as it is. + +## Webhooks + +SMTP has no webhooks, so moving a site stops no SMTP event. The provider behind the host may post delivery events to a handler of the app (`sdk-reference.md`, "Webhooks"). That handler stops getting events for a site that moves. PostHog records the events of a moved send on the person it names with `distinctId`, when engagement events are on for the project: + +| Event of the provider behind the host | PostHog event | +|---|---| +| Sent, processed, or accepted | `$workflows_email_sent` | +| Delivered | `$workflows_email_delivered` | +| Bounced, hard or permanent | `$workflows_email_bounced` | +| Spam complaint | `$workflows_email_blocked` | +| Opened | `$workflows_email_opened` | +| Clicked | `$workflows_email_link_clicked` | +| Dropped or suppressed | none. PostHog refuses the send instead, such as with `recipient_suppressed` | +| Deferred, soft bounce, unsubscribe, and other events | none | + +PostHog's events carry no Message-ID of the SMTP message, no id of the provider behind the host, and no header the send set. A site whose events a handler matches by the Message-ID, by that provider's message id, or by a header stays on SMTP. For every other handler, the report tells the person which events it handles, which PostHog events take their place, and that a webhook destination in PostHog (Data pipelines, then Destinations) can post them to an endpoint. The handler itself keeps working for the sites that stay. Never change it. + +When the app has no such handler, the report says that SMTP sends no delivery events, and that the moved sends get PostHog's events on the person. + +## Rewrite + +How a site that moves is written with PostHog's SDK. Step 7 says when to use each part, and `posthog-sdks.md` has the client and the send in each language. + +**The PostHog client.** Set `migratedFrom: 'smtp'` on the client that Step 7 configures, when its client options take it. + +**The call.** The SMTP send and its connection become one PostHog send. Build its fields from the message the site built: + +- Nodemailer `sendMail(message)`: the fields of `message`, with the transporter's `defaults`. +- Python `EmailMessage`: `msg["From"]`, `msg["To"]`, `msg["Cc"]`, `msg["Bcc"]`, `msg["Reply-To"]`, and `msg["Subject"]`; `set_content(text)` is `text`, `add_alternative(html, subtype="html")` is `html`, and each `add_attachment(data, maintype, subtype, filename=)` is an attachment. With `MIMEMultipart`, each `MIMEText(body, "plain")` and `MIMEText(body, "html")` part is `text` and `html`, and each part with a `Content-Disposition` header is an attachment. Recipients in `sendmail`'s `to_addrs` that no header names are `bcc`. +- Ruby `mail`: `from`, `to`, `cc`, `bcc`, `reply_to`, `subject`, `text_part`, `html_part`, and `body`, which is `text` unless its content type is `text/html`. +- PHPMailer: `setFrom`, `addAddress`, `addReplyTo`, `Subject`, `Body` (`html` after `isHTML(true)`, else `text`), and `AltBody` (`text`). Symfony `Email`: `from`, `to`, `replyTo`, `subject`, `html`, and `text`. +- Go gomail: `SetHeader("From")`, `SetHeader("To")`, `SetHeader("Subject")`, `SetHeader("Cc")`, `SetBody`, and `AddAlternative`, by content type. go-mail: `From`, `To`, `Cc`, `Bcc`, `ReplyTo`, `Subject`, `SetBodyString`, and `AddAlternativeString`. +- MailKit: `From`, `To`, `ReplyTo`, `Subject`, and the `BodyBuilder`'s `TextBody` and `HtmlBody`. `System.Net.Mail`: `From`, `To`, `ReplyToList`, `Subject`, and `Body` (`html` when `IsBodyHtml`, else `text`), with a `text/html` or `text/plain` `AlternateView` as the other one. +- Elixir `gen_smtp`: the `From`, `To`, and `Subject` headers and the `text/plain` and `text/html` parts of the `:mimemail.encode` tuple. +- Rust `lettre`: `from`, `to`, `reply_to`, `subject`, and the body, or the plain and HTML bodies of `MultiPart::alternative_plain_html`. + +**The fields.** + +- `{ name: 'Example', address: 'hello@mail.example.com' }` and every other name and address pair in `from` becomes `'Example '`. A `from` without a name becomes the bare address. +- `to`, `cc`, and `bcc` become bare addresses. A list with one entry in `to` becomes that entry. When a helper takes a list of recipients and every caller passes one address, change the parameter to one address and update the callers. Never send one email per address in a loop. +- A `Buffer` in `text` or `html` becomes its string. A `path` to a file becomes the file's content, read with the same encoding. +- An attachment `{ filename, content, contentType, cid }` becomes `{ filename, content, contentType, contentId: cid }`. A `Buffer` content stays a `Buffer`. A string content with `encoding: 'base64'` stays the same base64 string, and a string content without `encoding` becomes `Buffer.from(content)`. Drop `contentDisposition: 'attachment'`: `contentId` makes a file inline. +- An attachment by `path` becomes `{ filename, content: await readFile(path), contentType }`, with `readFile` from `node:fs/promises`, and `filename` the path's base name when the entry has none. In Python, `content` is the file's bytes, read with `open(path, "rb")`. In Go, `Content` is `os.ReadFile(path)`. In Python and Go, name the content type the client guessed from the file name, such as `application/pdf`. +- A send with rendered content gets a `templateKey`, as Step 7 says. + +**Template renderers.** + +- `nodemailer-express-handlebars`: create the view engine once next to the PostHog client, as the plugin does, and render the view before the send: + + ```js + import { create } from 'express-handlebars' + import path from 'node:path' + + const viewEngine = create(hbsOptions.viewEngine) + const html = await viewEngine.renderView(path.join(hbsOptions.viewPath, 'welcome' + (hbsOptions.extName ?? '.handlebars')), context) + ``` + + `hbsOptions` is the object the code passed to the plugin. When its `viewEngine` is already an engine with `renderView`, use it as it is. Share that object between the plugin and the moved code rather than copying it. +- `email-templates`: `const { subject, html, text } = await email.renderAll(template, locals)`, then one PostHog send with those and the `message` fields. + +**Errors.** Keep the site failing the same way it did: + +- **Node.js.** `posthog.emails.send` never throws on an API error: it answers `{ data, error }`. Where the awaited `sendMail` threw into a `catch`, or the promise rejected to its caller, throw from the error so the same path runs: `if (error) throw Object.assign(new Error(error.message), { code: error.code, responseCode: error.statusCode })`. A callback `(err, info) => { ... }` becomes the same code run with `err` from `error` and `info` from `data`. `info.messageId` becomes `data.id`. +- **Python.** PostHog's SDK raises too. Replace `except smtplib.SMTPException` and its subclasses, and `except OSError` around the send, with the PostHog exception `posthog-sdks.md` names, and `e.smtp_code` with its status code. A check of the dict of refused recipients becomes the `except` branch. +- **Ruby.** Replace `rescue` of a `Net::SMTP` error, or of `StandardError` around the send, with `PostHog::EmailError`. +- **PHP.** With PHPMailer exceptions on, replace `catch (PHPMailer\PHPMailer\Exception $e)` with `catch (PostHog\EmailError $e)`. With exceptions off, the `if (!$mail->send())` branch runs in a `catch (PostHog\EmailError $e)`, and `$mail->ErrorInfo` becomes `$e->getMessage()`. For Symfony Mailer, replace `catch (TransportExceptionInterface $e)` with `catch (PostHog\EmailError $e)`. +- **Go.** `if err := d.DialAndSend(m); err != nil` becomes `if _, err := client.Emails().Send(ctx, ...); err != nil`, with the same branch. +- **.NET.** Replace `catch (SmtpCommandException)`, `catch (SmtpException)`, and the other client exceptions with `catch (PostHog.Emails.EmailException)`, and `StatusCode` with its `StatusCode`. +- **Elixir.** A `case` on the receipt binary and `{:error, ...}` becomes a `case` on `{:ok, result}` and `{:error, error}`, with the same branches. +- **Rust.** `mailer.send(&email)?` becomes `posthog.emails().send(...).await?` where the function's error type takes `EmailError`, or a `match` with the same branches. +- PostHog refuses a send to a suppressed address with `recipient_suppressed`, where an SMTP server took it and the provider behind it dropped it later. When the site throws, retries, or reports a failure on any error, handle `recipient_suppressed` before that branch the way the site handled a send that went out: log it and go on. Note each site that got this branch in the report. + +**The answer.** The message id of a moved send is PostHog's. Code that logged the Message-ID, or stored it for the app's own use, takes PostHog's message id instead. + +**The SMTP client.** Keep the transporter, the SMTP connection code, its import, and the template plugin while any site or other call still uses them. When nothing uses them, remove the import, the transporter or client, and its connection code, and keep the package and the SMTP variables: the person removes them after the cutover. diff --git a/context/skills/migrate-email/references/smtp/sdk-reference.md b/context/skills/migrate-email/references/smtp/sdk-reference.md new file mode 100644 index 00000000..d2291e0d --- /dev/null +++ b/context/skills/migrate-email/references/smtp/sdk-reference.md @@ -0,0 +1,281 @@ +# SMTP SDK reference + +This file says what an SMTP send looks like in a codebase. It does not describe PostHog. `mapping.md` maps each field to PostHog. + +An SMTP site sends through an SMTP client library to an SMTP server. The server is run by some provider, such as Google Workspace, Microsoft 365, Mailgun, or the person's own mail server. This skill moves the sends of the code. The provider behind the host decides the DNS records and the suppression list: `account.md` says how. + +## Key + +The SMTP password is the key. Apps keep it in `SMTP_PASSWORD`, `SMTP_PASS`, `MAIL_PASSWORD`, or `EMAIL_HOST_PASSWORD` (Django), or in their own name: take the name the client reads. It sits next to a host and a user variable, such as `SMTP_HOST` and `SMTP_USER`, `MAIL_HOST` and `MAIL_USERNAME`, or `EMAIL_HOST` and `EMAIL_HOST_USER`. A connection URL, such as `SMTP_URL=smtps://user:pass@smtp.example.com` or Symfony's `MAILER_DSN`, holds all three in one variable. + +## Packages + +| Language | Package | Send | +|---|---|---| +| Node.js | `nodemailer` | `transporter.sendMail(message)` | +| Python | `smtplib` and `email` (standard library), `aiosmtplib` | `smtp.send_message(msg)`, `smtp.sendmail(from_addr, to_addrs, msg)`, `aiosmtplib.send(msg, ...)` | +| Ruby | `mail`, `net-smtp` | `mail.deliver`, `Mail.deliver { ... }`, `smtp.send_message(msgstr, from, to)` | +| PHP | `phpmailer/phpmailer`, `symfony/mailer` used on its own | `$mail->send()`, `$mailer->send($email)` | +| Go | `net/smtp` (standard library), `gopkg.in/mail.v2` (gomail), `github.com/wneessen/go-mail` | `smtp.SendMail(...)`, `d.DialAndSend(m)`, `c.DialAndSend(m)` | +| .NET | `MailKit` with `MimeKit`, `System.Net.Mail` | `await client.SendAsync(message)`, `client.Send(message)`, `client.SendMailAsync(message)` | +| Elixir | `gen_smtp` | `:gen_smtp_client.send_blocking({from, [to], body}, options)` | +| Rust | `lettre` | `mailer.send(&email)` | + +## Clients and sends + +**Node.js: `nodemailer`.** `nodemailer.createTransport(options, defaults)` makes the transporter. SMTP options are `host`, `port`, `secure`, and `auth: { user, pass }`, or a `service` name, such as `service: 'gmail'`, that sets the host and port, or a URL string, such as `'smtps://user:pass@smtp.example.com'`. `pool: true` keeps connections open. The second argument, `defaults`, sets message fields for every send of that transporter, such as `from` or `headers`. + +```js +import nodemailer from 'nodemailer' + +const transporter = nodemailer.createTransport({ + host: process.env.SMTP_HOST, + port: 587, + secure: false, + auth: { user: process.env.SMTP_USER, pass: process.env.SMTP_PASSWORD }, +}) + +const info = await transporter.sendMail({ + from: '"Example" ', + to: 'jane@example.com', + subject: 'Welcome', + text: 'Welcome', + html: '

Welcome

', +}) +``` + +- `sendMail(message)` answers a promise. It rejects when the server refuses the connection, the login, the sender, or every recipient. The error has `code`, such as `EAUTH`, `ECONNECTION`, `ETIMEDOUT`, `EENVELOPE`, or `EMESSAGE`, and `responseCode` (the SMTP code, such as `550`), `response`, `command`, and `rejected`. +- It resolves when the server takes the message for at least one recipient. `info` has `messageId`, `envelope`, `accepted`, `rejected` (the recipients the server refused), and `response`. +- `sendMail(message, (err, info) => { ... })` is the callback form, with the same `err` and `info`. +- `transporter.use('compile', plugin)` adds a plugin that changes each message before it is built, such as `nodemailer-express-handlebars`. `mapping.md` says under "Templates" how those sites move. + +**Python: `smtplib`.** The standard library builds the message with `email.message.EmailMessage`, or with `email.mime` classes such as `MIMEMultipart`, `MIMEText`, and `MIMEApplication`, and sends it with `smtplib.SMTP` or `smtplib.SMTP_SSL`. + +```python +import smtplib +from email.message import EmailMessage + +msg = EmailMessage() +msg["From"] = "Example " +msg["To"] = "jane@example.com" +msg["Subject"] = "Welcome" +msg.set_content("Welcome") +msg.add_alternative("

Welcome

", subtype="html") + +with smtplib.SMTP(os.environ["SMTP_HOST"], 587) as smtp: + smtp.starttls() + smtp.login(os.environ["SMTP_USER"], os.environ["SMTP_PASSWORD"]) + smtp.send_message(msg) +``` + +- `send_message(msg)` and `sendmail(from_addr, to_addrs, msg)` raise a subclass of `smtplib.SMTPException` on an error: `SMTPRecipientsRefused` when every recipient was refused, `SMTPSenderRefused`, `SMTPDataError`, `SMTPAuthenticationError` from `login`, or `SMTPServerDisconnected`. `SMTPException` is a subclass of `OSError`. A response error has `smtp_code` and `smtp_error`. +- They answer a dict of the recipients the server refused, empty when it took every one. +- `msg.add_attachment(data, maintype=..., subtype=..., filename=...)`, or a `MIMEApplication` part with a `Content-Disposition` header, adds an attachment. +- `sendmail(from_addr, to_addrs, msg)` takes the envelope apart from the message: `to_addrs` can hold recipients that no header names, which is how such code sends a bcc. +- `aiosmtplib.send(msg, hostname=..., port=..., username=..., password=...)` is the async client, with the same message classes. +- smtplib sets no Message-ID. The code sets one with `email.utils.make_msgid()` when it needs it. + +**Ruby: `mail` and `net-smtp`.** The `mail` gem builds and sends the message. `delivery_method :smtp, address:, port:, user_name:, password:, authentication:, enable_starttls_auto:` sets the server, in `Mail.defaults` or on one message. + +```ruby +Mail.defaults do + delivery_method :smtp, address: ENV['SMTP_HOST'], port: 587, + user_name: ENV['SMTP_USER'], password: ENV['SMTP_PASSWORD'], authentication: 'plain' +end + +Mail.deliver do + from 'Example ' + to 'jane@example.com' + subject 'Welcome' + text_part { body 'Welcome' } + html_part do + content_type 'text/html; charset=UTF-8' + body '

Welcome

' + end +end +``` + +- `deliver` raises the `Net::SMTP` error on a failure, such as `Net::SMTPAuthenticationError`, `Net::SMTPServerBusy`, `Net::SMTPSyntaxError`, `Net::SMTPFatalError`, or `Net::SMTPUnknownError`, unless the message sets `raise_delivery_errors` to `false`. +- `add_file filename:, content:`, `add_file '/path'`, and `attachments['name.pdf'] = data` add attachments. +- `message_id` is the Message-ID, which the gem sets when the code doesn't. +- `Net::SMTP.start(host, port, user:, secret:, authtype:) { |smtp| smtp.send_message(msgstr, from, to) }` sends a message string the code wrote itself. + +**PHP: PHPMailer and Symfony Mailer.** PHPMailer: + +```php +$mail = new PHPMailer(true); +$mail->isSMTP(); +$mail->Host = getenv('SMTP_HOST'); +$mail->SMTPAuth = true; +$mail->Username = getenv('SMTP_USER'); +$mail->Password = getenv('SMTP_PASSWORD'); +$mail->SMTPSecure = PHPMailer::ENCRYPTION_STARTTLS; +$mail->Port = 587; +$mail->setFrom('hello@mail.example.com', 'Example'); +$mail->addAddress('jane@example.com'); +$mail->isHTML(true); +$mail->Subject = 'Welcome'; +$mail->Body = '

Welcome

'; +$mail->AltBody = 'Welcome'; +$mail->send(); +``` + +- `new PHPMailer(true)` throws `PHPMailer\PHPMailer\Exception` on an error. `new PHPMailer()` without `true` throws nothing: `send()` answers `false`, and `$mail->ErrorInfo` holds the error. +- `Body` is HTML after `isHTML(true)`, and text otherwise. `AltBody` is the text part. +- `addCC`, `addBCC`, `addReplyTo`, `addAttachment($path, $name)`, `addStringAttachment($content, $filename)`, `addEmbeddedImage`, `addStringEmbeddedImage`, `addCustomHeader`, `Priority`, `Ical`, and `Sender` set the rest. `getLastMessageID()` is the Message-ID. + +`symfony/mailer` used on its own, without the Symfony framework: `new Mailer(Transport::fromDsn(getenv('MAILER_DSN')))`, and `$mailer->send($email)` with a `Symfony\Component\Mime\Email` built with `from`, `to`, `cc`, `bcc`, `replyTo`, `priority`, `subject`, `text`, `html`, `attach`, `attachFromPath`, and `addPart`. `send` answers nothing and throws a `TransportExceptionInterface` on an error. A `MailerInterface` that the framework injects is a framework mailer: see "Frameworks". + +**Go.** `net/smtp` sends bytes the code writes itself. It answers an `error`. + +```go +auth := smtp.PlainAuth("", os.Getenv("SMTP_USER"), os.Getenv("SMTP_PASSWORD"), host) +err := smtp.SendMail(host+":587", auth, from, []string{to}, msg) +``` + +`gopkg.in/mail.v2` (gomail, package `mail`) builds the message: + +```go +m := mail.NewMessage() +m.SetHeader("From", "hello@mail.example.com") +m.SetHeader("To", "jane@example.com") +m.SetHeader("Subject", "Welcome") +m.SetBody("text/plain", "Welcome") +m.AddAlternative("text/html", "

Welcome

") +err := mail.NewDialer(host, 587, user, password).DialAndSend(m) +``` + +`SetAddressHeader("Cc", address, name)`, `Attach(path)`, and `Embed(path)` set the rest. The older import `gopkg.in/gomail.v2` has the same API in package `gomail`. + +`github.com/wneessen/go-mail` builds it with `m := mail.NewMsg()`, `m.From(...)`, `m.To(...)`, `m.Cc(...)`, `m.Bcc(...)`, `m.ReplyTo(...)`, `m.Subject(...)`, `m.SetBodyString(mail.TypeTextPlain, ...)`, `m.AddAlternativeString(mail.TypeTextHTML, ...)`, `m.AttachFile(path)`, and `m.AttachReader(name, reader)`, and sends with `c, err := mail.NewClient(host, mail.WithPort(587), mail.WithSMTPAuth(mail.SMTPAuthPlain), mail.WithUsername(user), mail.WithPassword(password))` and `c.DialAndSend(m)`. + +Every Go client answers an `error` and never panics on an SMTP refusal. + +**.NET: MailKit and System.Net.Mail.** MailKit sends a MimeKit `MimeMessage`: + +```csharp +var message = new MimeMessage(); +message.From.Add(new MailboxAddress("Example", "hello@mail.example.com")); +message.To.Add(MailboxAddress.Parse("jane@example.com")); +message.Subject = "Welcome"; +message.Body = new BodyBuilder { TextBody = "Welcome", HtmlBody = "

Welcome

" }.ToMessageBody(); + +using var client = new SmtpClient(); +await client.ConnectAsync(host, 587, SecureSocketOptions.StartTls); +await client.AuthenticateAsync(user, password); +await client.SendAsync(message); +await client.DisconnectAsync(true); +``` + +- It throws on an error: `SmtpCommandException` (with `StatusCode`, `ErrorCode`, and `Mailbox`), `SmtpProtocolException`, or `AuthenticationException`. +- `message.Cc`, `Bcc`, `ReplyTo`, `builder.Attachments.Add(...)`, `builder.LinkedResources`, and `message.Headers` set the rest. `message.MessageId` is the Message-ID. + +`System.Net.Mail.SmtpClient` sends a `MailMessage` with `From`, `To`, `CC`, `Bcc`, `ReplyToList`, `Subject`, `Body`, `IsBodyHtml`, `AlternateViews`, `Attachments`, `Headers`, and `Priority`. `Send` and `SendMailAsync` throw `SmtpException` (with `StatusCode`), `SmtpFailedRecipientException`, or `SmtpFailedRecipientsException`. + +**Elixir: `gen_smtp`.** `:gen_smtp_client.send_blocking({from, [to], body}, relay: host, port: 587, username: user, password: password, tls: :always)` sends a message body. The body is a string the code wrote, or the output of `:mimemail.encode({type, subtype, headers, params, parts})`. It answers the server's receipt as a binary on success, and an `{:error, ...}` tuple on a failure. `:gen_smtp_client.send/2` sends in a spawned process. + +**Rust: `lettre`.** + +```rust +let email = Message::builder() + .from("Example ".parse()?) + .to("jane@example.com".parse()?) + .subject("Welcome") + .multipart(MultiPart::alternative_plain_html(String::from("Welcome"), String::from("

Welcome

")))?; + +let mailer = SmtpTransport::relay(&host)? + .credentials(Credentials::new(user, password)) + .build(); +mailer.send(&email)?; +``` + +- `send` answers a `Result`: the error is a value, not a panic. `AsyncSmtpTransport::` sends with `.send(email).await`. +- `.cc()`, `.bcc()`, `.reply_to()`, `.header()`, and `Attachment::new(filename).body(content, content_type)` set the rest. + +## SMTP host + +Look for the host in transport options, `host` and `service` settings, connection URLs, `SMTP_HOST`, `MAIL_HOST`, `EMAIL_HOST`, `MAIL_SERVER`, and `MAILER_DSN` in example env files and config, and in framework mail config. The host names the provider behind it. A host under the same domain, such as a regional one, is the same provider. + +| Host | Provider behind it | +|---|---| +| `smtp.gmail.com`, `smtp-relay.gmail.com`, Nodemailer `service: 'gmail'` | Gmail or Google Workspace | +| `smtp.office365.com`, `smtp-mail.outlook.com` | Microsoft 365, or Outlook.com | +| `email-smtp..amazonaws.com` | Amazon SES | +| `smtp.sendgrid.net` | SendGrid | +| `smtp.mailgun.org` | Mailgun | +| `smtp.postmarkapp.com` | Postmark | +| `smtp.resend.com` | Resend | +| `smtp-relay.brevo.com` | Brevo | +| `in-v3.mailjet.com`, `in.mailjet.com` | Mailjet | +| `smtp.sparkpostmail.com` | SparkPost | +| `smtp.mandrillapp.com` | Mandrill (Mailchimp Transactional) | +| `smtp.zoho.com` | Zoho Mail | +| Another host on the person's own domain, such as `mail.example.com` | A server the person or their host runs | + +- A Nodemailer `service` name, such as `'SendGrid'`, `'Mailgun'`, or `'SES-EU-WEST-1'`, stands for that provider's host. +- A host such as `localhost`, `127.0.0.1`, `mailhog`, or `mailpit` in an example env file is a local catcher for development. The deployed host comes from the deployed environment: ask the person which provider it is. +- When the host comes from a variable with no value in the repository, ask the person which provider runs it. + +## Frameworks + +Framework mailers that send through SMTP: Django's `send_mail`, `EmailMessage`, and `EmailMultiAlternatives` with `EMAIL_BACKEND = 'django.core.mail.backends.smtp.EmailBackend'` (the default) and `EMAIL_HOST`; `Flask-Mail` and `Flask-Mailman` with `MAIL_SERVER`; Rails Action Mailer with `delivery_method = :smtp` and `smtp_settings`; Laravel `Mail` with `MAIL_MAILER=smtp` and `MAIL_HOST`; Symfony Mailer through the framework, with `MAILER_DSN=smtp://...` and an injected `MailerInterface`; Phoenix and Elixir apps with Swoosh's `Swoosh.Adapters.SMTP` or Bamboo's SMTP adapter; and NestJS's `@nestjs-modules/mailer`, which wraps Nodemailer behind `MailerService.sendMail`. Each mailable or mailer that sends is a site. + +## Send fields + +Nodemailer names, with the part of the MIME message that each one sets. Other clients set the same parts with their own names: a `From` header, a `text/html` part, a `Content-Disposition: attachment` part. + +| Nodemailer field | In the message | Value | +|---|---|---| +| `from` | `From` header | An address, `'Name
'`, or `{ name, address }` | +| `sender` | `Sender` header | An address | +| `to`, `cc`, `bcc` | `To`, `Cc` headers, and the envelope. `Bcc` is in the envelope only | An address, a comma-separated string, `{ name, address }`, or a list of them | +| `replyTo` | `Reply-To` header | Same as `to` | +| `inReplyTo`, `references` | `In-Reply-To`, `References` headers | Message-IDs | +| `subject` | `Subject` header | String | +| `text`, `html` | The `text/plain` and `text/html` parts | A string, a `Buffer`, a stream, or `{ path }` or `{ href }` to read it from | +| `watchHtml`, `amp` | An Apple Watch HTML part, an AMP for Email part | Same as `html` | +| `alternatives` | More parts of the alternative | A list of `{ content, contentType }` | +| `icalEvent` | A calendar part and an `.ics` attachment | `{ method, content }`, `{ path }`, or `{ href }` | +| `attachments` | `Content-Disposition: attachment` parts, or inline parts with `Content-ID` | A list of `{ filename, content, path, href, encoding, contentType, contentDisposition, cid, headers, raw }` | +| `headers` | Any header | Object of header names and values | +| `list` | `List-*` headers, such as `List-Unsubscribe` | Object keyed by `help`, `unsubscribe`, `subscribe`, `post`, `owner`, `archive`, `id` | +| `priority` | `X-Priority`, `Importance` headers | `'high'`, `'normal'`, or `'low'` | +| `messageId`, `date`, `xMailer`, `otp` | `Message-ID`, `Date`, `X-Mailer`, `OTP-Token` headers | Strings | +| `envelope` | The SMTP `MAIL FROM` and `RCPT TO`, in place of the ones from the headers | `{ from, to, cc, bcc }` | +| `raw` | The whole message, as the code built it | A string, `Buffer`, or stream | +| `dsn` | Delivery status notification options | `{ id, return, notify, recipient }` | +| `dkim` | DKIM signature of this message | Keys and selector | +| `attachDataUrls` | Turns `data:` images in `html` into inline attachments | Boolean | +| `encoding`, `textEncoding`, `disableFileAccess`, `disableUrlAccess` | How Nodemailer writes the message, and which sources it may read | Strings, booleans | +| `template`, `context`, `text_template` | Read by the `nodemailer-express-handlebars` plugin, not by Nodemailer | View names and values | + +## Templates + +SMTP has no stored templates. The app renders its HTML before the send, with Handlebars, EJS, Pug, MJML, React Email, Jinja, ERB, Blade, Twig, or string formatting, so the content is in the code. Two Node.js packages render inside the send: + +- `nodemailer-express-handlebars`: `transporter.use('compile', hbs({ viewEngine, viewPath, extName }))`, then `sendMail({ template: 'welcome', context: { ... } })`. The plugin renders `/