Spec: MailChat MVP — email-as-chat

[Hohnik]

Synthesized from the wayfinder planning effort on MailChat MVP: email-as-chat spec (closed) and its five resolved tickets. No interview conducted for this write-up — this is a synthesis of decisions already made.

Problem Statement

Email clients present every account as a flat, undifferentiated list of messages. When I go back and forth with a person over email, I have to manually reconstruct the conversation by scanning subjects and senders — there's no view that reads like an ongoing chat with that person, the way texting or WhatsApp does. This gets worse with multiple connected accounts (the same contact might reach me on more than one address) and with real-world subject-line mangling (reply/forward prefixes inserted inconsistently by different mail clients and vendor ticketing systems), which defeats naive attempts to group messages by subject.

Solution

MailChat connects to my existing IMAP/SMTP mailboxes (starting with a real T-Online account, already proven reachable) and ingests mail via background polling. Instead of a flat inbox, it groups messages into chat-style conversations keyed by (person, subject) — one conversation per topic per contact, closer to a chat app than a traditional inbox, while still respecting genuine topic boundaries so unrelated subjects with the same person don't collapse into one undifferentiated stream. A lightweight, server-rendered web UI (htmx + htpy, no JS build step) presents these as chat conversations, updating via polling. It's a single-user, self-hosted tool — each user runs their own instance against their own mailboxes.

User Stories

1. As a user, I want to connect an IMAP/SMTP mailbox by providing my address and app-specific password, so that MailChat can start syncing my mail.
2. As a user connecting a provider (like T-Online) that requires a separate app-specific password, I want that requirement surfaced clearly, so I'm not confused when my regular account password fails to authenticate.
3. As a user, I want my mailbox credentials encrypted at rest, so a leak of the database doesn't expose my plaintext email password.
4. As a user, I want the encryption master key sourced from an environment variable rather than stored in the database, so I control key management for my self-hosted instance.
5. As a user, I want to connect multiple mailboxes to the same MailChat instance, so I can see all my email accounts in one unified place.
6. As a user, I want each synced message attributed to the connected account it arrived through, so I know which mailbox received it.
7. As a user, I want MailChat to periodically poll each connected account for new mail in the background, so new messages show up without me manually refreshing or reconnecting.
8. As a user, I want one account's slow or unavailable mail server to not block syncing my other connected accounts, so a single flaky provider doesn't stall the whole app.
9. As a user, I want raw fetched mail parsed from RFC822/MIME into a structured message (headers, body), so the app can group and display it correctly.
10. As a user, I want HTML email bodies converted to safely-renderable content, so I can read messages without loading remote trackers or executing scripts embedded in HTML mail.
11. As a user, I want messages from the same contact about the same topic grouped into a single conversation, so it reads like one ongoing chat rather than a scattered list.
12. As a user, I want a new subject from the same contact to start a new conversation, so unrelated topics with the same person stay visually and logically separate.
13. As a user, I want reply/forward prefixes (Re:, Aw:, Fwd:, Fw:, Wg:) stripped from subjects wherever they occur in the string — not just at the very start — so real-world subject-mangling (e.g. a vendor's ticketing system inserting Re: after its own ticket-ID prefix) doesn't fracture one human conversation into multiple conversation cards.
14. As a user, I want the primary correspondent (the sender for an inbound message, the recipient for an outbound one) to define who a conversation is with, excluding all of my own connected-account addresses, so I never see myself listed as a conversation partner.
15. As a user, I want people CC'd on a message shown on that message but excluded from conversation grouping, so someone being looped in or dropped from a thread mid-conversation doesn't fragment it into separate conversations.
16. As a user, I want each distinct email address treated as its own contact identity for now (no automatic alias merging), so the grouping model stays simple and predictable, even though this means the same human using two addresses currently shows up as two contacts.
17. As a user, I want the same contact's messages across two of my connected accounts merged into one conversation, so I see a single unified chat with that person regardless of which of my inboxes they used to reach me.
18. As a user, I want Message-ID/In-Reply-To/References headers stored per message even though they don't drive conversation grouping, so future features (e.g. reply ordering, thread visualization) can use them without re-deriving them from scratch.
19. As a user, I want a contact record automatically built and updated from message history (address + display name), so I don't have to manually add people I've corresponded with.
20. As a user, I want to see basic contact info (name, address) alongside a conversation, so I know at a glance who I'm chatting with.
21. As a user, I want the messages within a conversation shown in chronological order, so I can follow the exchange the way I'd read a chat log.
22. As a user, I want to send a reply within a conversation via SMTP, so I can respond to someone without leaving MailChat.
23. As a user, I want my reply correctly addressed and subject-lined to continue the existing conversation, so the recipient's own mail client threads it properly on their end.
24. As a user, I want to mark a conversation or message as read, so my unread state reflects what I've actually seen.
25. As a user, I want a chat-style web UI listing my conversations grouped and labeled by contact, so I can browse my mail the way I'd browse a messaging app's chat list.
26. As a user, I want the conversation list and open conversation to refresh via periodic polling rather than requiring a manual page reload, so new mail appears on its own, consistent with the polling-based sync backend.
27. As a user, I want my synced mail data persisted in a local SQLite database, so the app works without me having to stand up a separate database server.
28. As a user, I want that SQLite database continuously replicated to S3 (via Litestream), so I don't lose my synced conversation history if the machine running MailChat is lost.
29. As a developer extending MailChat later, I want the ability to add dedicated JSON API endpoints alongside the existing HTML routes without conflict, so a future native app (e.g. iOS) can be added without re-architecting the backend.
30. As a developer, I want the existing, live-verified IMAP/SMTP client code reused as-is by the sync pipeline, so the already-proven connection/auth behavior isn't re-implemented or accidentally regressed.

Implementation Decisions

Stack: FastAPI + async SQLAlchemy + SQLite. SQLite is the production database too (not just dev/test) — replicated continuously to S3 via Litestream. No separate server database process: this is a single-user, single-process tool with no concurrent-writer or multi-machine access pattern to support.

Frontend: htmx (v4, currently beta/RC — pin now, expect a small migration once it reaches stable) for interactivity, and htpy (pure-Python HTML generation via function calls) for rendering — no Jinja2, no separate template files, no JS bundler or Node toolchain required. FastAPI routes render HTML directly for this UI. This supersedes an earlier "backend exposes a generic JSON API" framing from initial project notes — the backend is not JSON-only; a JSON API is not built for the MVP, though dedicated JSON endpoints can be added alongside the HTML routes later without conflict if a second consumer (e.g. a native app wrapping the site via Capacitor/Tauri, or a real native client) needs one. Build a minimal read-only conversation view early — as soon as the sync pipeline produces real grouped conversations — to visually validate the grouping model against messy real data before building send/reply interactivity.

accounts module: Account model (one row per connected mailbox — provider/host/port configuration, address) with a linked Credentials model (encrypted secret). Credentials are encrypted at rest using Fernet-style symmetric encryption; the master key is read from an environment variable, never persisted in the database. The service layer handles connecting (validating credentials against the real IMAP/SMTP servers via the already-built imap_client/smtp_client) and disconnecting an account.

sync module:

• IMAP and SMTP clients are already implemented and live-verified against a real T-Online account: stdlib-only (imaplib, smtplib), no third-party dependencies, direct-SSL connections (IMAP on port 993, SMTP on port 465 — no STARTTLS needed for this provider). Reuse as-is.
• A parser converts raw RFC822/MIME bytes into a structured intermediate representation: headers (From/To/Cc/Subject/Message-ID/In-Reply-To/References/Date) plus a safely-renderable body (HTML converted/sanitized for display).
• A normalizer takes that parsed representation and persists it as a Message, resolving or creating the associated Contact and Conversation via the grouping logic below.
• The grouping logic (conceptually "threading," though it does not use header-based threading — see below) computes: (a) the primary correspondent — sender for inbound, recipient for outbound, excluding every address across all of the user's connected accounts; (b) a normalized subject — repeatedly stripping Re/Aw/Fwd/Fw/Wg tokens (case-insensitive) followed by a colon, wherever they occur in the subject string (not anchored to the start), then collapsing whitespace and trimming, applied idempotently; (c) a conversation key of (contact_id, normalized_subject), with no account component (so the same contact across multiple connected accounts merges into one conversation). Message-ID/In-Reply-To/References are parsed and stored on the Message but are not consulted by this grouping logic — they exist for future use only.
• A background worker runs one polling task per connected account, fetching recent messages via the IMAP client and routing each through parser → normalizer.
• gmail_client remains an unimplemented stub (out of scope — generic IMAP already covers Gmail).

conversations module: Conversation model keyed by (contact, normalized_subject), tracking enough to order conversations by recency (e.g. last-message timestamp). No separate Participant model is needed at the conversation level — since CC'd participants don't affect grouping (per the grouping-model decision), participant/CC information is tracked per-Message, not as a conversation-level join. The service layer finds-or-creates a conversation as each message is normalized. Routes render the conversation list and an individual conversation's message thread as HTML (via htpy), with hx-trigger-based polling for updates.

messages module: Message model (conversation reference, contact/direction, raw subject as received for display, safely-rendered body, stored Message-ID/In-Reply-To/References, read/unread flag, timestamp). The service layer sends replies by reusing smtp_client.send_message, constructing a properly-addressed, properly-subjected outbound email that continues the conversation, and supports marking messages/conversations as read.

contacts module: Contact model keyed by exact email address (case-insensitive) — no automatic cross-address alias merging in the MVP. Display name is kept up to date from the most recent message's From/To header. A manual contact-merge feature (for cases like the same human using two addresses) is explicitly deferred past this MVP.

shared module: async SQLAlchemy engine/session setup and declarative base; Fernet-based encryption/decryption helpers (master key from environment); app-specific exception types; logging setup.

Persistence/deployment: SQLite is the sole datastore for both development and production, continuously replicated to S3 via Litestream. The exact hosting/process-supervision target for running the FastAPI app and sync worker in production is intentionally undecided — it doesn't block starting implementation (see Out of Scope).

Testing Decisions

A good test in this codebase asserts on external/observed behavior — persisted database state, rendered HTML content, or the arguments a mocked mail-server client was called with — not on internal implementation details or private helper call counts. This matches the pattern already established by the existing IMAP/SMTP client tests.

• Sync pipeline seam: mock only the IMAP/SMTP connection boundary (imaplib.IMAP4_SSL / smtplib.SMTP_SSL), exactly as the existing imap_client/smtp_client tests already do. Feed real captured raw-message fixtures — e.g. a recreation of the real multi-message Louis.de DL-4916597 thread (which exercises the non-anchored subject-normalization requirement) and a message with a CC'd participant (e.g. the real Horbach example, To one person / Cc another) — through the real parser → normalizer → grouping pipeline against a real temporary/in-memory SQLite database. Assert on the resulting Conversation/Message/Contact rows: correct grouping, correct contact identity, no accidental splitting of one contact's cross-account messages into separate conversations.
• Credential encryption seam: direct unit tests of the encrypt/decrypt round-trip in the shared security module — pure function, no mocking, no I/O.
• htmx routes seam: FastAPI TestClient tests against routes with a seeded test SQLite database, asserting on rendered HTML content (e.g. one conversation card per grouped conversation, correct contact/subject shown, hx-trigger polling attributes present) rather than exact byte-for-byte HTML strings, to avoid brittle tests.
• Prior art: tests/test_sync/test_imap_client.py and tests/test_sync/test_smtp_client.py already establish the "mock the stdlib client at the connection boundary" pattern to follow for any new code that talks to IMAP/SMTP.
• Fixtures: real .eml sample files (in the already-planned tests/fixtures/emails/ location) sourced from real T-Online account samples gathered during planning, with any sensitive personal content anonymized before being committed to the repo.

Out of Scope

• Gmail-specific OAuth API integration (gmail_client stays a stub) — generic IMAP already covers Gmail; provider-specific APIs are a future effort.
• Realtime WebSocket push — polling is sufficient for the MVP; the realtime module stays unimplemented.
• Search, attachments beyond basic display, and multi-user auth on the app itself — not needed to prove the core concept.
• Participant-set/group-chat-aware conversation grouping (letting the CC set itself affect grouping, beyond just the primary correspondent) — a real anticipated future need, deliberately deferred past this MVP's grouping model.
• Manual contact-merge UX (for unifying two addresses used by the same human) — deferred; contact identity is exact address only for the MVP.
• Native or WebView-wrapped mobile app packaging (e.g. via Capacitor/Tauri) — a real future direction, not built now.
• A generic, reusable JSON API — superseded by the htmx+htpy decision for the MVP UI; dedicated JSON endpoints can be added later alongside the HTML routes if a second consumer appears, but building that general-purpose contract now is out of scope.
• The deployment/hosting target for running the app in production — Litestream+S3 already covers persistence; where/how the process itself runs (bare metal, container host, etc.) is undecided and does not block starting implementation.

Further Notes

• This spec synthesizes the full decision trail from a wayfinder planning effort. The detailed reasoning and real-data grounding behind each decision lives in the now-closed GitHub issues: MailChat MVP: email-as-chat spec (map), T-Online app-password research, IMAP proof of concept, SMTP proof of concept, conversation-grouping model, and frontend technology.
• htmx is pinned to v4 while it's still in beta/RC (targeting a "Summer '26" general-availability release). A small migration may be needed once it reaches stable — accepted as low-stakes for a single-user tool with no other users depending on uptime.
• The T-Online credential gotcha is already resolved and encoded in src/mailchat/.env: T-Online requires a separate "Passwort für E-Mail-Programme" for IMAP/SMTP, distinct from the regular account password, regardless of TOTP/MFA status.