Outbox — Email Job Scheduler

A ReachInbox-style cold-email scheduler: a Node/Express + BullMQ backend that schedules and sends bulk email campaigns reliably (survives restarts, enforces per-sender hourly rate limits, never double-sends), and a Next.js dashboard to compose campaigns and watch them move from Scheduled → Sent.

Built incrementally over 10 stages, each committed separately — git log reads top to bottom as the build order: schema → scheduling API → worker → rate limiter → restart-persistence proof → login/dashboard shell → compose panel → tables → docs → backend auth + CI.

📹 Demo Video

Watch the demo (~5 min) — covers scheduling, dashboard, restart persistence.

Architecture

flowchart LR
    subgraph Frontend [Next.js]
        Compose[Compose panel]
        Dash[Scheduled / Sent tables]
    end

    subgraph Backend [Express API]
        Schedule[POST /api/emails/schedule]
        List[GET /api/emails, /stats]
    end

    subgraph Infra
        PG[(Postgres)]
        Redis[(Redis)]
    end

    Worker[BullMQ Worker<br/>— separate process]
    Ethereal[Ethereal SMTP]

    Compose -->|JSON: recipients, subject,<br/>body, schedule| Schedule
    Schedule -->|1 row per recipient| PG
    Schedule -->|1 delayed job per recipient,<br/>jobId = email-uuid| Redis
    Dash -->|reads| List
    List --> PG

    Redis -->|delayed job becomes due| Worker
    Worker -->|claim + idempotency +<br/>rate-limit check| PG
    Worker -->|reserve slot<br/>INCR/EXPIRE Lua script| Redis
    Worker -->|send| Ethereal
    Worker -->|status: sent/failed| PG

The scheduling path (POST /api/emails/schedule, backend/src/routes/emails.ts): for every recipient, create one Email row (status scheduled) before enqueuing anything, then add one BullMQ delayed job per recipient with jobId = email-{uuid} and a delay computed from startAt + index × delayBetweenEmails. Rows are created first so a crash mid-loop leaves orphaned scheduled rows (safe — just re-enqueue) rather than orphaned jobs pointing at rows that don't exist.

The worker (backend/src/worker.ts) runs as a separate process from the API (npm run dev:worker, not part of server.ts) — an SMTP hang or OOM there can't take down the dashboard, and the two scale independently. Its status FSM:

scheduled ──claim──► sending ──rate-limited──► scheduled (redelayed to next hour, no attempt burned)
                      │
                      ├──SMTP ok───────────────► sent
                      ├──SMTP transient err────► (throw; BullMQ retries w/ backoff)
                      └──retries exhausted─────► failed  (written ONLY by the
                                                          'failed' listener,
                                                          never the processor)

sending is the crash-safety state: if a worker dies between "SMTP handoff began" and "outcome recorded," the row is left at sending with no live job holding it. A later redelivery of that job sees status === 'sending' on its first attempt (attemptsMade === 0, since a genuine SMTP retry would have incremented that) and refuses to resend — marking it failed with an explanatory message instead. At-most-once beats at-least-once here: a missed email costs one lead, a duplicate costs a spam report.

Idempotency has three independent layers, each closing a gap the one before it can't:

jobId = email-{uuid} — BullMQ dedupes a repeated add() while the job is still in Redis.
A DB status check (already sent → no-op) — survives Redis being flushed or the job already having completed and been evicted.
The crash-ambiguity guard above — survives the worker itself crashing mid-send.

Rate limiting (backend/src/lib/rateLimiter.ts) is enforced entirely in Redis via a Lua script that makes "increment, then decide whether to keep it" atomic — without that, concurrent workers could both read "under limit" before either writes back. Two independent hourly buckets: global (MAX_EMAILS_PER_HOUR, shared across all senders) and per-sender (min(campaign's hourlyLimit, MAX_EMAILS_PER_HOUR_PER_SENDER)). When a reservation is rejected, the worker reverts the row to scheduled and calls job.moveToDelayed() + throws BullMQ's DelayedError — its documented mechanism for a job to defer itself without incrementing attemptsMade or firing the failed listener. Redelivery targets the next hour boundary plus up to 30s of jitter, so a batch that got throttled together doesn't recreate the exact same burst an hour later.

Restart-persistence is a property of Redis, not of any process staying alive: a delayed job lives in a Redis sorted set keyed by its due timestamp. Kill the worker, wait past the due time with zero workers running, start a fresh worker — it reattaches to the same queue name and finds the overdue job waiting, no different from a job that was always going to fire late. backend/scripts/verify-restart-persistence.ts proves exactly this by spawning and SIGKILLing a real OS process, not just closing an in-memory Worker instance.

How the backend authenticates requests

Every /api/emails/* route requires an Authorization: Bearer <token> header (backend/src/middleware/auth.ts), and the sender identity for a request always comes from that verified token — never from a senderEmail field in the request body or query string. A caller can't schedule a campaign as someone else by editing a form field, and can't list or read another sender's campaigns by changing a URL parameter.

The token isn't NextAuth's own session cookie. NextAuth v4 JWT sessions are encrypted (JWE, not a plain signed JWT) with a key derived from NEXTAUTH_SECRET — decrypting that from a separate Node service would mean either depending on next-auth internals outside of Next.js or reimplementing its HKDF/A256GCM scheme. Instead, the two services share a second, distinct secret (SESSION_SECRET, set identically in both backend/.env and frontend/.env):

A Server Component or Route Handler on the frontend calls getServerSession() — the same authoritative check dashboard/layout.tsx already uses for route protection.
Only after that succeeds, frontend/lib/sessionToken.ts signs a short-lived (5 minute) HS256 token containing the verified email.
The backend verifies that token with the shared secret and trusts the email inside it — the token literally cannot exist unless step 1 already passed, so verifying the signature is sufficient.

Server Components (the Scheduled/Sent tables, sidebar counts) already have a verified session in scope, so they sign a token directly. The Compose form is a Client Component and can't call getServerSession() itself (no access to the request's cookies from the browser); it instead fetches the same-origin /api/session-token route, which does have that access, before calling the Express backend.

Email/password sign-in

The frontend has no database of its own — Sender rows, and now their optional passwordHash, live only in the backend's Postgres (see the Sender model's comment in schema.prisma). So NextAuth's CredentialsProvider.authorize() in frontend/lib/auth.ts doesn't check a password locally — it's a server-to-server POST to the backend's own /api/auth/login (backend/src/routes/credentials.ts), which bcrypt-compares against the stored hash and returns the user object NextAuth then builds a session from. /api/auth/register and /api/auth/login are the only two unauthenticated routes on the whole backend, so both sit behind their own IP-keyed rate limiter (backend/src/middleware/rateLimit.ts) — an unthrottled password endpoint would just be a smaller copy of the exact "anything can hit this" gap the bearer-token scheme above closes for everything else.

Pagination

The Scheduled tab is actually two backend queries merged client-side (status=scheduled + status=sending, since "queued or currently sending" reads as one list to a user), and Sent is sent + failed the same way — see the comments in scheduled/page.tsx and sent/page.tsx. That merge is why "Load more" pages each sub-query independently via page (components/dashboard/PaginatedEmailList.tsx) rather than one shared cursor across both — a real page-N pagination scheme doesn't compose cleanly over two independently-paginated queries being zipped together, so each keeps its own page/totalPages and the merged, re-sorted list only grows once a status has more of its own pages left.

Tech stack & why

Decision	Choice	Why
ORM	Prisma	Superior TS codegen, first-class migration CLI. Drizzle is lighter but the migration story and relation ergonomics are more mature in Prisma — easier to defend in an interview.
Queue	BullMQ + Redis	Delayed jobs live in a Redis sorted set — restart-persistence is free, not something built by hand on top of `node-cron`.
DB	PostgreSQL	Relational data (Sender → Email 1:many) with real FK/cascade semantics; `rate_limit_windows` as an audit trail benefits from real querying.
SMTP	Ethereal (nodemailer)	Real SMTP protocol path (auth, TLS, envelope, 4xx/5xx) without emailing strangers; every send gets a shareable preview URL.
Auth	NextAuth v4, JWT sessions	Google OAuth is the mandatory, must-be-real piece; a `CredentialsProvider` sits alongside it for real email/password sign-in. JWT (not database) sessions because there's no NextAuth-owned `users` table — the backend's `Sender` row (now with an optional `passwordHash`) is upserted/created lazily, either from the session email on first campaign or from a real `POST /api/auth/register`.
Backend auth	Shared-secret HS256 token (`jsonwebtoken`)	Bridges the real NextAuth session to the separate Express service without depending on `next-auth`'s internal JWE cookie format — see How the backend authenticates requests.
Rich text	Tiptap v3	Wraps ProseMirror; consistent bold/italic/underline/list HTML output vs. the deprecated, browser-inconsistent `document.execCommand`.
Styling	Tailwind v4	CSS-first config (no `tailwind.config.js`) — this repo was scaffolded on Next.js 16, which is genuinely new enough that some conventions differ from older Next docs (e.g. `middleware.ts` → `proxy.ts`; see frontend/proxy.ts).

Getting started

Prerequisites

Node.js 20+, Docker Desktop
A Google Cloud project for OAuth (see below) — the one piece of setup that can't be skipped, since Google sign-in is a real integration, not a mock.

1. Infrastructure

docker compose up -d

Starts Postgres (localhost:5433 → container's 5432, offset to avoid clashing with a locally-installed Postgres) and Redis (localhost:6379), both with named volumes so data survives a plain docker compose down.

2. Backend

cd backend
cp .env.example .env      # fill in GOOGLE_CLIENT_ID/SECRET, see below;
                           # set SESSION_SECRET to any random string
npm install
npm run prisma:migrate     # applies migrations, generates the Prisma client
npm run dev                # API on :4000

In a second terminal, start the worker (separate process, see Architecture):

npm run dev:worker

Leave ETHEREAL_USER/ETHEREAL_PASS blank on first run — the worker mints a throwaway Ethereal account on its first send and prints the credentials to the console. Copy them into .env to keep one inbox across restarts (otherwise every restart gets a fresh, empty inbox and old preview links stop working).

3. Google OAuth (required for real sign-in)

console.cloud.google.com → APIs & Services → Credentials → Create OAuth client ID → Web application.
Authorized redirect URI, exactly: http://localhost:3000/api/auth/callback/google
Paste the client ID/secret into both backend/.env and frontend/.env (same Google app — Google OAuth itself is app-level, not service-level; the backend's copy is genuinely unused, since only NextAuth talks to Google).

4. Frontend

cd frontend
cp .env.example .env      # NEXTAUTH_SECRET: openssl rand -base64 32;
                           # SESSION_SECRET: must match backend/.env EXACTLY
npm install
npm run dev                # dashboard on :3000

Unlike GOOGLE_CLIENT_ID/SECRET, SESSION_SECRET is not shared with Google — it's a value you make up yourself, but it has to be the same string in both .env files (see How the backend authenticates requests). A mismatch here fails every write and every dashboard fetch with a 401.

Visit http://localhost:3000 → redirects to /login → Continue with Google, or Register for an email/password account instead (no Google Cloud project needed for that path) → lands on /dashboard/scheduled.

Environment variables

Both backend/.env.example and frontend/.env.example are the source of truth, with a comment on every non-obvious variable. Summary:

Variable	Where	Notes
`DATABASE_URL`	backend	Port `5433`, matches docker-compose's offset mapping
`WORKER_CONCURRENCY`	backend	Parallel jobs per worker process; tied to the SMTP pool's `maxConnections`
`MAX_EMAILS_PER_HOUR` / `MAX_EMAILS_PER_HOUR_PER_SENDER`	backend	Global vs. per-sender send rate caps — see Architecture
`AUTH_RATE_LIMIT_WINDOW_MS` / `AUTH_RATE_LIMIT_MAX`	backend	Unrelated to the two above — throttles `POST /api/auth/register`/`login` specifically, see Email/password sign-in
`ETHEREAL_USER` / `ETHEREAL_PASS`	backend	Blank = fresh throwaway inbox every restart
`GOOGLE_CLIENT_ID` / `GOOGLE_CLIENT_SECRET`	both	Same Google app; frontend's copy is the one actually used for sign-in
`NEXTAUTH_SECRET`	frontend	Signs/encrypts NextAuth's own session JWT — never seen by the backend
`SESSION_SECRET`	both	Must be identical in both `.env` files — signs/verifies the short-lived token the frontend mints for the backend, see How the backend authenticates requests
`NEXT_PUBLIC_API_URL`	frontend	Where the browser and Server Components reach the Express API

Verifying it actually works

Rather than mocked unit tests, the backend's correctness claims are proven against real Redis/Postgres/BullMQ with standalone scripts — this is infrastructure-timing behavior (locks, delays, atomic counters), which a mocked unit test would just assert away rather than exercise:

cd backend
npm run verify:idempotency   # 6 checks — jobId dedup, DB status guard, crash-ambiguity guard
npm run verify:rate-limit    # 11 checks — per-sender cap, global cap, audit trail, end-to-end overflow→redelay
npm run verify:restart       # 7 checks — spawns & SIGKILLs a real worker process, proves the job still fires
npm run verify               # all three, in sequence

Each is safe to run repeatedly against a live dev environment (they create their own throwaway Sender rows and clean up after themselves) and each prints a PASS/FAIL line per assertion, not just a final pass/fail.

All 24 assertions run automatically on every push and PR (.github/workflows/ci.yml), against real Postgres and Redis service containers — not mocked. The workflow starts a worker, waits for its boot line, runs verify:idempotency and verify:rate-limit against it, kills it, then runs verify:restart alone (it manages its own worker processes). A separate job typechecks, lints, and builds the frontend.

verify:restart's whole premise is "zero workers attached to the queue during the delay window" — if you have npm run dev:worker running in another terminal while you run it, that worker will legitimately race in and process the test's job itself, and the script will report two failures that look alarming but are really just "another worker was, in fact, also listening," not a bug. Stop any standalone worker before running verify:restart (or just npm run verify, which runs it last).

Every screen was also driven end-to-end in a real headless browser (Playwright) against the live backend during development: typed the compose form, uploaded a CSV, picked a Send Later time, submitted, and confirmed the campaign appeared in the Scheduled table; separately confirmed the loading skeleton and both empty states by forcing an artificial delay. That caught two real bugs a screenshot alone wouldn't have (Tiptap v3 double-registering its bundled Underline extension, and a toolbar-button focus race that dropped keystrokes) — both fixed, not worked around. It's also how Pagination's "Load more" found its own bug during development: GET /api/emails clamps limit to 100 server-side no matter what's requested, so an early "just ask for a bigger limit" version silently capped at 100 forever — only visible by actually clicking the button in a browser against 100+ real rows, not by reading the code.

Jest (npm test in both backend/ and frontend/) covers what the verify scripts above don't: pure functions with no infrastructure dependency at all, where a mocked unit test is the right tool instead of the wrong one. backend/test/unit/ covers the RFC 4180 CSV parser (csvParser.ts — header detection, quoted fields, BOM stripping, malformed rows) and the scheduling request's Zod schema (schedule.ts — boundary cases on every field); frontend/lib/extractEmails.test.ts covers the client-side recipient-chip regex, including its own documented false-positive case (an email wrapped in parentheses inside prose comes through parens and all — the test asserts that real, slightly-imperfect behavior rather than an idealized one). This is additive to the verify scripts, not a replacement for them - infrastructure-timing behavior still isn't something a unit test can honestly exercise.

Features implemented

Backend

Frontend

Login: real Google OAuth and real email/password (NextAuth CredentialsProvider), plus a register page
Route protection via proxy.ts (optimistic) + getServerSession (authoritative) on every server component, /login and /register included
Every backend call carries a session-backed bearer token — server-fetched pages sign one directly, the Compose form fetches one from /api/session-token first
Dashboard shell: sidebar, live Scheduled/Sent counts, user menu with sign-out
Compose panel: chip-based recipient input (typed + pasted + CSV upload, client-side dedup), merge tag insertion pills ({{firstName}}, {{company}}, etc.), Tiptap rich-text body, delay/hourly-limit inputs, Send Later popover (quick picks + custom date/time)
Scheduled/Sent tables on a shared Table primitive with real "Load more" pagination, per-route loading skeletons (Next.js Suspense), illustrated empty states, and live background sync
Sent rows link to their live Ethereal preview
Typed API layer (types/api.ts mirrors the backend's actual response shapes) and a small shared component set (Button, Input, Avatar, Badge, Popover, Table, Skeleton)
Jest unit coverage for the recipient-extraction regex used by the compose chip input

Design Decisions & Tradeoffs

Key engineering tradeoffs and architectural decisions documented here:

Dual CSV parsing (Client-side chips vs. Server-side RFC 4180): The backend provides a full RFC 4180 parser (backend/src/lib/csvParser.ts) for raw multipart uploads and merge tag variable extraction ({{firstName}}, {{company}}). The browser parses CSVs client-side into editable chip state so users can inspect, remove invalid recipients, and click merge tags before scheduling.
Unified account linking (Google OAuth + Email/Password): Senders are keyed uniquely by verified email in Postgres. A user who originally logged in via Google OAuth can register a password via POST /api/auth/register to link password credentials to their existing Sender record without creating disjoint accounts.
Sliding-window rate limiting over fixed hourly buckets: Send throughput is enforced via Redis sorted sets (backend/src/lib/rateLimiter.ts) using atomic Lua scripts (ZREMRANGEBYSCORE, ZCARD, ZADD). This enforces limits against a strict rolling 60-minute window, eliminating boundary bursts.
At-most-once delivery on worker crash ambiguity: If a worker dies after SMTP handoff begins but before outcome persistence (status === 'sending' on attempt 0), redelivery will not re-send to prevent spam reports. A duplicate email damages domain reputation far worse than a logged send failure.
Persistent Ethereal preview inbox: Auto-minted Ethereal SMTP accounts are cached locally across worker restarts (backend/src/lib/mailer.ts), preserving shareable preview links even across repeated restarts. Setting ETHEREAL_USER/PASS in .env remains supported to pin accounts across machines.

Project structure

backend/
  Dockerfile               multi-stage production container build
  prisma/                  schema + migrations
  src/
    routes/
      emails.ts            scheduling + list/stats API + SSE stream
      credentials.ts       POST /api/auth/register, /login (bcrypt + rate limited + account linking)
    middleware/
      auth.ts              verifies the frontend's session token
      rateLimit.ts         Redis-backed rate limiter for auth routes
    worker.ts               BullMQ worker — separate process from the API (publishes live Redis events)
    lib/
      queue.ts        BullMQ Queue (API side)
      rateLimiter.ts  Redis Lua-script rate limiter (send throughput, sliding window)
      mailer.ts       Ethereal SMTP transport
      csvParser.ts    RFC 4180 CSV parsing + template variable extraction
  scripts/                 standalone verification scripts (see above)
  test/unit/               Jest — csvParser, merge tags, scheduling Zod schema

frontend/
  Dockerfile               multi-stage Next.js production container build
  app/
    login/                 Google + email/password login card
    register/              email/password registration
    dashboard/
      layout.tsx           sidebar shell, session guard, live counts
      compose/             compose panel (CSV, merge tags, rich text, send-later)
      scheduled/, sent/    tables + loading.tsx + live sync
    api/
      auth/[...nextauth]/  NextAuth route handler
      session-token/       mints a backend token for Client Components
  components/
    ui/                    generic primitives (Button, Input, Table, ...)
    dashboard/
      PaginatedEmailList.tsx  "Load more" + real-time status syncing
      ...                     other app-specific components
    compose/               compose-specific components (merge tag pills, recipient chips)
  lib/
    api.ts, clientApi.ts   typed backend API clients (server- and client-side)
    sessionToken.ts        signs the shared-secret token for the backend
    extractEmails.test.ts  Jest — recipient-chip regex
  proxy.ts                 route protection (Next.js 16's renamed Middleware)

.github/workflows/ci.yml   verify scripts + both Jest suites + frontend lint/build, on every push/PR
docker-compose.yml         Postgres + Redis + Backend + Worker + Frontend (full local stack)

Navigation Menu

Repository navigation

About

Repository files navigation

Outbox — Email Job Scheduler

📹 Demo Video

Contents

Architecture

How the backend authenticates requests

Email/password sign-in

Pagination

Tech stack & why

Getting started

Prerequisites

1. Infrastructure

2. Backend

3. Google OAuth (required for real sign-in)

4. Frontend

Environment variables

Verifying it actually works

Features implemented

Design Decisions & Tradeoffs

Project structure

About

Releases

Packages

Contributors

Languages

Name		Name	Last commit message	Last commit date
.github/workflows		.github/workflows
backend		backend
frontend		frontend
.gitattributes		.gitattributes
.gitignore		.gitignore
README.md		README.md
docker-compose.yml		docker-compose.yml

Latest commit

History

Folders and files

Repository files navigation

Outbox — Email Job Scheduler

📹 Demo Video

Contents

Architecture

How the backend authenticates requests

Email/password sign-in

Pagination

Tech stack & why

Getting started

Prerequisites

1. Infrastructure

2. Backend

3. Google OAuth (required for real sign-in)

4. Frontend

Environment variables

Verifying it actually works

Features implemented

Design Decisions & Tradeoffs

Project structure

About

Topics

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages