test: website e2e as Markdown specs in the ui-verify grammar; smoke every CLI command and MCP tool

[dash0-dev]

Why

A merge should prove the core still works — every page of the website, every lorekit command and every MCP tool — and adding a test should be trivial, in the same spec format /ui-verify uses.

What changed

• packages/web-e2e — website tests are Markdown blocks in specs/*.md, written in aw-tester's WHEN/THEN grammar (the body of a ui-verify v1 PR block). A zero-dependency parser runs them in Playwright and refuses, by line number, what it cannot check deterministically (semantic:, intent specs). spec:import / spec:export move specs to and from ui-verify PR blocks and aw-tester's compiled routes. specs.test.ts fails a PR when a spec does not parse or a page.tsx has no spec; every signed-in spec also gets a generated "signed-out visitor goes to /login" check.
• packages/cli/test/smoke — all 32 commands through the real binary: offline inside cli:test, and against preview at deploy time. cli-smoke.test.mjs fails a PR when a command, alias or MCP tool has no smoke.
• scripts/smoke/smoke-mcp-tools.mjs — covers the three tools nothing called (memory.protect, policy.update, groom.run).
• Workflows (ci.yml, deploy.yml, preview.yml) — a web-e2e PR job (local build + local Supabase, no secrets); Playwright on the staged production build before promotion and on www.lorekit.io after (the apex redirects, which would hide same-origin 5xx); the CLI lane in smoke-preview and /preview. A failed test-account sign-in only warns, so it never blocks promotion or rolls back, and deploy-time runs record no Playwright traces (a trace keeps headers and cookies, and this repo's artifacts are public).
• /onboarding crash fix — the new suite caught a signed-in visit throwing React chore: release main #310 before redirecting: its loading.tsx streamed the server redirect() to the client (the crash next.config.ts already documents for /settings). The skeleton is deleted.
• Docs — docs/smoke-tests.md and packages/web-e2e/specs/README.md.

How to verify

pnpm nx test web-e2e && node --test packages/cli/test/cli-smoke.test.mjs — the guards plus the offline CLI/MCP lane. The browser suite runs in the Web — Playwright e2e CI job (58 tests; also green locally 3× against a local production build).

Notes for reviewers

Optional repo secrets (each gate skips with a warning while unset): VERCEL_AUTOMATION_BYPASS_SECRET, LOREKIT_TESTING_USER_NAME / LOREKIT_TESTING_USER_PW. The title is test:, so release notes will not list the /onboarding fix unless the squash title changes.

[dash0-dev]

⚠️ No findings — 1 gate needs attention

The www.lorekit.io fix covers every production step and the docs, so no new code findings. The PR description still points at the deleted workflow patch and leaves out the /onboarding change.

Warnings: Body still cites the deleted pending-workflow-changes.patch; wiring is in ci/deploy/preview.yml. Omits /onboarding

Checked: 34 of 39 changed files read · 6 of 9 dependent files traced · no possible issues found

Progress: open review threads 1 → 2 → 1 → 1 → 0 · blocking 1 → 0 → 1 → 0 → 0 across the last 5 reviews

What this change reaches — 15 changed exports · 9 dependent files · 6 checked · 3 not checked

flowchart LR
  subgraph pr["Changed in this PR"]
    s1["NO_USER_REASON<br/>added"]:::changed
    s2["TARGET<br/>added"]:::changed
    s3["USER_SOURCE<br/>added"]:::changed
    s4["allowing<br/>added"]:::changed
    s5["BASE_URL<br/>added"]:::changed
    s6["signInThroughTheForm<br/>added"]:::changed
    smore["+9 more changed exports<br/>listed below"]:::changed
  end
  s1c1["packages/web-e2e/src/ · 2 files<br/>✓ checked"]:::ok
  s1c2["packages/web-e2e/src/journeys/ · 2 files<br/>✓ checked"]:::ok
  s2c1["packages/web-e2e/src/journeys/ · 2 files<br/>✓ checked"]:::ok
  s2c2["packages/web-e2e/playwright.config.ts<br/>✓ checked"]:::ok
  s2c3["packages/web-e2e/src/specs.e2e.ts<br/>✓ checked"]:::ok
  s3c1["packages/web-e2e/src/ · 2 files<br/>✓ checked"]:::ok
  s3c2["packages/web-e2e/src/journeys/ · 2 files<br/>✓ checked"]:::ok
  s4c1["packages/web-e2e/src/journeys/ · 2 files<br/>✓ checked"]:::ok
  s4c2["packages/web-e2e/src/specs.e2e.ts<br/>✓ checked"]:::ok
  s5c1["packages/web-e2e/playwright.config.ts<br/>✓ checked"]:::ok
  s5c2["packages/web-e2e/src/support/fixtures.ts<br/>✓ checked"]:::ok
  s6c1["packages/web-e2e/src/auth.setup.ts<br/>✓ checked"]:::ok
  s6c2["packages/web-e2e/src/journeys/login.e2e.ts<br/>✓ checked"]:::ok
  s1 --> s1c1
  s1 --> s1c2
  s2 --> s2c1
  s2 --> s2c2
  s2 --> s2c3
  s3 --> s3c1
  s3 --> s3c2
  s4 --> s4c1
  s4 --> s4c2
  s5 --> s5c1
  s5 --> s5c2
  s6 --> s6c1
  s6 --> s6c2
  classDef changed fill:#eef2ff,stroke:#6366f1,color:#1e1b4b
  classDef ok fill:#e7f6ec,stroke:#16a34a,color:#14532d
  classDef partial fill:#fef9c3,stroke:#ca8a04,color:#713f12
  classDef unknown fill:#f3f4f6,stroke:#9ca3af,stroke-dasharray:4 3,color:#374151
  classDef bad fill:#fde8e8,stroke:#dc2626,color:#7f1d1d
  classDef warn fill:#fff4e5,stroke:#d97706,color:#78350f
  classDef imports fill:#f8fafc,stroke:#64748b,stroke-dasharray:4 3,color:#1e293b

Loading

• NO_USER_REASON (packages/web-e2e/src/support/env.ts) — added change · 4 consumer files · 4 verified unaffected
• TARGET (packages/web-e2e/src/support/env.ts) — added change · 4 consumer files · 4 verified unaffected
• USER_SOURCE (packages/web-e2e/src/support/env.ts) — added change · 4 consumer files · 4 verified unaffected
• allowing (packages/web-e2e/src/support/fixtures.ts) — added change · 3 consumer files · 3 verified unaffected
• BASE_URL (packages/web-e2e/src/support/env.ts) — added change · 2 consumer files · 2 verified unaffected
• signInThroughTheForm (packages/web-e2e/src/support/auth.ts) — added change · 2 consumer files · 2 verified unaffected
• STORAGE_STATE (packages/web-e2e/src/support/auth.ts) — added change · 2 consumer files · 2 verified unaffected
• VERCEL_BYPASS_SECRET (packages/web-e2e/src/support/env.ts) — added change · 2 consumer files · 2 verified unaffected
• E2E_PORT (packages/web-e2e/src/support/env.ts) — added change · 1 consumer file · 1 not traced
• EMPTY_STORAGE_STATE (packages/web-e2e/src/support/auth.ts) — added change · 1 consumer file · 1 not traced
• HAS_AUTH_BACKEND (packages/web-e2e/src/support/env.ts) — added change · 1 consumer file · 1 verified unaffected
• originHeaders (packages/web-e2e/src/support/fixtures.ts) — added change · 1 consumer file · 1 verified unaffected
• ProvisionedUser (packages/web-e2e/src/support/env.ts) — added change · 1 consumer file · 1 not traced
• provisionUser (packages/web-e2e/src/support/auth.ts) — added change · 1 consumer file · 1 verified unaffected
• STARTS_LOCAL_SERVER (packages/web-e2e/src/support/env.ts) — added change · 1 consumer file · 1 not traced

Review details

Needs attention

Gate	Status	Details
Description vs. code	⚠️	Body still cites the deleted pending-workflow-changes.patch; wiring is in ci/deploy/preview.yml. Omits /onboarding.
Prior review feedback	✅	Earlier review comments are resolved.
Documentation	✅	docs/smoke-tests.md, specs/README.md, CLAUDE.md package map and adding-an-operation.md cover the suites.
Self-review signals	✅	No debug logs, leftover TODOs, or unreviewed stubs.
Code review	✅	The multi-lens review found no blocking issues.

Found

Quality — produced 0 → posted inline 0 · cleared 0 · carried forward 0 · deferred 0 · below-bar 0
Optimality — ran — approach fits the repo's table-driven guard pattern; no proposal
Standards — ran — 2 governing docs, 62 bullets; 0 findings
Measurability — ran — quiet: test and CI wiring only, no new observable product behaviour

Run

full · 19 lines in delta · tier deep · depth checkout
Integrations — dependency finder: @playwright/test 1.56.0 added to the new web-e2e package, same version as the root pin; no usage-site change
CI — still pending: Typecheck, Test & Lint (affected), Web — Playwright e2e (local build + local Supabase), Web — Storybook interaction + visual tests (changed) · 11 passed
Memories — no relevance rules or lessons consulted

^{Nothing to report — severity, 0 files skipped.}

^{pr-reviewer · commit 6297f24 · full review · how these findings are produced · updated Oct 18, 2026 9:15am UTC}

[mthines]

Local verify — ✅ verified

The suite now passes end to end on a local production build + local Supabase: 58/58 on three consecutive runs, signed-in tests included. Getting there took five fixes, one per commit: three test bugs that made CI's web-e2e job red on every retry, a sign-out flake, and a pre-existing /onboarding crash the new suite caught. Two low items are left open.

PR	#700 · no ticket
Commit verified	026fbf4c (branch test/playwright-e2e-and-smoke-coverage)
Stack	supabase (start) · website next build with NEXT_PUBLIC_USE_LOCAL_SUPABASE=true + next start :3100 (Playwright webServer) · this checkout
Telemetry	default via dash0-dev · SKIPPED (see Under the hood)
Run	2026-10-18T08:37:09Z → 2026-10-18T08:53:06Z

Acceptance criteria

ID	Criterion	Result	Evidence
AC-1	Full Playwright suite (setup, public, signed-in, sign-out) passes locally with signed-in tests running	fixed → pass (F-2, F-3, F-4, F-5)	58 passed × 3 runs; CI run on e71d75a1 failed the same 3 tests
AC-2	Generated "signed-out visitor → /login?next=" check per signed-in spec	pass	24 auth-gate tests green
AC-3	nx test web-e2e passes; fails with file:line on a bad spec and on a page with no spec	fixed → pass (F-1)	20/20; EC-1–3 below
AC-4	Runner fails a spec on uncaught exception / console.error / same-origin 5xx	pass	It caught F-3 (React #310) and F-5 (401s)
AC-5	A false assertion fails	pass	Mutated heading → forgot-password spec ✘
AC-6	spec:export → spec:import round-trips; v2 intent imports as a refused draft	pass	Flow lines identical; v2 → url, preconditions, network: translated, 2 TODO lines refused at v2.md:11
AC-7	Offline CLI smoke passes; guard fails when a command has no smoke	pass	42/42; removing purge-expired's smokes → "No smoke case for: purge-expired"
AC-8	Applied workflow patch is valid	pass	actionlint: no new findings (3 SC2129 style notes, all pre-existing on main)

Edge cases

ID	Probe	Result	Evidence
EC-1	Typo'd locator key / unknown action	pass (F-6 on wording)	pages.md:25: unknown locator key "rol"
EC-2	New page.tsx with no spec	pass	Guard names /zz-probe
EC-3	semantic: in a spec	pass	pages.md:219: semantic: needs an LLM judge
EC-4	Assertion on a heading that does not exist	pass	Readable locator failure
EC-5	No test user configured	pass	24 skipped, 34 passed
EC-6	E2E_SUPABASE_URL = production	pass	Refused before any request: "throwaway users are only provisioned on a local stack"
EC-7	sign-out project revokes sessions	pass	Runs last; signed-in project unaffected
EC-8	Duplicate header/footer links	pass	Landing journeys scoped; green
EC-9	Export → import renumbering	pass	Round trip identical
EC-10	Parallel workers, repeated runs	fixed → pass (F-5)	3/3 green after fix

Not applicable (2)

• Viewport / a11y / numbers & time: the PR adds tests, not UI.
• Permissions beyond the auth gate: the suite has no role checks.

Findings

ID	Severity	Kind	What happened	Status
F-3	high	crash	pre-existing: signed-in /onboarding threw React #310 in Next's app-router (its loading.tsx turned the server redirect into a client-side hop)	fixed
F-2	high	functional	test.use({ allowedConsoleErrors: [/a/, /b/] }) → "not iterable": Playwright reads a 2-element array with an object second element as [value, options]	fixed
F-4	high	functional	App-shell journey looked for Settings inside Main navigation; it is in the sidebar footer	fixed
F-1	medium	performance	web-e2e:test depended on web:build:production, a full website build before a 7 ms guard	fixed
F-5	medium	functional	Sign-out flaked on 401s from /lore queries still in flight when the token was revoked	fixed
F-6	low	ux	Unknown WHEN action is reported as a CSS-selector error (right line, wrong reason)	open
F-7	low	performance	pre-existing: evals:test still pulls web:build into nx affected -t test	open

Fixes

Commit	Finding	Change	Re-verified by
643537de	F-1	web-e2e test.dependsOn: []	nx run web-e2e:test --graph → test alone; 20/20
27664a4f	F-2	allowing(patterns) helper returns the explicit tuple; both call sites use it	Wrong-password journey green
30d73453	F-3	Delete (dashboard)/onboarding/loading.tsx	No pageerror on next dev; Onboarding spec green
f16dc992	F-4	Unscoped Settings link query (mobile copy is display:none)	App-shell journey green
026fbf4c	F-5	Allow status of 401 on /functions/v1/ in the sign-out test only	3/3 full runs green

Screenshots — 0 (0 to upload)

_{No screenshots: the final runs are green, and evidence for the fixed failures is in the CI logs and the commit bodies.}

Under the hood

Telemetry not reviewed: the local build runs with NEXT_TELEMETRY_DISABLED and the served edge functions export only when supabase/functions/.env sets OTLP keys (absent), so the configured Dash0 dataset holds nothing for this run. Second witness instead: the edge-runtime container logged 786 lines across all runs, none an error (401s only from the sign-out race, F-5).

Open and escalated

• F-6 (low): give the parser an "unknown action — use one of: page loads, is clicked, …" message. Recommendation: follow-up, not blocking.
• F-7 (low, pre-existing): evals:test → web:build. Suggested ticket: "evals:test should not build the website".
• Not verified locally: the deploy-time lanes (Playwright against a staged build / lorekit.io, CLI against preview). They need VERCEL_AUTOMATION_BYPASS_SECRET and the test-account secrets; /preview exercises the CLI lane.

Lessons

• local-verify-lessons::web-e2e-local-run-recipe (new)

_{Generated by /local-verify · full artifacts in .agent/test/playwright-e2e-and-smoke-coverage/local-verify/}

[github-actions]

The dashboard preview for this PR — redeployed on each push that changes the web app.

Project	Deployment	Actions	Updated
lorekit	Ready	Preview	Oct 18, 2026 8:54am UTC

_{Preview always points at this PR's latest commit; Deployment is the immutable build of 026fbf4. · Comment /web-preview to force a redeploy. · Workflow logs}

[github-actions]

✅ Preview deploy succeeded

Branch: test/playwright-e2e-and-smoke-coverage @ 026fbf4

🔗 Open preview → https://lorekit-8ojpm24p0-mads-thines-projects.vercel.app

Step	Result
API (Supabase edge functions + migrations)	✅ success
Web (Vercel preview)	✅ success
Smoke tests	✅ success

Endpoints:

• API: https://dbfcmczhzvzemrmysbgp.supabase.co/functions/v1/mcp
• Web: https://lorekit-8ojpm24p0-mads-thines-projects.vercel.app

View workflow run