test: website e2e as Markdown specs in the ui-verify grammar; smoke every CLI command and MCP tool

[dash0-dev]

Why

A merge should prove the core still works — every page of the website, every lorekit command and every MCP tool — and adding a test should be trivial, in the same spec format /ui-verify uses.

What changed

• packages/web-e2e — website tests are Markdown blocks in specs/*.md, written in aw-tester's WHEN/THEN grammar (the body of a <!-- ui-verify:v1 --> PR block). A zero-dependency parser runs them in Playwright and refuses, by line number, what it cannot check deterministically (semantic:, intent specs). spec:import / spec:export move specs to and from ui-verify PR blocks and aw-tester's compiled routes. specs.test.ts fails a PR when a spec does not parse or a page.tsx has no spec; every signed-in spec also gets a generated "signed-out visitor goes to /login" check.
• packages/cli/test/smoke — all 32 commands through the real binary: offline inside cli:test, and against preview at deploy time. cli-smoke.test.mjs fails a PR when a command, alias or MCP tool has no smoke.
• scripts/smoke/smoke-mcp-tools.mjs — covers the three tools nothing called (memory.protect, policy.update, groom.run).
• Workflow wiring (.github/pending-workflow-changes.patch) — a web-e2e PR job (local build + local Supabase, no secrets); Playwright on the staged production build before promotion and on lorekit.io after; the CLI lane in smoke-preview and /preview. A failed test-account sign-in only warns, so it never blocks promotion or rolls back, and deploy-time runs record no Playwright traces (a trace keeps headers and cookies, and this repo's artifacts are public).
• Docs — docs/smoke-tests.md and packages/web-e2e/specs/README.md.

How to verify

pnpm nx test web-e2e && node --test packages/cli/test/cli-smoke.test.mjs — the guards plus the offline CLI/MCP lane. The browser suite and the remote CLI lane first run in CI once the patch is applied.

Notes for reviewers

The Dash0 GitHub App cannot push .github/workflows/*. Apply the wiring with gh pr checkout 700 && git am .github/pending-workflow-changes.patch && git rm .github/pending-workflow-changes.patch && git commit -m "ci: drop the applied workflow patch" && git push. Optional repo secrets (each gate skips with a warning while unset): VERCEL_AUTOMATION_BYPASS_SECRET, LOREKIT_TESTING_USER_NAME / LOREKIT_TESTING_USER_PW.

[dash0-dev]

🟠 1 finding — 1 blocking

Re-review at 5ec143d: both earlier notes are fixed. New: journeys.md Specs 1–2 click Docs/Blog links that appear twice on the landing page, so the strict locators fail in every lane.

Blocking: 1 blocking finding (see inline)

_{1 advisory finding below the confidence bar — see Less certain below.}

Checked: 34 of 34 changed files read · 9 of 9 dependent files traced · 10 possible issues → 1 confirmed → 1 posted

Progress: open review threads 2 → 1 → 1 → 2 → 1 · blocking 0 → 0 → 1 → 0 → 1 across the last 5 reviews

Finding	Where	Severity
Docs and Blog clicks match two links on the landing page	packages/web-e2e/specs/journeys.md:12	🟠 high · blocking

What this change reaches — 14 changed exports · 9 dependent files · 9 checked

flowchart LR
  subgraph pr["Changed in this PR"]
    s1["NO_USER_REASON<br/>added"]:::changed
    s2["TARGET<br/>added"]:::changed
    s3["USER_SOURCE<br/>added"]:::changed
    s4["BASE_URL<br/>added"]:::changed
    s5["signInThroughTheForm<br/>added"]:::changed
    s6["STORAGE_STATE<br/>added"]:::changed
    smore["+8 more changed exports<br/>listed below"]:::changed
  end
  s1c1["packages/web-e2e/src/ · 2 files<br/>✓ checked"]:::ok
  s1c2["packages/web-e2e/src/journeys/ · 2 files<br/>✓ checked"]:::ok
  s2c1["packages/web-e2e/src/journeys/ · 2 files<br/>✓ checked"]:::ok
  s2c2["packages/web-e2e/playwright.config.ts<br/>✓ checked"]:::ok
  s2c3["packages/web-e2e/src/specs.e2e.ts<br/>✓ checked"]:::ok
  s3c1["packages/web-e2e/src/ · 2 files<br/>✓ checked"]:::ok
  s3c2["packages/web-e2e/src/journeys/ · 2 files<br/>✓ checked"]:::ok
  s4c1["packages/web-e2e/playwright.config.ts<br/>✓ checked"]:::ok
  s4c2["packages/web-e2e/src/support/fixtures.ts<br/>✓ checked"]:::ok
  s5c1["packages/web-e2e/src/auth.setup.ts<br/>✓ checked"]:::ok
  s5c2["packages/web-e2e/src/journeys/login.e2e.ts<br/>✓ checked"]:::ok
  s6c1["packages/web-e2e/playwright.config.ts<br/>✓ checked"]:::ok
  s6c2["packages/web-e2e/src/auth.setup.ts<br/>✓ checked"]:::ok
  s1 --> s1c1
  s1 --> s1c2
  s2 --> s2c1
  s2 --> s2c2
  s2 --> s2c3
  s3 --> s3c1
  s3 --> s3c2
  s4 --> s4c1
  s4 --> s4c2
  s5 --> s5c1
  s5 --> s5c2
  s6 --> s6c1
  s6 --> s6c2
  classDef changed fill:#eef2ff,stroke:#6366f1,color:#1e1b4b
  classDef ok fill:#e7f6ec,stroke:#16a34a,color:#14532d
  classDef partial fill:#fef9c3,stroke:#ca8a04,color:#713f12
  classDef unknown fill:#f3f4f6,stroke:#9ca3af,stroke-dasharray:4 3,color:#374151
  classDef bad fill:#fde8e8,stroke:#dc2626,color:#7f1d1d
  classDef warn fill:#fff4e5,stroke:#d97706,color:#78350f
  classDef imports fill:#f8fafc,stroke:#64748b,stroke-dasharray:4 3,color:#1e293b

Loading

• NO_USER_REASON (packages/web-e2e/src/support/env.ts) — added change · 4 consumer files · 4 verified unaffected
• TARGET (packages/web-e2e/src/support/env.ts) — added change · 4 consumer files · 4 verified unaffected
• USER_SOURCE (packages/web-e2e/src/support/env.ts) — added change · 4 consumer files · 4 verified unaffected
• BASE_URL (packages/web-e2e/src/support/env.ts) — added change · 2 consumer files · 2 verified unaffected
• signInThroughTheForm (packages/web-e2e/src/support/auth.ts) — added change · 2 consumer files · 2 verified unaffected
• STORAGE_STATE (packages/web-e2e/src/support/auth.ts) — added change · 2 consumer files · 2 verified unaffected
• VERCEL_BYPASS_SECRET (packages/web-e2e/src/support/env.ts) — added change · 2 consumer files · 2 verified unaffected
• E2E_PORT (packages/web-e2e/src/support/env.ts) — added change · 1 consumer file · 1 verified unaffected
• EMPTY_STORAGE_STATE (packages/web-e2e/src/support/auth.ts) — added change · 1 consumer file · 1 verified unaffected
• HAS_AUTH_BACKEND (packages/web-e2e/src/support/env.ts) — added change · 1 consumer file · 1 verified unaffected
• originHeaders (packages/web-e2e/src/support/fixtures.ts) — added change · 1 consumer file · 1 verified unaffected
• ProvisionedUser (packages/web-e2e/src/support/env.ts) — added change · 1 consumer file · 1 verified unaffected
• provisionUser (packages/web-e2e/src/support/auth.ts) — added change · 1 consumer file · 1 verified unaffected
• STARTS_LOCAL_SERVER (packages/web-e2e/src/support/env.ts) — added change · 1 consumer file · 1 verified unaffected

Less certain (1) — advisory, below the confidence bar

• packages/web-e2e/src/spec/grammar.cjs:189 — suggestion: The grammar keeps ?query in a network path, but run.ts records responses by pathname, so such an assertion can never pass. Refuse ? here so the spec author learns it at parse time. (confidence 87)

Review details

Needs attention

Gate	Status	Details
Description vs. code	✅	Description matches the diff: all four surfaces, the 32-command count and the three new MCP checks verified.
Prior review feedback	✅	Earlier review comments are resolved.
Documentation	✅	docs/smoke-tests.md, specs/README.md and the env.ts header explain lanes, secrets and how to add a test.
Self-review signals	✅	No debug logs, leftover TODOs, or unreviewed stubs.
Code review	❌	1 blocking finding — see inline comments.

Found

Quality — produced 10 → posted inline 1 · cleared 1 · carried forward 0 · deferred 0 · below-bar 1
Dropped — 2 below-bar
Severity — 🟠 1 high
Optimality — ran — report mode O1–O3 in-context; 6 approach units judged optimal; 0 proposals (quiet early-exit)
Standards — ran — CLAUDE.md (48 bullets) vs all 34 files; 1 candidate, contradicted (CLAUDE.md:151 is scoped to the Storybook suite); 0 findings
Measurability — ran — quiet: gate 1 not met (no web/mobile/api/worker production path changed; test, CI and docs only)

Run

full · 12 lines in delta · tier deep · depth checkout
⚠️ no sub-agent dispatch available — the intent finder ran in-context with the other finders at effective thoroughness 0.95, instead of as its own sub-agent
Integrations — @playwright/test 1.56.0 (new devDependency of packages/web-e2e, same version as the root playwright override; the lockfile adds it and re-keys next's optional peer) — no breaking change found (lockfile rung; changelog not read)
CI — still pending: Typecheck, Test & Lint (affected) · 4 passed
Memories — 80 indexed · 2 used (1 knowledge · 1 lesson)

• lesson pr-reviewer-run::dash0hq-dash0-19495::lens-cluster-non-execution-2026-09-21
• knowledge knowledge::gated@packages/web-e2e/src/specs.e2e.ts

^{Nothing to report — 0 files skipped.}

^{pr-reviewer · commit 5ec143d · full review · how these findings are produced · updated Oct 17, 2026 5:34pm UTC}