Skip to content

chore(aw): add .agents/aw verification suite config - #720

Merged
mthines merged 1 commit into
mainfrom
chore/aw-verify-setup
Oct 21, 2026
Merged

mthines merged 1 commit into
mainfrom
chore/aw-verify-setup

Conversation

@dash0-dev

@dash0-dev dash0-dev Bot commented Oct 21, 2026

Copy link
Copy Markdown
Contributor

Summary

Runs /aw-verify setup for LoreKit, so aw-verify, ui-verify, telemetry-verify, aw-tester and aw read one committed config in .agents/aw/ instead of guessing URLs, logins or datasets. It replaces .claude/aw-targets/ and reuses the existing scripts/auth-bootstrap-credentials.mjs login.

Changes

  • environments.yml, local: web on :3000 against the hosted backend by default. A new supabase component (local Supabase + Edge Functions) starts for supabase/** and scripts/seed/** diffs. Its script exits 78 wherever no Docker daemon is reachable (the Agent0 / cloud sandbox), and the run carries on with web. web.sh switches to serve-local while that stack is up; AW_WEB_BACKEND forces a mode.
  • environments.yml, preview: the stable lorekit-pr-<n>-<scope>.vercel.app alias comes from the sticky lorekit-web-preview comment. GitHub deployments only cover main's staging and production deploys. The Vercel bypass header comes from VERCEL_AUTOMATION_BYPASS_SECRET, and the session is recaptured on every run.
  • ui.yml: the UI surface now includes packages/web/src/lib/** and middleware.ts (the default classifier missed 150+ files) and leaves out the app/api route handlers. The adversarial pass may only send mutations to the local stack, never to production Supabase.
  • telemetry.yml: dash0-mcp, with preview reading dataset default, namespace lorekit and web service.name=web. The event patterns match track({ name }) and the auth.* constants.
  • notes.md: repo facts the suite needs. Local web writes to production Supabase. Preview and production share one dataset, so a run's records are told apart by deployment.environment.name. Also covers sandbox limits and seed data.

Validation

  • YAML parses. Both scripts pass sh -n. The refusal (no CLI, or daemon down), booting → ready marker, reuse and cleanup paths were exercised with stubbed docker/supabase/curl, and every web.sh mode was tested with a stubbed pnpm.
  • Event patterns and service/environment values were checked against live Dash0 data, and the surface was checked with is-ui-diff.mjs over every tracked file.
  • Not run: the login probe and smoke spec, because the sandbox has no test credentials or .env.local and cannot reach *.vercel.app. Run /aw-verify setup locally once to confirm both.

Docs

None applied: this is contributor tooling config with no user-visible behaviour, so llms.txt and the MDX docs are unaffected.

Run /aw-verify setup for LoreKit: environments, ui and telemetry config plus
repo notes for the aw-verify / ui-verify / telemetry-verify / aw-tester suite.

- local: `web` on :3000 against the hosted backend by default; a `supabase`
  component (local Supabase + Edge Functions) for supabase/** diffs that
  refuses with exit 78 where no Docker daemon is reachable (Agent0 sandbox)
- preview: stable pr-<n> alias from the sticky web-preview comment, Vercel
  bypass header + recaptured session via scripts/auth-bootstrap-credentials.mjs
- telemetry: dash0-mcp, preview -> dataset default, web RUM event patterns
- remove .claude/aw-targets/, superseded by .agents/aw/environments.yml

Co-authored-by: dash0-dev[bot] <257284812+dash0-dev[bot]@users.noreply.github.com>
@mthines
mthines marked this pull request as ready for review October 21, 2026 18:26
@dash0-dev

dash0-dev Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor Author

Darkplane Automerge did not approve this pull request. Score 70.86 of 100, threshold 75. Blast radius 90, Evidence quality 62, Change footprint 60.

Darkplane Automerge did not approve this pull request. It scored 70.86 of 100, and 75 was required.

  • 🟡 Evidence quality scored 62.
    A human reviewer (mthines) approved the evaluated commit, and CI Summary and Typecheck, Test & Lint (affected) passed. The other checks were neutral, probably skipped. The PR body says the YAML parses and the scripts were exercised with stubs, but the login probe and smoke spec were not run, and this config has no automated tests.
  • 🟡 Change footprint scored 60.
    The change adds verification-suite config under .agents/aw/ (environments.yml, ui.yml, telemetry.yml, notes.md) and removes .claude/aw-targets/local.yml and preview.yml. That looks like a tool-written relocation, and nothing in the product runtime changes. It also adds two new hand-written shell scripts, supabase-local.sh and web.sh, with non-trivial control flow (Docker checks, a pid marker file, seeding). I only saw part of the diff, because the output was truncated partway through web.sh.
The other criterion
  • 🟢 Blast radius scored 90.
    The change is confined to a new .agents/aw/ verification-tool config (YAML, notes, two stack scripts) and deletes .claude/aw-targets/local.yml and .claude/aw-targets/preview.yml. No other tracked file references the deleted paths, and the new files only run when the verification suite is invoked, so a revert is trivial. I did not read the two shell scripts line by line, and the notes say local web can talk to production Supabase.

An AI evaluator produced these scores from the diff, the repository history, and the pull request text. The pull request text is author-controlled, so treat the scores as signals rather than proof. Reviewed for commit 9125b3f.

Was this decision right? Give feedback

@mthines
mthines merged commit c432ca4 into main Oct 21, 2026
20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant