fix(byod): generate the bootstrap from migrations and fix BYOD's limits

.gitattributes

@@ -0,0 +1,8 @@
+# Generated from supabase/migrations/ by `node supabase/byod/generate.mjs`.
+# Collapsed in GitHub diffs; review the migrations, not this projection.
+supabase/byod/bootstrap.sql linguist-generated=true
+
+# The generator's drift check compares bytes. Keep both sides LF on every
+# checkout, so core.autocrlf on Windows cannot turn a clean tree red.
+supabase/migrations/*.sql text eol=lf
+supabase/byod/bootstrap.sql text eol=lf

CLAUDE.md

@@ -254,7 +254,7 @@ Single `:` → 400 error. All segments lowercased. See [docs/scope-format.md](./
 **Critical:** `api_key` auth uses service-role. ALL queries must `.eq('user_id', userId)`.
 Write tools require write permission (`lk_rw_*` / `lk_wo_*`); read tools require read
 permission (`lk_rw_*` / `lk_ro_*`). `lk_ro_*` is denied on write tools; `lk_wo_*` is denied
-on read tools — both with the standard `-32001` permission-denied error. The gating logic
+on read tools — both with the `-32003` forbidden error (`-32001` is unauthenticated). The gating logic
 (`READ_TOOLS`/`WRITE_TOOLS`/`toolRequires`/`tokenPrefixFor`) is a shared pure module,
 `packages/mcp-core/src/auth/permissions.ts`, mirrored self-contained into
 `supabase/functions/mcp/permissions.ts` (the `limits.ts` pattern).

docs/api-tokens.md

@@ -129,20 +129,20 @@ routed into an org the token was never granted.
 
 | Tool | Read + Write (`lk_rw_`) | Read only (`lk_ro_`) | Write only (`lk_wo_`) |
 |------|------------------------|---------------------|------------------------|
-| `memory.write` | ✓ | ✗ (returns -32001) | ✓ |
-| `memory.read` | ✓ | ✓ | ✗ (returns -32001) |
-| `memory.list` | ✓ | ✓ | ✗ (returns -32001) |
-| `memory.delete` | ✓ | ✗ (returns -32001) | ✓ |
-| `memory.search` | ✓ | ✓ | ✗ (returns -32001) |
-| `memory.archive` | ✓ | ✗ (returns -32001) | ✓ |
-| `memory.restore` | ✓ | ✗ (returns -32001) | ✓ |
-| `memory.purge` | ✓ | ✗ (returns -32001) | ✓ |
-| `memory.purge_expired` | ✓ | ✗ (returns -32001) | ✓ |
-| `memory.list_archived` | ✓ | ✓ | ✗ (returns -32001) |
-| `org.list` | ✓ | ✓ | ✗ (returns -32001) |
-| `org.create` | ✓ | ✗ (returns -32001) | ✓ |
-| `org.rename` | ✓ | ✗ (returns -32001) | ✓ |
-| `org.delete` | ✓ | ✗ (returns -32001) | ✓ |
+| `memory.write` | ✓ | ✗ (returns -32003) | ✓ |
+| `memory.read` | ✓ | ✓ | ✗ (returns -32003) |
+| `memory.list` | ✓ | ✓ | ✗ (returns -32003) |
+| `memory.delete` | ✓ | ✗ (returns -32003) | ✓ |
+| `memory.search` | ✓ | ✓ | ✗ (returns -32003) |
+| `memory.archive` | ✓ | ✗ (returns -32003) | ✓ |
+| `memory.restore` | ✓ | ✗ (returns -32003) | ✓ |
+| `memory.purge` | ✓ | ✗ (returns -32003) | ✓ |
+| `memory.purge_expired` | ✓ | ✗ (returns -32003) | ✓ |
+| `memory.list_archived` | ✓ | ✓ | ✗ (returns -32003) |
+| `org.list` | ✓ | ✓ | ✗ (returns -32003) |
+| `org.create` | ✓ | ✗ (returns -32003) | ✓ |
+| `org.rename` | ✓ | ✗ (returns -32003) | ✓ |
+| `org.delete` | ✓ | ✗ (returns -32003) | ✓ |
 
 The four `org.*` rows are new: those tools used to be dashboard-JWT-only on the
 MCP surface, while the REST `/orgs` routes already served `lk_*` tokens. Both

docs/byod.md

@@ -1,80 +1,173 @@
 # Bring Your Own Database (BYOD)
 
-LoreKit's memory tools can point at a Supabase project you control instead of
-the hosted LoreKit backend. This is useful when you have data residency
-requirements, an existing Supabase subscription, or want full control over
-your memory data.
+BYOD stores memories in a Supabase project you control, separate from the
+project that runs LoreKit's `mcp` Edge Function. Use it when memory rows must
+live in a specific database — data residency, an existing Supabase
+subscription, or full control over the data.
+
+It covers the MCP server's tools. Read [How BYOD is wired](#how-byod-is-wired)
+before setting it up: what reaches the storage project, and what does not, is
+the part most likely to surprise you.
+
+## How BYOD is wired
+
+- **It needs your own deployment of the `mcp` function.** The function reads
+  `LOREKIT_STORAGE_URL`, the required `LOREKIT_STORAGE_SERVICE_KEY` and the
+  optional `LOREKIT_STORAGE_ANON_KEY` from its **own Edge Function secrets**. The
+  hosted endpoint (`https://pqokxlhvnosogizsjztg.supabase.co/functions/v1/mcp`)
+  has no BYOD storage configured, and an `env` block in your `.mcp.json` does
+  not reach it — those variables only reach the local `mcp-remote` process on
+  your machine. Deploy the function yourself first: [install.md](./install.md).
+- **Two projects, two jobs.** API tokens and users are resolved in the
+  function's own project; memory rows are read from and written to the
+  storage project. The storage project's `auth.users` never holds those users,
+  which is why the storage schema drops its foreign keys into `auth.users`
+  (step 2).
+- **Only the `mcp` function has a BYOD mode.** The `memories` REST function —
+  which backs the web dashboard, the CLI's remote commands and the hook
+  engine's lesson injection — always uses the function's own project. See
+  [Limitations](#limitations).
 
 ## Setup
 
-### 1. Create a Supabase project
+### 1. Create the storage project
 
-Create a free project at [supabase.com](https://supabase.com). Note your
-project URL and API keys from **Settings → API**.
+Create a project at [supabase.com](https://supabase.com). Note its URL and API
+keys (**Project Settings → API**) and its database connection string
+(**Connect**). Enable the `pg_cron` extension (**Database → Extensions**)
+before applying the schema: the purge jobs are scheduled only when it is
+already enabled.
 
 ### 2. Apply the LoreKit schema
 
-Run the bootstrap command to create the required tables and functions:
+The storage project needs the full LoreKit schema — the same migrations the
+hosted project runs. Pick one route.
+
+**Supabase CLI (recommended — the same commands upgrade it later):**
 
 ```bash
-LOREKIT_STORAGE_URL=https://your-project.supabase.co \
-LOREKIT_STORAGE_SERVICE_KEY=your-service-role-key \
-lorekit bootstrap
+git clone https://github.com/mthines/lorekit.git
+cd lorekit
+supabase link --project-ref <storage-project-ref>
+supabase db push
+psql "$DATABASE_URL" -v ON_ERROR_STOP=1 -f supabase/byod/storage-overlay.sql
 ```
 
-If the CLI is not installed, apply the schema directly with psql:
+The last line applies the **storage overlay**: it drops every foreign key from
+a `public` table into `auth.users`, because the users live in the function's
+project, not this one. Without it, every write made with an `lk_*` token fails
+with `memories_user_id_fkey`. It is idempotent; re-run it after every
+`db push`. Never apply it to the project that runs the `mcp` function.
+
+**One SQL file (empty database only):**
 
 ```bash
-psql "$DATABASE_URL" -f path/to/bootstrap.sql
+psql "$DATABASE_URL" -v ON_ERROR_STOP=1 -f supabase/byod/bootstrap.sql
 ```
 
-Or copy the raw SQL from the [LoreKit repository](https://github.com/mthines/lorekit/blob/main/supabase/byod/bootstrap.sql) and paste it into the Supabase dashboard → SQL Editor.
+or paste [`supabase/byod/bootstrap.sql`](https://github.com/mthines/lorekit/blob/main/supabase/byod/bootstrap.sql)
+into the Supabase dashboard → **SQL Editor** and run it (the file is about 1 MB;
+if the editor refuses it, use `psql`). `$DATABASE_URL` is the connection string
+from **Connect**; its password is your **database password**, not an API key.
 
-### 3. Configure your MCP server
+`bootstrap.sql` is generated from `supabase/migrations/` (CI fails when the two
+disagree) and ends with the storage overlay. It applies everything in one
+transaction, so a failure leaves the database untouched; it refuses to run
+against a database that already has the schema; and it records each migration
+in Supabase's migration history, so the CLI route above upgrades the database
+later.
 
-In your `.mcp.json` or equivalent MCP config, add the env vars:
+`lorekit bootstrap`, run with `LOREKIT_STORAGE_URL` and a key in your shell,
+checks whether the project is reachable and already has the schema, and prints
+these commands. It does not run any DDL itself — a Supabase API key cannot.
 
-```json
-{
-  "env": {
-    "LOREKIT_STORAGE_URL": "https://your-project.supabase.co",
-    "LOREKIT_STORAGE_ANON_KEY": "your-anon-key"
-  }
-}
-```
+### 3. Point your `mcp` function at it
 
-Or set them in your shell before starting the MCP server.
+Set the storage secrets on the project that runs your `mcp` function:
 
-## Billing
+```bash
+supabase secrets set --project-ref <function-project-ref> \
+  LOREKIT_STORAGE_URL=https://<storage-project-ref>.supabase.co \
+  LOREKIT_STORAGE_ANON_KEY=<storage-anon-key> \
+  LOREKIT_STORAGE_SERVICE_KEY=<storage-service-role-key>
+```
 
-Memories stored in your own database are **not counted** against LoreKit's
-hosted memory-count billing. LoreKit has no visibility into your private
-database and cannot meter memories stored there.
+The service key is **required**: without it, every call to the function fails
+with an error naming `LOREKIT_STORAGE_SERVICE_KEY`. The storage project's read policies need `auth.uid()`, which an API
+key never carries, so an anon-key client could write but would read back
+nothing. `LOREKIT_STORAGE_ANON_KEY` is accepted but unused once the service key
+is set.
 
-BYOD users are on a flat-rate or open-source tier — no per-memory billing.
+The service key bypasses row-level security, so the function scopes every call
+itself: reads and writes made with an `lk_*` token, or from a Supabase session,
+are filtered to and stamped with that caller's user id. Only the service-role
+tier (CI) sees every row.
 
-You are responsible for configuring memory limits and rate limiting in your
-own Supabase project. The bootstrap schema ships with a default 5,000-memory
-cap trigger, which you can adjust by editing the `lorekit_default_limit`
-function in your database.
+### 4. Check it
 
-## Limitations
+```bash
+LOREKIT_STORAGE_URL=https://<storage-project-ref>.supabase.co \
+LOREKIT_STORAGE_ANON_KEY=<storage-anon-key> \
+lorekit doctor
+```
 
-The following LoreKit features are **not available** with BYOD:
+The `byod storage` line passes when the project is reachable and holds the
+schema. It fails when the project is unreachable, rejects the key, or does not
+have the schema yet, and says which.
 
-- **Org/scope sharing** — requires the hosted backend's org management RPCs
-- **Web dashboard** — tightly coupled to hosted Supabase Auth
-- **Audit logs** — not included in the BYOD schema
-- **Webhook secrets** — not included in the BYOD schema
-- **Automatic schema upgrades** — when LoreKit ships schema changes, you
-  must apply them manually. Run `lorekit doctor` to check compatibility.
+## Upgrading
 
-## Checking connectivity
+When a LoreKit release adds migrations, upgrade the storage project from a
+checkout of that release:
 
 ```bash
-LOREKIT_STORAGE_URL=https://your-project.supabase.co \
-LOREKIT_STORAGE_ANON_KEY=your-anon-key \
-lorekit doctor
+supabase link --project-ref <storage-project-ref>
+supabase db push
+psql "$DATABASE_URL" -v ON_ERROR_STOP=1 -f supabase/byod/storage-overlay.sql
 ```
 
-The output will include a `[byod] storage ok` line if the database is reachable.
+`db push` applies only the migrations the database has not recorded, which
+works the same whether you set the database up with `db push` or with
+`bootstrap.sql`. Re-run the overlay after it, in case a new migration added a
+foreign key into `auth.users`. Upgrade the storage project before deploying the
+matching `mcp` function: the function calls the RPCs those migrations define.
+
+### Databases set up from the old `bootstrap.sql`
+
+`bootstrap.sql` used to be a hand-written subset of the schema. A database set
+up from that version has no migration history — `select count(*) from
+supabase_migrations.schema_migrations` errors or returns 0 — so `db push` would
+try to replay every migration from the first and fail, and its schema is
+missing most of the functions the current `mcp` function calls. Move to a
+fresh storage project instead: apply the current schema to it as above, copy the `memories`
+rows across (for example `pg_dump --data-only --table=public.memories` from the
+old database, loaded with `psql` into the new one), then point
+`LOREKIT_STORAGE_URL` at the new project.
+
+## Billing
+
+Memories stored in your own database are **not counted** against LoreKit's
+hosted memory-count billing. LoreKit has no visibility into your private
+database and cannot meter memories stored there.
+
+The memory cap (default 5,000 active memories per user) and the limits table
+live in your database: adjust them through `lorekit_default_limit` and
+`user_limits` there. In BYOD mode the function skips the hosted rate limiter
+and does not record hosted usage events.
+
+## Limitations
+
+- **Only the MCP tools use the storage project.** The web dashboard, the REST
+  API, the CLI's remote commands (`list`, `search`, `show`, `write` against the
+  remote store) and the hook engine's lesson injection go through the `memories`
+  function, which has no BYOD mode. They read and write the function's own
+  project and do not see memories stored through BYOD. To run the dashboard on
+  a database you control, self-host the whole stack instead
+  ([install.md](./install.md)).
+- **No row-level security in the storage project.** The function talks to it
+  with the service key and scopes every call in its own code (step 3); RLS
+  policies there do not apply to it.
+- **Tested surface.** The BYOD smoke test exercises write, read, list, search
+  and delete, and runs only where its secrets are configured. The schema now
+  also contains the org, audit, retention and webhook tables and RPCs, but BYOD
+  has not been exercised for those features — treat them as unsupported.

docs/cli.md

@@ -55,7 +55,10 @@ frontmatter, checked in BOTH the project and global install locations): a `curre
 `pass` naming its version, an `outdated` or `unknown` (legacy, predates version stamping) install is
 a `warn` — never a `fail`, the skill still works — naming the installed and shipped versions and
 pointing at `lorekit update --<scope>`, the same scope-aware wording the hooks-upgrade line above it
-uses.
+uses. When `LOREKIT_STORAGE_URL` is set, a `byod storage` check probes that project with
+`LOREKIT_STORAGE_ANON_KEY` or, failing that, `LOREKIT_STORAGE_SERVICE_KEY` (`probeByodStorage` in `shared/byod.mjs` — one PostgREST
+`GET /rest/v1/memories?limit=1`, plain `fetch`): it passes when the `memories` table answers and
+FAILS when the project is unreachable, rejects the key, or has no LoreKit schema yet.
 
 ### `update`
 
@@ -297,6 +300,18 @@ every param `encodeURIComponent(JSON.stringify(value))` and omits any param equa
 previously raw-encoded `?scope=<scope>`, which the dashboard's `JSON.parse` rejected into an
 all-scopes fallback; write-confirmations now deep-link to the exact lesson
 
+### `bootstrap`
+
+BYOD only ([byod.md](./byod.md)). Probes the project named by `LOREKIT_STORAGE_URL` with
+`LOREKIT_STORAGE_ANON_KEY` or `LOREKIT_STORAGE_SERVICE_KEY` (the same `probeByodStorage` as
+`doctor`), reports whether the LoreKit schema is already there, and prints the three routes that
+apply it: `psql -v ON_ERROR_STOP=1 -f supabase/byod/bootstrap.sql`, the SQL Editor, or
+`supabase link` + `supabase db push` followed by `supabase/byod/storage-overlay.sql` (which
+`bootstrap.sql` already includes). It runs no DDL — PostgREST, the only thing a Supabase API key
+reaches, cannot. The published package does not ship `supabase/`, so outside a LoreKit checkout it
+prints the file's download URL instead of a local path. It never prints a key: the `psql`
+connection string's password is the database password, not the service-role key.
+
 ### `migrate`
 
 Relocate a LoreKit-format local store, or push one to the hosted store — `--from <path>` is required

docs/deployment.md

@@ -1062,6 +1062,18 @@ The collision-proof end state is Supabase's **timestamp filenames**
 (`YYYYMMDDHHMMSS_…`), which are monotonic and parallel-safe — a larger,
 rename-everything change to consider separately.
 
+#### Every migration regenerates the BYOD bootstrap
+
+`supabase/byod/bootstrap.sql` — the single file a BYOD user applies to their own
+database ([byod.md](./byod.md)) — is **generated** from `supabase/migrations/`
+by `node supabase/byod/generate.mjs`. After adding or editing a migration, run
+it and commit the result. The `supabase` project's `test` target
+(`supabase/byod/generate.test.mjs`) fails when the committed file and the
+migrations disagree, and the `check` job's `nx affected -t test` runs it on any
+change under `supabase/`. The file used to be a hand-written subset, which
+drifted until BYOD's `memory_write` could no longer be resolved by the edge
+function's calls — a generated projection cannot drift.
+
 ---
 
 ## Prerequisites

docs/install.md

@@ -265,9 +265,12 @@ Store a read+write token as `LOREKIT_TOKEN` in your repo secrets, then use it in
 
 | Symptom | Likely cause | Fix |
 |---------|-------------|-----|
-| `{"code":-32001}` on write | Read-only (`lk_ro_`) token | Generate a `lk_rw_` or `lk_wo_` token |
-| `{"code":-32001}` on read | Write-only (`lk_wo_`) token, or missing/wrong token | Generate a `lk_rw_` or `lk_ro_` token; check the `Authorization` header |
-| `{"code":-32603}` | Scope validation error | Use `::` as separator; check scope format |
+| `{"code":-32003}` on write | Read-only (`lk_ro_`) token | Generate a `lk_rw_` or `lk_wo_` token |
+| `{"code":-32003}` on read | Write-only (`lk_wo_`) token | Generate a `lk_rw_` or `lk_ro_` token |
+| `{"code":-32001}` | Missing, invalid, expired or revoked token | Check the `Authorization` header; generate a new token |
+| `{"code":-32603}` naming `LOREKIT_STORAGE_SERVICE_KEY` | BYOD storage URL set without the service key | Set `LOREKIT_STORAGE_SERVICE_KEY` on the function project (see [byod.md](./byod.md)); retrying will not help |
+| `{"code":-32603}` otherwise | Server fault — DB unreachable or an unexpected exception | Retry; if it persists, check the function logs |
+| Tool result with `isError: true` mentioning the scope | Invalid scope format | Use `::` as separator; check scope format |
 | `{"db":false}` on health check | DB not reachable from Edge Function | Verify Supabase secrets in Step 5 |
 | Dashboard auth loop | Redirect URL not set | Add your URL in Supabase Auth settings (Step 8) |