feat: filter in / filter out from the detail panel, with tag exclusions that AND correctly

docs/decisions.md

@@ -20,6 +20,8 @@ rationale before proposing a change that reverses one.
 
 **The Lore Explorer filters through ONE two-level command menu and a row of pills — never one picker per dimension, and never a client-side narrowing.** The Explorer filtered on labels alone, behind a bespoke popover (`LabelFilter`). It now filters on nine dimensions — label, kind, host, owner, agent, trigger, repo, branch, pull request — and the shape that scales is Linear's: a single **Filter** trigger opening a menu whose first level is the DIMENSION list and whose second is that dimension's values, with committed conditions rendered below as three-segment pills. A picker per dimension would put nine controls in a row that has to survive a phone and would still not answer "what else can I filter by?"; here a new dimension costs one row in `FILTER_FIELDS` and nothing else. **Semantics are OR within a dimension, AND across dimensions**, because that is the only combination a flat pill row can render without inventing a precedence grammar — cross-dimension OR and nested groups stay in `POST /memories/search`'s `filter` tree, which already expresses them. **Every dimension is filtered SERVER-side** (`GET /memories` gained `source_agent` / `trigger` / `origin_repo` / `origin_branch` / `origin_pr` plus a `*_mode` of `in`/`nin`, `tags_mode` gained `none`, and `owner` — value `personal` or an org slug — landed in 00064), never by narrowing the loaded pages: those are keyset pages, so a client-side narrowing of a filter the user can actually type is "no matches" on page one and matches on page three. **Owner was the last exception** — a separate client-side `OwnershipFilterBar` over the loaded pages — and 00064 folded it into the same machinery as a real facet dimension: `lorekit_memory_facets`, `lorekit_memory_activity` and `GET /memories` all resolve `personal` → `org_id is null` and an org slug → the caller's member org (slugs resolve against member orgs only, which the visibility predicate already guarantees), so the owner filter agrees with the list, the drill-down counts and the stat header like every other dimension. The `?owner=` deep link the accept-invite flow used is superseded: it is still READ (a `'personal'` value → an `owner` filter; `'all'` / the legacy `{orgId}` form → none) but never written, and the flow now deep-links a slug-keyed `owner` filter directly. **The value catalog is its own query, never derived from the loaded pages** (`GET /memories/facets`, 00052) for `useTagCatalog`'s reason, which only sharpens with more dimensions — derived from the loaded pages the options would shrink to whatever the current page returned, and the cross-dimension type-ahead could only ever surface values already visible in the list, which is exactly when you did not need it. The ENDPOINT is no longer filter-INDEPENDENT: **00057 made the counts drill-down** — it takes the caller's active filters and counts each dimension with every OTHER one applied but not its own (self-exclusion), so a value's count is what selecting it would yield while the dimension you are standing in still shows everything you could switch to. Unfiltered, it returns 00052's global catalog unchanged; the Explorer now passes its active filter bar and the selected scope (`listFacetsPostRequest` ← `filtersToFacetBody` + `useFacetCatalog`'s `scope`, keyed on both — `listFacetsRequest` ← `filtersToFacetParams` is the equivalent GET form, kept and spec-covered but no longer called from this package), so its menu counts drill down as you build the bar and match the scoped list. `kind`/`host`/`owner` have `FILTER_FIELDS` rows now, so all nine dimensions are filterable — `FacetName` and `FilterField` are one-to-one, and `filters.spec.ts` executes that against `MemoryFacetSchema` so a facet added server-side without a pill fails a test instead of being silently dropped. The one residual widen/switch loss is that a value counting zero under the other filters emits no row, so it drops out of the menu until the filter is cleared. The counts do NOT mirror `q` / `key` — with a search active they are upper bounds, not exact yields, because mirroring `q` would mean a second implementation of `likeNeedle`'s escaping in plpgsql. **`created_since` / `created_until` and the five retention thresholds ARE mirrored as of 00108**, and the gap they closed was a disagreement rather than a missing feature: only the list applied the thresholds, so `max_opened_count => 0` narrowed the rows and left the facet counts, the stat cards and the matrix counting the un-narrowed population — every number on the page describing a different set from the one below it. All four readers now compose ONE shared `lorekit_match_retention`, so a count cannot drift from the list it describes; it is applied in the `base` CTE's WHERE rather than as an `ok_*` flag, which is what makes a threshold narrow even the SELF-EXCLUDED dimension — deliberately unlike a dimension, because self-exclusion exists so the dimension you are standing in still lists what you could switch to, and switching `host` does not stop you looking at lore older than 30 days. **Selection is URL-backed** (`?filters=`), so a filtered view is a link; the pre-existing `?tags=` param is still READ (never written) and translated by `filtersFromLegacyTags` into a `label` filter with the `all` operator, which is what it has always meant — links in PRs, Slack and `lorekit link` output keep resolving to the filter they name. The decision logic is pure and node-tested in `lib/filters.ts`; `FilterMenu`/`FilterPill` are the shell. **Do not re-add a per-dimension trigger, a client-side dimension filter, or a second catalog query.** **The Explorer reads those server-side filters over a JSON BODY** (`POST /memories/list` / `/facets` / `/activity`), not a query string: a query string caps each dimension at 2048 characters (`ValueListSchema`), so the number of values a dimension can hold depended on how long the values happened to be — with realistic ~28-character host names the Explorer stopped loading somewhere between 50 and 75 selected values and rendered only "Failed to load memories. Please refresh." — and even under that cap eight dimensions compose a URL past what the gateway accepts, which fails with no LoreKit error envelope at all. Raising the cap only moves the first wall and makes the second arrive first, so this is a transport change rather than a bigger number. **The GET forms stay fully supported and unchanged** (the CLI, MCP and API-token callers use them, and a link carrying a few filters is better as a URL); the two transports cannot drift because neither handler decides anything — `dimensionsFromQuery` and `dimensionsFromBody` (`@lorekit/schemas/dimensions`) both produce ONE `MemoryDimensions` and a single predicate function turns that into SQL, so a dimension is one field and one line rather than two of each. Body dimensions are arrays bounded at 1000 values × 512 characters — an explicit limit rather than an emergent one — and both forms report under the SAME `usage_events.tool_name`, because splitting `memory.list` in two would have read as an outage on the day the dashboard switched.
 
+**Tags are the one dimension with TWO pills: an include and an exclude (00114).** A row carries many labels, so "includes perf" and "includes none of flaky" are independent conditions that can both hold — and the old one-pill model could not say both, because `tags` has one mode. The bar now keeps an include slot (`all`/`any`) and an exclude slot (`nin`) for `label` (`FilterFieldDescriptor.multiValued`), ANDed together; on the wire the exclude slot travels as `exclude_tags` only beside an include, and as the historical `tags_mode=none` on its own. Every SINGLE-valued dimension keeps one pill, because "is A or B" AND "is not X" simplifies exactly into one condition — `applyValueFilter` (the detail panel's filter in / filter out) does that simplification, and moves a value between polarities rather than ever producing "required AND forbidden". A retention policy carries one `tags` condition, so the Explorer withholds "Create retention policy" while a tag is both included and excluded (`hasExplorerOnlyFilters`), mirroring how it already withholds it for the reverse activity directions.
+
 **00110 refines self-exclusion for the one dimension where it was misleading: labels under AND mode.** Self-exclusion (00057) always meant "count every OTHER filter applied, but not this dimension's own" — right for a scalar dimension (kind, host, agent, trigger, repo, branch, pr, owner: there is only ever one selected value to exclude) and right for labels under `tags_mode='any'` (any ONE of several OR'd values could be swapped for another). It stopped being right for labels under `tags_mode='all'`: with `perf` already selected in AND mode, self-exclusion reported a candidate label's TOTAL count across every row carrying it, including rows that never carried `perf` at all — so picking a second label could show a count larger than the number of rows the resulting AND-of-both-labels filter would actually match, which reads as a promise the click does not keep. `lorekit_memory_facets` (migration 00110, same signature — `create or replace`, no drop) fixes this by making the label cell's `matched` flag additionally require every already-selected tag (`ok_tag`) whenever `p_tags_mode = 'all'`; under `tags_mode='any'` the added condition is a no-op and self-exclusion is bit-for-bit unchanged, and no scalar dimension's `matched` expression changed at all. The fix is WITHIN one dimension, never across dimensions — a `kind` filter still narrows the `label` counts the same way it always did (cross-group AND is untouched); only the label facet's relationship to OTHER already-selected labels changed. The migration also switches every facet's counting from "filter matching rows, then group" to "enumerate every candidate value from the full (retention/scope/tenant-filtered) population, then `count(*) filter (where matched)`" — so a value with zero matches under the current filters now reports `count: 0` and stays in the menu (the reader can see there is nothing there before clicking) rather than silently disappearing, which is what "filter then group" always did to a value whose every row got excluded. `supabase/tests/migrations.test.sql`'s own added block asserts all four: co-occurrence under AND, the zero-stays row, unchanged self-exclusion under OR, and that a cross-group (non-label) filter still narrows the label counts.
 
 ## Mobile transient selection surfaces use the BottomSheet primitive

docs/key-files.md

@@ -181,6 +181,7 @@ else. Grouped loosely by subsystem in migration/PR order.
 | `packages/mcp-core/src/memory/read-refs.ts` (+ its edge mirror) | The pure core behind batch `memory.read` (`refs`), shared by MCP's `toolRead` and REST's `handleRead`. Import-free by construction — it takes only already-PARSED `{scope,key}` pairs, never re-deriving the `scope::key` grammar `parseMemoryRefs` alone owns. `groupRefsByScope` collapses N refs into one `{scope, keys}` group per DISTINCT scope (plan D5) — the transport issues exactly one `.eq('scope', s).in('key', keys)` query per group, awaited concurrently, instead of one query per ref. `isQueryableKey` rejects a key carrying a comma/paren/quote/backslash BEFORE it reaches the query, since postgrest-js's `.in()` filter quotes a comma/paren but does not escape a quote or backslash inside the value — a bad ref becomes a `missing` entry rather than a malformed filter list. `missingRefs` reports every unresolved ref as its own caller-supplied `scope::key` string, in REQUEST order — but it only ever sees what `parseMemoryRefs` RETURNED, so a ref dropped as malformed or truncated past the 32-ref cap reaches neither `entries` nor `missing` (the gap is visible as `lorekit.refs.requested` vs `lorekit.refs.count`); matching stays against the VERBATIM stored scope, deliberately diverging from the singular `memory.read` path's `validateScope` normalisation, because the transports differ in what they hand `memory_write` and no reader can infer how a row was cased — see `docs/decisions.md#memoryread-batching-is-conditional-verbatim-scoped-and-inflates-a-signal-it-doesnt-count` |
 | `supabase/migrations/00108_retention_conditions_on_aggregates.sql` | Gives `lorekit_memory_facets` (24→31 args), `lorekit_memory_activity` (26→31) and `lorekit_memory_pivot` (27→34) the five retention thresholds `lorekit_memory_list` has had since 00092, plus a `created_since`/`created_until` pair on facets and pivot (activity's own `since`/`until` already bound `created_at`). **The bug it fixes is a DISAGREEMENT, not a missing feature**: only the list applied the thresholds, so `max_opened_count => 0` narrowed the Explorer's rows and left its facet counts, stat cards and matrix counting the un-narrowed population — every number on the page describing a different set from the one below it. Introduces `lorekit_match_retention`, ONE inlinable row test now composed by all four readers, so a fourth inline copy cannot drift (00101 records that exact drift happening once already between two copies). It takes CUTOFF INSTANTS rather than day counts, which is what keeps it `immutable` and index-usable — each caller resolves its `now()`-relative bounds once per query in a declare block — and it holds 00100's never-opened rule (`coalesce(last_opened_at, created_at)`) in one place. **Applied in the `base` CTE's WHERE, never as an `ok_*` flag**, so a threshold narrows even the SELF-EXCLUDED dimension: self-exclusion exists so the dimension you are standing in still lists what you could switch to, and switching `host` does not stop you looking at lore older than 30 days. The three aggregates are DROPPED at their exact old signatures and recreated (`create or replace` cannot change a signature, and leaving both overloads makes every PostgREST named-arg call ambiguous — 00092's footgun); `lorekit_memory_list` is a true `create or replace` at its verbatim 00105 signature, so its grants persist. Pinned by §108 |
 | `supabase/migrations/00113_explorer_reverse_activity_filters_and_opened_sort.sql` | The Explorer's REVERSE directions: `opened_within_days` and the `min_seen_count`/`min_read_count`/`min_opened_count` floors, the mirror of four of the five retention thresholds, for "which lore IS used?" rather than "what can I archive?". Widens `lorekit_match_retention` 10→14 args (the 10-arg form is dropped LAST, after every caller is recreated), appends the four to `_facets`/`_activity`/`_pivot` and, with a `p_cursor_count`, to `lorekit_memory_list`, which also accepts `p_sort => 'opened_count'` (the format() splices `$44` or `$29` as the keyset placeholder). `opened_within_days` does NOT age a never-opened lesson from `created_at` — the opposite of 00100's rule for `unseen_days`, on purpose. READ-only: no groom RPC or `GroomConditionsSchema` change. Restores the `anon` revoke on `lorekit_memory_list` that 00105's drop-and-recreate lost. Pinned by §113; §108's helper-ACL check now derives the signature |
+| `supabase/migrations/00114_exclude_tags_dimension.sql` | `p_exclude_tags text[]` appended to `lorekit_memory_list`/`_facets`/`_activity`/`_pivot` (each dropped at its exact 00113 signature first): an always-applied "carries none of these labels" conjunct, independent of `p_tags_mode`, reusing `lorekit_match_tags(…, 'none')` — so "tagged a AND NOT tagged b" is expressible. In `_facets` it is an `ok_tag_exclusion` flag the tag facet NEVER self-excludes (an excluded label stays listed at `count = 0`); `_pivot` narrows `base`; the list binds it as `$45`. The dashboard sends it only when a tag include is also active — a lone exclusion still travels as `tags_mode=none`, and retention policies never gain it. Pinned by §114 |
 | `packages/mcp-core/src/rest/count-cursor.ts` | The `sort=opened_count` keyset cursor (`{ id, opened_count }`, base64url) and its `buildCountPage`. Separate from `_shared/api/paginate.ts` because that cursor carries a timestamp and is byte-mirrored into the MCP function, which has no count sort. Mirrored verbatim into `supabase/functions/_shared/api/count-cursor.ts`, guarded by `edge-parity.spec.ts`; tested by `count-cursor.spec.ts` |
 | `supabase/functions/_shared/api/retention.ts` | 00108's fix one layer up: `retentionFrom` picks the five thresholds off a validated query or body, `retentionRpcParams` maps them to the `p_*` arguments all four RPCs share. Used by `list.ts` and the three aggregate handlers, so a route cannot be given the parameter and then forget to forward it. `?? null` and never a truthiness check — `max_opened_count => 0` is the whole point of 00105, so a falsy test turns the most useful threshold in the set into no filter at all |
 | `packages/web/src/lib/retention-filter.ts` → `retentionConditionsToAggregateBody` | The client half. Delegates to `retentionConditionsToListBody` (the field names are identical by construction — one shared `retentionConditionBodyFields` spread in the schemas) but is typed against `ListFacetsBody`, so a schema that stops carrying one of the five is a TYPE error rather than a threshold the aggregate silently ignores. Its spec reads each field back off the PARSED schema output rather than asserting `safeParse(...).success`: these schemas are not `.strict()`, so zod strips an unknown key and still reports success — a success-only check would pass against the pre-00108 routes and prove nothing |

packages/schemas/src/domain/memory.ts

@@ -259,6 +259,16 @@ export const ListMemoriesQuerySchema = z.object({
   key_prefix: z.string().min(1).max(512).optional(),
   tags: z.string().optional(),
   tags_mode: TagsModeSchema.optional().default('any'),
+  /**
+   * Labels the row must carry NONE of (comma-separated, like `tags`).
+   *
+   * An independent conjunct, ANDed with whatever `tags`/`tags_mode` ask — which
+   * is what makes "tagged `perf` AND NOT tagged `flaky`" expressible: one
+   * `tags` list has one mode, so it can include or exclude but never both.
+   * Same `not.ov` semantics as `tags_mode=none` (carries none of them), never
+   * the negation of containment. Migration 00114.
+   */
+  exclude_tags: z.string().optional(),
   /**
    * Case-insensitive substring match against `key` OR `value`.
    *
@@ -547,6 +557,8 @@ export const ListFacetsQuerySchema = z.object({
   scope: RawScopeSchema.optional(),
   tags: z.string().optional(),
   tags_mode: TagsModeSchema.optional().default('any'),
+  /** Labels the row must carry NONE of — see `ListMemoriesQuerySchema.exclude_tags`. */
+  exclude_tags: z.string().optional(),
   source_agent: ValueListSchema.optional(),
   source_agent_mode: ScalarFilterModeSchema.optional().default('in'),
   trigger: ValueListSchema.optional(),
@@ -638,6 +650,8 @@ export const PivotQuerySchema = z.object({
   scope: RawScopeSchema.optional(),
   tags: z.string().optional(),
   tags_mode: TagsModeSchema.optional().default('any'),
+  /** Labels the row must carry NONE of — see `ListMemoriesQuerySchema.exclude_tags`. */
+  exclude_tags: z.string().optional(),
   source_agent: ValueListSchema.optional(),
   source_agent_mode: ScalarFilterModeSchema.optional().default('in'),
   trigger: ValueListSchema.optional(),
@@ -722,6 +736,8 @@ export const ActivityQuerySchema = z.object({
   scope: RawScopeSchema.optional(),
   tags: z.string().optional(),
   tags_mode: TagsModeSchema.optional().default('any'),
+  /** Labels the row must carry NONE of — see `ListMemoriesQuerySchema.exclude_tags`. */
+  exclude_tags: z.string().optional(),
   source_agent: ValueListSchema.optional(),
   source_agent_mode: ScalarFilterModeSchema.optional().default('in'),
   trigger: ValueListSchema.optional(),
@@ -1065,6 +1081,8 @@ const DimensionValuesSchema = z
 const dimensionBodyFields = {
   tags: DimensionValuesSchema.optional(),
   tags_mode: TagsModeSchema.optional().default('any'),
+  /** Labels the row must carry NONE of — see `ListMemoriesQuerySchema.exclude_tags`. */
+  exclude_tags: DimensionValuesSchema.optional(),
   source_agent: DimensionValuesSchema.optional(),
   source_agent_mode: ScalarFilterModeSchema.optional().default('in'),
   trigger: DimensionValuesSchema.optional(),