Skip to content

Containment recall reads indexes instead of scanning the base - #12

Merged
gedaiu merged 1 commit into
re-devfrom
perf/containment-trgm
Oct 19, 2026
Merged

gedaiu merged 1 commit into
re-devfrom
perf/containment-trgm

Conversation

@gedaiu

@gedaiu gedaiu commented Oct 19, 2026

Copy link
Copy Markdown

Depends on #10 (migration 9001 facts_live_predicate_idx); this branch is built on its head, so merge #10 first, then this shows only its own commit.

Lore's memory mirror pushes prose to api sources, and every new name in it runs containment_reviews. On prod (re-cinq/lore base: 39k entities, 39k alias facts, 960k facts) that took 1.1 s per call, 833 ms of it JIT; 25 at once pinned the DB at 4 CPU.

  • SET LOCAL jit = off for the query.
  • Alias facts are found through the predicate looked up as a scalar subquery, so 9001's index applies.
  • 9002: pg_trgm + a GIN trigram index on live entity names for the forward LIKE '%mention%' branch.
  • The reverse branch (mention LIKE '%name%') matches the mention's own substrings (≥ MIN_CONTAIN_CHARS) by equality on existing btree indexes: same semantics, no escaping needed.

Seeded base of production shape, jit off: 72 ms → 15 ms for a name with no hits, 73 → 60 ms for one with 3,900. A trigram index on all facts was faster for rare names but slower for common ones, 55 MB and on the write path, so it is left out. New tests: contained_names units, and the Mr. Holmes ↔ Sherlock Holmes alias pairing. fmt, clippy and cargo test --workspace green. Schema version 104 → 105; renumber if #10 changes.

Fork only for now; upstream deeplethe#1104 / deeplethe#1322 rewrite this predicate, noted in the code for the next sync.

…kup, names by a trigram index, and the reverse branch by the mention's own substrings.

Signed-off-by: Bogdan Szabo <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant