Repository navigation
Docs: Lore and Utopia compared as the store an agent's context comes from - #2797
Merged
Merged
Conversation
…re context is assembled from
Contributor
|
Lore is reviewing this PR — 98e9bfd3-d3af-4090-aedd-cdcf8dc002d0. Submit a review that requests changes to have Lore push a fix for it. Comment |
🔍 Lore Spec ImpactExamined 1 changed file(s): 0 had graph data (no coupling found), 1 had none — no ingested test run covers them, so this check cannot speak for them. Deterministic · graph @ |
![lore-agent[bot]](https://avatars.githubusercontent.com/u/141229953?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Lore review — Approved
Clear record of the first side-by-side context-evals comparison; two small factual corrections on how useful_share, the Utopia reader, and the backfill behave in code.
Must fix (0) · Consider (1) · Nits (2)
Reviewed by gemini-4-argon
Submit a review that requests changes to have Lore push a fix for it. Comment @lore review to re-run the review.
Posted by floor, visit 861998b4-7395-454d-acfd-35f5146c936e.
|
|
||
| - **Found**: the assembled context included the document the question was written from. | ||
| - **Answered**: a model, given only the assembled context, answered the question, judged against the document. | ||
| - **Useful**: the share of the assembled context the judge found relevant to the question. |
Contributor
There was a problem hiding this comment.
nit: State that the answering model (not the judge) selects used_sources for Useful
Suggested change
| - **Useful**: the share of the assembled context the judge found relevant to the question. | |
| - **Useful**: the share of returned context tokens in the sources the answering model named as used. |
|
|
||
| ## What we read in it | ||
|
|
||
| - **Where Utopia has the data, it already finds the right document.** On bowman-ui it found more source documents than Lore (18 against 15) and answered as many questions (19). |
Contributor
There was a problem hiding this comment.
suggestion: Correct what utopiaDocuments returns: it joins only matched hit chunks (not every statement in the doc) and omits similarity so RELEVANCE_MARGIN never trims unrelated neighbours
Suggested change
| - **Where Utopia has the data, it already finds the right document.** On bowman-ui it found more source documents than Lore (18 against 15) and answered as many questions (19). | |
| - **Less of what Utopia returns gets used.** Its useful share is roughly half of Lore's wherever it answers at all. On CFOAgent it answered 11 questions while finding only 2 of their documents, so its answers came from neighbouring statements rather than the source. Useful share is a precision measure: the tokens of the sources the answer used, over all tokens returned. Two differences in the Utopia reader explain the gap: `utopiaDocuments` already keeps only the `doc:` and `stmt:` chunks returned in the top-50 search page (`libs/shared/src/outbound/project/knowledge/context-assembly-utopia.ts`), not the whole document, and its items carry no `similarity`, so `closeToTheBest` (`RELEVANCE_MARGIN`) never drops unrelated neighbour documents the way it does for Lore's vector hits. |
| - **Less of what Utopia returns gets used.** Its useful share is roughly half of Lore's wherever it answers at all. On CFOAgent it answered 11 questions while finding only 2 of their documents, so its answers came from neighbouring statements rather than the source. Useful share is a precision measure: the tokens of the sources the answer used, over all tokens returned. The likely cause is granularity: the Utopia reader returns one item per document, its title followed by *every* statement of the document, so a question about one rule carries the whole spec with it. Lore returns heading-sized chunks ranked one by one, so it hands over the matching sections. This is not verified yet: the eval does not record the size of each source, and the test is to have the Utopia reader keep only the statements its search hits matched, then compare. | ||
| - **Utopia almost never answers what Lore cannot.** Across the run Utopia answered one question Lore missed (on Otto); Lore answered 56 that Utopia missed. | ||
| - **The two stores rarely pick the same sources.** Overlap stays between 0% and 12%. Even on bowman-ui, where both find most documents, they keep different neighbours, which suggests the ranking differs a lot (Utopia ranks statements by its hybrid search; Lore ranks chunks by its own). | ||
| - **HALEngine is a real gap, not a missing base.** It has a Utopia base, but Utopia found none of its 20 documents while Lore found and answered all 20. Why is not known yet; the first thing to check is whether its documents reached the mirror whole (the base is on ontology version 2). |
Contributor
There was a problem hiding this comment.
nit: Clarify why the five unmirrored repos have no Utopia base (backfill auto-creates bases when UTOPIA_TOKEN and UTOPIA_WORKSPACE_ID are set, per FR6.1/FR7.2)
Suggested change
| - **HALEngine is a real gap, not a missing base.** It has a Utopia base, but Utopia found none of its 20 documents while Lore found and answered all 20. Why is not known yet; the first thing to check is whether its documents reached the mirror whole (the base is on ontology version 2). | |
| - **Five repositories have no Utopia base at all**, so their Utopia columns are zeros by construction; auto-setup (`UTOPIA_TOKEN` and `UTOPIA_WORKSPACE_ID`) was not enabled when they last ingested or when the backfill ran. |
gedaiu
added a commit
that referenced
this pull request
Oct 17, 2026
Corrects `docs/building-lore/context-store-comparison.md` (#2797). It gave "granularity" as the likely cause of Utopia's low useful share: the reader returning every statement of a document. A check of the reader (`context-assembly-utopia.ts`) and the mirror (`doc-statements.ts`) shows that is wrong: each `stmt:` search hit carries exactly one statement, a doc head contributes only its title, and an item holds only the statements the search matched. The doc now says the cause is not known, names the likelier suspects (ranking and recall across one 50-hit search; weak single-statement matches taking a document slot), and replaces next step 2 with measuring source sizes in the eval, then trying a per-document floor or cap. Docs only. Generated with [Claude Code](https://claude.com/claude-code)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Records the first side-by-side measurement of Lore's own stores against Utopia as the source of an agent's context, in the shape of
decompose-model-comparison.md.The comparison ran locally (the eval runner on a developer machine, against the deployed lore-api at #2794 and Utopia
0.1.0-re.7), with the side-by-side grading from #2792: one question per document, context assembled once from each store at the full budget, answered and judged the same way.Headline: no store should switch yet.
The doc lists the next steps in order: set up the missing bases, return only matched statements, push whole files, then run again.
Docs only.
Generated with Claude Code