Repository navigation
Utopia reads: search a knowledge base and read a pushed item back by its external id - #2791
Conversation
…its external id
utopiaReadsFrom reads a repository's knowledge base as the Lore person
(UTOPIA_TOKEN): POST /kbs/{kb}/search with q, top_k and offset, and
GET .../documents/by-external-id, a 404 reading as null. The pure
read-shapes module parses the scored search page, the stored document
and a statements chunk, and pushedPathOf turns a pushed external id
back into its kind, path and statement ordinal. Fixtures are copied
from Utopia v0.1.0-re.7. UtopiaPushError becomes UtopiaError, since
reads throw it too; a 422 such as page_too_deep is not retried.
Spec: utopia-graph-mirror FR11.1-11.3.
|
Lore is reviewing this PR — 37804ac8-790b-418d-bd71-5058de29f838. Submit a review that requests changes to have Lore push a fix for it. Comment |
🔍 Lore Spec Impact — advisoryThis PR touches 12 statement(s) across 1 spec(s), and 5 of them have validating tests this PR does not change. Feature Specification: The spec-to-code chain mirrored in Utopia · 6 statement(s)
The push
validated by The ingest only its test links changed — the statement text itself is unchanged
validated by The ingest only its test links changed — the statement text itself is unchanged
validated by Test runs only its test links changed — the statement text itself is unchanged
validated by Test runs only its test links changed — the statement text itself is unchanged
validated by Test runs only its test links changed — the statement text itself is unchanged
validated by Weaker signals (6) — linked by a spec, not proven by a test runFeature Specification: The spec-to-code chain mirrored in Utopia · 6 statement(s)
The push
validated by The push
validated by The ingest
validated by Setup
validated by Setup
validated by Setup on first ingest
validated by 2 changed statement(s) had no validating test, so no coverage broke. 22 new statement(s) have no test link yet. Deterministic · graph @ |
![lore-agent[bot]](https://avatars.githubusercontent.com/u/141229953?s=60&v=4){kind=small}
There was a problem hiding this comment.
Lore review — Approved
Clean, well-tested Utopia read client and response parsers matching FR11 and ADR-052.
Must fix (0) · Consider (0) · Nits (0)
Reviewed by gemini-4-argon
Submit a review that requests changes to have Lore push a fix for it. Comment @lore review to re-run the review.
Posted by floor, visit d66e1b6e-a269-4124-ac1b-c5410ace1b2a.
…e Utopia mirror (#2794) The first real Utopia arm for the Lore-vs-Utopia experiment (#2787, #2792): under backend `utopia`, and in the Utopia half of `both`, the context's **Conventions** (`repo`) and **Architecture Decisions** (`adrs`) sections come from Utopia instead of answering "no Utopia reader yet". It needs no new writes: it reads the spec and ADR statements the mirror (ADR-052) already pushed, through the read client from #2791. ## What changed - **One shared section pipeline.** `context-assembly-pipeline.ts` (moved out of `context-assembly.ts`): budget, fetching per section through a reader, packing, the debug trace. Lore and Utopia both go through it, so an eval's `found` grades Utopia's assembly the same way as Lore's. The Lore path is unchanged; the existing context-assembly tests pass without edits. - **`context-assembly-utopia.ts`.** - One search per assembly (`topK` 50), a second page only when a section is short and the first page says `hasMore`. - Keeps statements-source hits whose pushed id is a `doc:` or `stmt:` item (other sources are dropped, since search has no source filter yet), sorts each path into `repo` or `adrs` with Lore's `classifyFile`, and groups by path in first-rank order. - Each item is the document's title then its statement texts in order, under the Lore path, with Utopia's score; same item limits as Lore. - Every other section still answers "no Utopia reader yet", as all do when the deployment has no reader or the repo has no `settings.utopia`. - **Wiring.** `repoContextSettings` also reads `settings.utopia` {`kb_id`, `source_id`} in the same query. `/api/context` and the evals pass `utopiaReadsFrom(env)` and those ids. Under `both`, the two halves now assemble in parallel; the debug trace stays Lore's. Caveats, recorded in the ADR-052 amendment: a tombstoned document can still match until Utopia's search can exclude missing documents; post-filtering by source id until search takes a source filter; code items carry no text yet; typed graph reads wait for the release that types Lore's relations. Spec: `specs/utopia-graph-mirror` "The Utopia arm" FR12.1–FR12.6, `specs/context-assembly` FR-1.5. (FR12 may need renumbering if #2790 lands first.) ## Tests Pure and reader tests (libs/shared): one item per spec from three statement chunks in order; another source's hits dropped; a second page only when short; code has no reader. End-to-end with the real templates (server-core): the ADR under Architecture Decisions with its statement texts; the trace lists the kept paths; no settings falls back to no-reader sections. Fixtures are built from the real `docStatementItems` output with a hand-written fake `UtopiaReads`. `repoContextSettings` reads the mirror ids. `tsc` clean in libs/shared, server-core, lore-api; vitest libs/shared knowledge + utopia 143 passed (2 skipped), server-core context 27, lore-api context + context-evals + utopia 95; eslint clean on changed files and `specs adrs`. Generated with [Claude Code](https://claude.com/claude-code)
The first read path from Lore into Utopia, for the side-by-side experiment (#2787): a small client for the two reads the first Utopia context arm needs, against the routes now live in Utopia
0.1.0-re.7(#2789).What changed
outbound/utopia/utopia-reads.ts:utopiaReadsFrom(env)(null whenUTOPIA_API_URLorUTOPIA_TOKENis unset) reads a repo's knowledge base as the Lore person.search(kbId, q, {topK, offset})→POST /kbs/{kb}/search, scored hits with source andexternal_id.documentByExternalId(kbId, sourceId, externalId)→GET …/documents/by-external-id; a 404 isnull.page_too_deep,empty_query) fails at once.domain/utopia/read-shapes.ts(pure, Zod):parseSearchPage,parseStoredDocument,parseStatementsChunk(null for a chunk Utopia cut mid-item),statementTexts, andpushedPathOf, which turns a pushed id such asstmt:specs/…/spec.md#0003.2back into kind, path and statement ordinal.ScoredChunk,by_external_id, a pushed statements item).utopia-client.ts:utopiaRequesterbuilds the/api/v1base and retry for both clients;UtopiaPushErrorbecomesUtopiaError, since reads throw it too.Nothing calls the client yet; the Utopia arm for the
repoandadrscontext sections comes next.specs/utopia-graph-mirror: new "Reads" section FR11.1–FR11.3; FR3.3 namesUtopiaError.Tests
read-shapes.test.tsandutopia-reads.test.ts(the HTTP side against a localnode:httpserver answering the fixtures): the search page with null ranks kept,pushedPathOfforstmt/doc/unknown ids, statement texts from a pretty-printed chunk and a cut chunk, the stored document, the request shape and PAT, 404 → null, 422 not retried, unset env → null.tscclean in libs/shared; vitestsrc/domain/utopia+src/outbound/utopia51 passed; eslint clean on changed files andspecs adrs.Generated with Claude Code