Repository navigation
Utopia: the default branch's commit history, under ontology v4 - #2790
Conversation
Lore Spec ImpactLore API unreachable (HTTP 413). This check could not run. |
|
Lore is reviewing this PR — 2b3b83d8-6b3e-45b4-b782-3248a3b83272. Submit a review that requests changes to have Lore push a fix for it. Comment |
There was a problem hiding this comment.
Lore review — Changes suggested
Backfill sweep/script ignore commit-walk failures under summary.commits.failed, overlapping commit walks can move commits_head backwards, and re-running setup wipes commits_head.
Must fix (2) · Consider (3) · Nits (0)
Reviewed by gemini-4-argon
Submit a review that requests changes to have Lore push a fix for it. Comment @lore review to re-run the review.
Notes on lines outside changed hunks
apps/lore-api/src/work/utopia/utopia-backfill.ts:30 — issue (blocking): Surface commit walk failures to backfill callers (also in apps/stations/src/work/utopia-backfill-tick/utopia-backfill-tick.ts:69 and scripts/infra/utopia-backfill.sh:41)
const commits = await mirrorBranchCommits(deps.history, branch);
return {
...documents,
failed: [...documents.failed, ...commits.failed],
commits,
};
apps/lore-api/src/outbound/utopia-sources.ts:59 — issue (blocking): Merge into settings.utopia instead of replacing it so setup and ontology upgrades do not wipe commits_head
await pool.query(
`UPDATE lore.repos SET settings = jsonb_set(COALESCE(settings, '{}'), '{utopia}', COALESCE(settings->'utopia', '{}') || ($1::jsonb)->'utopia') WHERE full_name = $2`,
[JSON.stringify({ utopia }), repo],
);
Posted by floor, visit 8da9fc28-4814-4cc5-9c39-ed66a75943c9.
|
Both blocking notes are taken in the pushed fix: the backfill now adds the walk's failure to its failed list (the sweep and the script count it, FR11.8), and saveUtopiaMirrorSettings merges into the utopia block with jsonb_set so a setup or an ontology upgrade keeps commits_head (integration test on the real table). The three suggestions are taken too, answered on their threads. |
There was a problem hiding this comment.
Lore review — Approved
All prior review findings are cleanly resolved with matching spec statements, ADR updates, and tests.
Must fix (0) · Consider (0) · Nits (0)
Reviewed by gemini-4-argon
Submit a review that requests changes to have Lore push a fix for it. Comment @lore review to re-run the review.
Posted by floor, visit 97cc1270-cc05-4b8f-b3f6-a8a3f1ae948d.
…elation marked with its temporal Four classes (Commit, PullRequest, Issue, Task), five event relations (changes, follows, merges, closes, delivers) and the attributes they carry join the turtle; the structural relations are marked eternal. Utopia's importer binds a pushed phrase by the declaration only when the relation is not a state, so every object property now says which it is. A base on version 3 is upgraded on its next mirrored push, as before.
…ect ends with linkedIssueNumbers lists Closes, Fixes, Resolves and Refs numbers once each in order, for the commit mirror that links a pull request to every issue it names; mergedPullRequestNumber reads the (#NNNN) a squash-merged commit's subject carries.
commitItems turns one default-branch commit into a commit:<sha> item at its author date: Repository contains Commit, its sha, subject, body, author, time and permalink, the parents it follows, the pull request its squash subject names with the issues and the Lore task that pull request's body names, and each changed path as the Spec, Adr or SourceFile the other mirrors name. The text cut moves to fragments.ts so the doc builder and this one share it; a commit, a pull request and an issue are named by their GitHub page path.
…ad in detail listBranchHistory walks a branch's commits newest-first until the cursor sha, then reads the oldest limit of them with repos.getCommit and answers them oldest-first: subject, body, author, time, parents and each changed path with its status. A run that cannot finish the history leaves a cursor the next one continues from. It replaces listCommitsSince, which nothing called.
mirrorBranchCommits walks the branch's history above settings.utopia.commits_head, pushes each commit oldest-first through the mirror an ingest uses, with its paths classed as the ingest classes them, and moves the cursor past each commit that landed whole; the first that did not stops the walk, so the next one retries it. The backfill runs the walk after the documents and answers it under commits, so the Sunday sweep covers commits as it is.
…ault branch The route runs the commit walk on its own, for a push that just landed on the default branch; a body naming any other branch is answered skipped with no GitHub read, since Utopia holds the default branch's history alone. The guards are the backfill route's: 503 without a Utopia, 404 for a repo Lore does not hold, 409 for one with no source.
…into Utopia github.push joins the repository events the stations claim; the handler posts lore-api's commits route for the pushed branch and logs what the walk did, with a repo that has no Utopia source and a deployment with no Utopia read as skips. The lore-api client gains the commits post beside the backfill one, sharing its request, and the target it reads from the environment.
The mirror spec gains the Commits section and ADR-052 the amendment that records the name shapes, the by-name merge that makes CI ordering irrelevant, the cursor, and the correction: Utopia's importer gave every imported relation temporal=state, so the rule of #1242 bound no Lore phrase until deeplethe/utopia#1291 reads the annotation. The history read moves to platform-github-history.ts, the docs classifier to ingested-docs.ts (breaking the backfill/commits cycle), the three Utopia routes to utopia-routes.ts; the OpenAPI document and the web-ui types carry the new route.
A walk's failure reaches the backfill's failed list, so the Sunday sweep and the script count it; the settings writer merges into the utopia block, so a setup or an ontology upgrade keeps the commit cursor; the walks of one repo run one after the other in a process, so a push during a walk cannot move the cursor back; a split commit carries its body on the first part only, as a split statement does; and a pull request refers_to an issue it only names with Refs, closing only what it claims to resolve.
ce9a829 to
0420ddc
Compare
There was a problem hiding this comment.
Lore review — Approved
All five prior review points and the FR12 renumbering after rebase are cleanly resolved.
Must fix (0) · Consider (0) · Nits (0)
Reviewed by gemini-4-argon
Submit a review that requests changes to have Lore push a fix for it. Comment @lore review to re-run the review.
Posted by floor, visit 25ace9f0-659a-4397-a249-d8a75746f31b.
…e Utopia mirror (#2794) The first real Utopia arm for the Lore-vs-Utopia experiment (#2787, #2792): under backend `utopia`, and in the Utopia half of `both`, the context's **Conventions** (`repo`) and **Architecture Decisions** (`adrs`) sections come from Utopia instead of answering "no Utopia reader yet". It needs no new writes: it reads the spec and ADR statements the mirror (ADR-052) already pushed, through the read client from #2791. ## What changed - **One shared section pipeline.** `context-assembly-pipeline.ts` (moved out of `context-assembly.ts`): budget, fetching per section through a reader, packing, the debug trace. Lore and Utopia both go through it, so an eval's `found` grades Utopia's assembly the same way as Lore's. The Lore path is unchanged; the existing context-assembly tests pass without edits. - **`context-assembly-utopia.ts`.** - One search per assembly (`topK` 50), a second page only when a section is short and the first page says `hasMore`. - Keeps statements-source hits whose pushed id is a `doc:` or `stmt:` item (other sources are dropped, since search has no source filter yet), sorts each path into `repo` or `adrs` with Lore's `classifyFile`, and groups by path in first-rank order. - Each item is the document's title then its statement texts in order, under the Lore path, with Utopia's score; same item limits as Lore. - Every other section still answers "no Utopia reader yet", as all do when the deployment has no reader or the repo has no `settings.utopia`. - **Wiring.** `repoContextSettings` also reads `settings.utopia` {`kb_id`, `source_id`} in the same query. `/api/context` and the evals pass `utopiaReadsFrom(env)` and those ids. Under `both`, the two halves now assemble in parallel; the debug trace stays Lore's. Caveats, recorded in the ADR-052 amendment: a tombstoned document can still match until Utopia's search can exclude missing documents; post-filtering by source id until search takes a source filter; code items carry no text yet; typed graph reads wait for the release that types Lore's relations. Spec: `specs/utopia-graph-mirror` "The Utopia arm" FR12.1–FR12.6, `specs/context-assembly` FR-1.5. (FR12 may need renumbering if #2790 lands first.) ## Tests Pure and reader tests (libs/shared): one item per spec from three statement chunks in order; another source's hits dropped; a second page only when short; code has no reader. End-to-end with the real templates (server-core): the ADR under Architecture Decisions with its statement texts; the trace lists the kept paths; no settings falls back to no-reader sections. Fixtures are built from the real `docStatementItems` output with a hand-written fake `UtopiaReads`. `repoContextSettings` reads the mirror ids. `tsc` clean in libs/shared, server-core, lore-api; vitest libs/shared knowledge + utopia 143 passed (2 skipped), server-core context 27, lore-api context + context-evals + utopia 95; eslint clean on changed files and `specs adrs`. Generated with [Claude Code](https://claude.com/claude-code)
…2804) Utopia's database has been pinned at its 4-CPU limit while Lore pushes. For every named entity in a pushed item, Utopia ran a nearest-neighbour search over the whole knowledge base's name vectors, even after an exact name match had already found the entity. That search only proposes "possible duplicate" pairs, and for Lore's machine-made names (`/pull/2790`) those pairs are always false. 0.1.0-re.9 is re.8 plus one change: a name in a pushed statements chunk is resolved by its spelling, and extraction computes no name vector for it, so the search never runs for pushes. Pushed names still get their vectors stored afterwards, so documents that aren't pushes can still find them. Upstream discussion: deeplethe/utopia#1293 and #1296. The branch is `gedaiu/utopia` `perf/pushed-skip-name-vectors`. The merged re.9 tree passes fmt, clippy and 1,616 tests. Tag `v0.1.0-re.9` (re-dev 78a652ab), image `ghcr.io/re-cinq/utopia:0.1.0-re.9`, digest `sha256:ac5fc71b530f02a47aa63efb14a7264df55140627742595ed7cb06a5b1868463`. No new migration. After merge: `terraform apply -target='kubernetes_deployment_v1.utopia[0]'` from main. Generated with [Claude Code](https://claude.com/claude-code)
Every default-branch commit of a mirrored repo now lands in its Utopia knowledge base as one
commit:<sha>item at its author date: Repository contains Commit, the commit's sha, subject, body, author, time and permalink, the parents it follows, the pull request its squash subject names with the issues and the Lore task that pull request's body names, and each path it changed, named exactly as the doc, test-report and code mirrors name it. A commit item never says which tests or statements it affected: CI pushes those after the merge, Utopia merges same-named entities whichever item came first, and the walk from a commit through its file to the test is always as current as the last ingest.lore-api walks the branch above
settings.utopia.commits_head, oldest first, 2,000 commits a walk, moving the cursor past each commit that landed whole and stopping at the first that did not, so the whole history arrives over as many walks as it takes. The weekly backfill runs the walk after the documents; the stations claimgithub.pushand post the newPOST /api/repos/{owner}/{repo}/utopia/commitson every push, which skips any branch but the default one. The GitHub port's never-calledlistCommitsSincebecomeslistBranchHistory.Ontology version 4 declares
Commit,PullRequest,IssueandTask, the relationschanges,follows,merges,closesanddelivers, and marks every object property withutopia:temporal: the commit relations are events at the item'sdoc_time, the structural ones eternal.Blocked on Utopia re.8. Utopia binds a pushed phrase by the declaration alone (deeplethe/utopia#1242) only when the relation is not a state, and its OWL importer gave every imported relation
statewithout reading theutopia:temporalannotation the export writes, so no Lore relation in any live base has been ontology-decided so far. deeplethe/utopia#1291 teaches the importer to read it on create and on re-import; once it ships as re.8 and is applied, the version-4 upgrade on the next mirrored push flips every live base's relations. Merging this before that leaves the relations atstateas today, so this stays a draft until re.8 is live.Specs:
specs/utopia-graph-mirrorFR11 and the version rows; ADR-052 gains the amendment, including the correction above.🤖 Generated with Claude Code