Skip to content

Utopia: the default branch's commit history, under ontology v4 - #2790

Merged
gedaiu merged 11 commits into
mainfrom
utopia-commit-history
Oct 16, 2026
Merged

gedaiu merged 11 commits into
mainfrom
utopia-commit-history

Conversation

@gedaiu

@gedaiu gedaiu commented Oct 16, 2026

Copy link
Copy Markdown
Collaborator

Every default-branch commit of a mirrored repo now lands in its Utopia knowledge base as one commit:<sha> item at its author date: Repository contains Commit, the commit's sha, subject, body, author, time and permalink, the parents it follows, the pull request its squash subject names with the issues and the Lore task that pull request's body names, and each path it changed, named exactly as the doc, test-report and code mirrors name it. A commit item never says which tests or statements it affected: CI pushes those after the merge, Utopia merges same-named entities whichever item came first, and the walk from a commit through its file to the test is always as current as the last ingest.

lore-api walks the branch above settings.utopia.commits_head, oldest first, 2,000 commits a walk, moving the cursor past each commit that landed whole and stopping at the first that did not, so the whole history arrives over as many walks as it takes. The weekly backfill runs the walk after the documents; the stations claim github.push and post the new POST /api/repos/{owner}/{repo}/utopia/commits on every push, which skips any branch but the default one. The GitHub port's never-called listCommitsSince becomes listBranchHistory.

Ontology version 4 declares Commit, PullRequest, Issue and Task, the relations changes, follows, merges, closes and delivers, and marks every object property with utopia:temporal: the commit relations are events at the item's doc_time, the structural ones eternal.

Blocked on Utopia re.8. Utopia binds a pushed phrase by the declaration alone (deeplethe/utopia#1242) only when the relation is not a state, and its OWL importer gave every imported relation state without reading the utopia:temporal annotation the export writes, so no Lore relation in any live base has been ontology-decided so far. deeplethe/utopia#1291 teaches the importer to read it on create and on re-import; once it ships as re.8 and is applied, the version-4 upgrade on the next mirrored push flips every live base's relations. Merging this before that leaves the relations at state as today, so this stays a draft until re.8 is live.

Specs: specs/utopia-graph-mirror FR11 and the version rows; ADR-052 gains the amendment, including the correction above.

🤖 Generated with Claude Code

@github-actions

Copy link
Copy Markdown

Lore Spec Impact

Lore API unreachable (HTTP 413). This check could not run.

@gedaiu
gedaiu marked this pull request as ready for review October 16, 2026 19:31
@lore-agent

lore-agent Bot commented Oct 16, 2026

Copy link
Copy Markdown
Contributor

Lore is reviewing this PR — 2b3b83d8-6b3e-45b4-b782-3248a3b83272.

Submit a review that requests changes to have Lore push a fix for it. Comment @lore review to re-run the review.

@lore-agent lore-agent Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lore review — Changes suggested

Backfill sweep/script ignore commit-walk failures under summary.commits.failed, overlapping commit walks can move commits_head backwards, and re-running setup wipes commits_head.

Must fix (2) · Consider (3) · Nits (0)

Reviewed by gemini-4-argon

Submit a review that requests changes to have Lore push a fix for it. Comment @lore review to re-run the review.

Notes on lines outside changed hunks

apps/lore-api/src/work/utopia/utopia-backfill.ts:30 — issue (blocking): Surface commit walk failures to backfill callers (also in apps/stations/src/work/utopia-backfill-tick/utopia-backfill-tick.ts:69 and scripts/infra/utopia-backfill.sh:41)

  const commits = await mirrorBranchCommits(deps.history, branch);

  return {
    ...documents,
    failed: [...documents.failed, ...commits.failed],
    commits,
  };

apps/lore-api/src/outbound/utopia-sources.ts:59 — issue (blocking): Merge into settings.utopia instead of replacing it so setup and ontology upgrades do not wipe commits_head

  await pool.query(
    `UPDATE lore.repos SET settings = jsonb_set(COALESCE(settings, '{}'), '{utopia}', COALESCE(settings->'utopia', '{}') || ($1::jsonb)->'utopia') WHERE full_name = $2`,
    [JSON.stringify({ utopia }), repo],
  );

Posted by floor, visit 8da9fc28-4814-4cc5-9c39-ed66a75943c9.

Comment thread apps/lore-api/src/transport/routes/repos/utopia-commits.ts
Comment thread libs/shared/src/domain/utopia/commit-statements.ts
Comment thread specs/utopia-graph-mirror/spec.md Outdated
@gedaiu

gedaiu commented Oct 16, 2026

Copy link
Copy Markdown
Collaborator Author

Both blocking notes are taken in the pushed fix: the backfill now adds the walk's failure to its failed list (the sweep and the script count it, FR11.8), and saveUtopiaMirrorSettings merges into the utopia block with jsonb_set so a setup or an ontology upgrade keeps commits_head (integration test on the real table). The three suggestions are taken too, answered on their threads.

@lore-agent lore-agent Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lore review — Approved

All prior review findings are cleanly resolved with matching spec statements, ADR updates, and tests.

Must fix (0) · Consider (0) · Nits (0)

Reviewed by gemini-4-argon

Submit a review that requests changes to have Lore push a fix for it. Comment @lore review to re-run the review.

Posted by floor, visit 97cc1270-cc05-4b8f-b3f6-a8a3f1ae948d.

gedaiu and others added 11 commits October 16, 2026 22:56
…elation marked with its temporal

Four classes (Commit, PullRequest, Issue, Task), five event relations (changes,
follows, merges, closes, delivers) and the attributes they carry join the
turtle; the structural relations are marked eternal. Utopia's importer binds
a pushed phrase by the declaration only when the relation is not a state, so
every object property now says which it is. A base on version 3 is upgraded
on its next mirrored push, as before.
…ect ends with

linkedIssueNumbers lists Closes, Fixes, Resolves and Refs numbers once each in
order, for the commit mirror that links a pull request to every issue it
names; mergedPullRequestNumber reads the (#NNNN) a squash-merged commit's
subject carries.
commitItems turns one default-branch commit into a commit:<sha> item at its
author date: Repository contains Commit, its sha, subject, body, author, time
and permalink, the parents it follows, the pull request its squash subject
names with the issues and the Lore task that pull request's body names, and
each changed path as the Spec, Adr or SourceFile the other mirrors name. The
text cut moves to fragments.ts so the doc builder and this one share it; a
commit, a pull request and an issue are named by their GitHub page path.
…ad in detail

listBranchHistory walks a branch's commits newest-first until the cursor sha,
then reads the oldest limit of them with repos.getCommit and answers them
oldest-first: subject, body, author, time, parents and each changed path with
its status. A run that cannot finish the history leaves a cursor the next one
continues from. It replaces listCommitsSince, which nothing called.
mirrorBranchCommits walks the branch's history above settings.utopia.commits_head,
pushes each commit oldest-first through the mirror an ingest uses, with its
paths classed as the ingest classes them, and moves the cursor past each
commit that landed whole; the first that did not stops the walk, so the next
one retries it. The backfill runs the walk after the documents and answers it
under commits, so the Sunday sweep covers commits as it is.
…ault branch

The route runs the commit walk on its own, for a push that just landed on
the default branch; a body naming any other branch is answered skipped with
no GitHub read, since Utopia holds the default branch's history alone. The
guards are the backfill route's: 503 without a Utopia, 404 for a repo Lore
does not hold, 409 for one with no source.
…into Utopia

github.push joins the repository events the stations claim; the handler posts
lore-api's commits route for the pushed branch and logs what the walk did,
with a repo that has no Utopia source and a deployment with no Utopia read as
skips. The lore-api client gains the commits post beside the backfill one,
sharing its request, and the target it reads from the environment.
The mirror spec gains the Commits section and ADR-052 the amendment that
records the name shapes, the by-name merge that makes CI ordering
irrelevant, the cursor, and the correction: Utopia's importer gave every
imported relation temporal=state, so the rule of #1242 bound no Lore phrase
until deeplethe/utopia#1291 reads the annotation. The history read moves to
platform-github-history.ts, the docs classifier to ingested-docs.ts (breaking
the backfill/commits cycle), the three Utopia routes to utopia-routes.ts; the
OpenAPI document and the web-ui types carry the new route.
A walk's failure reaches the backfill's failed list, so the Sunday sweep and
the script count it; the settings writer merges into the utopia block, so a
setup or an ontology upgrade keeps the commit cursor; the walks of one repo
run one after the other in a process, so a push during a walk cannot move
the cursor back; a split commit carries its body on the first part only, as
a split statement does; and a pull request refers_to an issue it only names
with Refs, closing only what it claims to resolve.
@gedaiu
gedaiu force-pushed the utopia-commit-history branch from ce9a829 to 0420ddc Compare October 16, 2026 20:59

@lore-agent lore-agent Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lore review — Approved

All five prior review points and the FR12 renumbering after rebase are cleanly resolved.

Must fix (0) · Consider (0) · Nits (0)

Reviewed by gemini-4-argon

Submit a review that requests changes to have Lore push a fix for it. Comment @lore review to re-run the review.

Posted by floor, visit 25ace9f0-659a-4397-a249-d8a75746f31b.

@gedaiu
gedaiu merged commit 7ba7120 into main Oct 16, 2026
34 checks passed
gedaiu added a commit that referenced this pull request Oct 16, 2026
…e Utopia mirror (#2794)

The first real Utopia arm for the Lore-vs-Utopia experiment (#2787,
#2792): under backend `utopia`, and in the Utopia half of `both`, the
context's **Conventions** (`repo`) and **Architecture Decisions**
(`adrs`) sections come from Utopia instead of answering "no Utopia
reader yet". It needs no new writes: it reads the spec and ADR
statements the mirror (ADR-052) already pushed, through the read client
from #2791.

## What changed

- **One shared section pipeline.** `context-assembly-pipeline.ts` (moved
out of `context-assembly.ts`): budget, fetching per section through a
reader, packing, the debug trace. Lore and Utopia both go through it, so
an eval's `found` grades Utopia's assembly the same way as Lore's. The
Lore path is unchanged; the existing context-assembly tests pass without
edits.
- **`context-assembly-utopia.ts`.**
- One search per assembly (`topK` 50), a second page only when a section
is short and the first page says `hasMore`.
- Keeps statements-source hits whose pushed id is a `doc:` or `stmt:`
item (other sources are dropped, since search has no source filter yet),
sorts each path into `repo` or `adrs` with Lore's `classifyFile`, and
groups by path in first-rank order.
- Each item is the document's title then its statement texts in order,
under the Lore path, with Utopia's score; same item limits as Lore.
- Every other section still answers "no Utopia reader yet", as all do
when the deployment has no reader or the repo has no `settings.utopia`.
- **Wiring.** `repoContextSettings` also reads `settings.utopia`
{`kb_id`, `source_id`} in the same query. `/api/context` and the evals
pass `utopiaReadsFrom(env)` and those ids. Under `both`, the two halves
now assemble in parallel; the debug trace stays Lore's.

Caveats, recorded in the ADR-052 amendment: a tombstoned document can
still match until Utopia's search can exclude missing documents;
post-filtering by source id until search takes a source filter; code
items carry no text yet; typed graph reads wait for the release that
types Lore's relations.

Spec: `specs/utopia-graph-mirror` "The Utopia arm" FR12.1–FR12.6,
`specs/context-assembly` FR-1.5. (FR12 may need renumbering if #2790
lands first.)

## Tests

Pure and reader tests (libs/shared): one item per spec from three
statement chunks in order; another source's hits dropped; a second page
only when short; code has no reader. End-to-end with the real templates
(server-core): the ADR under Architecture Decisions with its statement
texts; the trace lists the kept paths; no settings falls back to
no-reader sections. Fixtures are built from the real `docStatementItems`
output with a hand-written fake `UtopiaReads`. `repoContextSettings`
reads the mirror ids.

`tsc` clean in libs/shared, server-core, lore-api; vitest libs/shared
knowledge + utopia 143 passed (2 skipped), server-core context 27,
lore-api context + context-evals + utopia 95; eslint clean on changed
files and `specs adrs`.

Generated with [Claude Code](https://claude.com/claude-code)
gedaiu added a commit that referenced this pull request Oct 17, 2026
…2804)

Utopia's database has been pinned at its 4-CPU limit while Lore pushes.
For every named entity in a pushed item, Utopia ran a nearest-neighbour
search over the whole knowledge base's name vectors, even after an exact
name match had already found the entity. That search only proposes
"possible duplicate" pairs, and for Lore's machine-made names
(`/pull/2790`) those pairs are always false.

0.1.0-re.9 is re.8 plus one change: a name in a pushed statements chunk
is resolved by its spelling, and extraction computes no name vector for
it, so the search never runs for pushes. Pushed names still get their
vectors stored afterwards, so documents that aren't pushes can still
find them. Upstream discussion: deeplethe/utopia#1293 and #1296. The
branch is `gedaiu/utopia` `perf/pushed-skip-name-vectors`. The merged
re.9 tree passes fmt, clippy and 1,616 tests.

Tag `v0.1.0-re.9` (re-dev 78a652ab), image
`ghcr.io/re-cinq/utopia:0.1.0-re.9`, digest
`sha256:ac5fc71b530f02a47aa63efb14a7264df55140627742595ed7cb06a5b1868463`.
No new migration.

After merge: `terraform apply
-target='kubernetes_deployment_v1.utopia[0]'` from main.

Generated with [Claude Code](https://claude.com/claude-code)
@lore-agent lore-agent Bot mentioned this pull request Oct 22, 2026
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant