Repository navigation
Utopia commits: message only, the newest 200, new ones on push - #2801
Conversation
A commit item named every file the commit changed, and Utopia resolves each named entity against the whole base: a forty-file commit cost forty resolutions, and on 2026-10-17 Utopia's queue grew faster than its database could drain it. A commit item now carries the commit's words, author, time, parents and its pull request, issues and task. A walk takes the newest 200 commits above the cursor from GitHub's commit listing alone, with no read per commit, so a repo's history starts 200 commits back and later commits arrive through the push handler. changes stays declared in the ontology, unpushed.
|
Lore is reviewing this PR — 65f38f1a-ee93-4e9d-9d94-71e0a1961c61. Submit a review that requests changes to have Lore push a fix for it. Comment |
🔍 Lore Spec Impact — advisoryThis PR touches 66 statement(s) across 9 spec(s), and 34 of them have validating tests this PR does not change. Feature Specification: The spec-to-code chain mirrored in Utopia · 30 statement(s)
Backfill
validated by Backfill
validated by The ontology only its test links changed — the statement text itself is unchanged
validated by The ontology only its test links changed — the statement text itself is unchanged
validated by The ontology only its test links changed — the statement text itself is unchanged
validated by The push only its test links changed — the statement text itself is unchanged
validated by The push only its test links changed — the statement text itself is unchanged
validated by The push - … any other refusal, throws a `UtopiaPushEr…
+ … any other refusal, throws a `UtopiaError`…validated by The ingest only its test links changed — the statement text itself is unchanged
validated by The ingest only its test links changed — the statement text itself is unchanged
validated by …and 28 more statement(s). Weaker signals (28) — linked by a spec, not proven by a test runFeature Specification: The spec-to-code chain mirrored in Utopia · 1 statement(s)
Backfill
validated by Feature Specification: Shared Utilities · 3 statement(s)
Functional Requirements
validated by Functional Requirements
validated by Functional Requirements
validated by Feature Specification: Autonomous Review Loop · 7 statement(s)
validated by
validated by
validated by
validated by
validated by
validated by …and 18 more statement(s). 2 changed statement(s) had no validating test, so no coverage broke. 309 new statement(s) have no test link yet. Deterministic · graph @ |
![lore-agent[bot]](https://avatars.githubusercontent.com/u/141229953?s=60&v=4){kind=small}
There was a problem hiding this comment.
Lore review — Approved
Drops per-file changes statements and per-commit GitHub detail calls from the Utopia commit mirror, capping walks to the newest 200 commits from the branch listing.
Must fix (0) · Consider (1) · Nits (1)
Reviewed by gemini-4-argon
Submit a review that requests changes to have Lore push a fix for it. Comment @lore review to re-run the review.
Posted by floor, visit 1773a964-f607-44d9-9235-a6fdb058a7e6.
…d what an absent cursor means
There was a problem hiding this comment.
Lore review — Approved
Comment and spec wording updates accurately reflect that changed files are read from the repository and an absent cursor takes the newest limit commits.
Must fix (0) · Consider (0) · Nits (0)
Reviewed by gemini-4-argon
Submit a review that requests changes to have Lore push a fix for it. Comment @lore review to re-run the review.
Posted by floor, visit 2462cb9d-f515-4643-9adc-7c21b2daf912.
…5) (#2802) Utopia held each repo's default-branch commits, but a pull request or an issue appeared only as a number and a link, taken from a squash-merge's commit message, and releases and tags not at all. This PR mirrors GitHub's own record of every issue, pull request, tag and published release into the repo's knowledge base (`specs/utopia-graph-mirror` FR14 and FR15, ADR-052 amendment of 2026-10-17). **Issues and pull requests.** GitHub's issues listing returns pull requests too, with their merge time, so a single walk covers both. It reads oldest update first from a new cursor, `settings.utopia.tracker_since`, and pushes `pull:<n>` and `issue:<n>` items. Each item holds title, state (open, closed or merged), the login that opened it, labels, the open, close and merge times, and the description. A pull request's `Closes`/`Refs`/`Lore-Task:` lines link it to its issues and task, stated through a module the commit mirror now shares, so both mirrors say the same thing about the same entity. The walk takes up to 2,000 entries per run, moves the cursor past each one that lands, and stops at the first failure. **Tags and releases.** A `tag:<name>` item points to the commit the commit mirror names, and a `release:<tag>` item (title, author, publish time, notes) publishes its tag. That way a release reaches the pull requests behind it. A repo has few of these, so every push sends them all, with no cursor: an unchanged item answers unchanged, and a refused one is sent again next time. Drafts are skipped. **When it runs.** - A changed issue or pull request emits `github.tracker.changed`; a published or edited release, or a created tag, emits `github.release.changed`. Both come from the same delivery under its own dedupe key (`github:<id>:utopia`). - The stations answer by posting `POST /api/repos/{owner}/{repo}/utopia/tracker` or `/releases`. - The weekly backfill runs both after the commits. - A repo's hook now requires the `release` and `create` events. An existing hook gets them when the ensure step runs for that repo, as `push` did in #2796. - The commit, tracker and release walks share one loop and one per-repo queue (`cursor-walk.ts`). This branch is rebased onto #2801: commit items carry no file names, and the commit walk takes the newest 200. **Ontology version 5.** Version 5 adds `Release` and `Tag`, `points_to` and `publishes`, and the tracker attributes. Once it deploys, every base below 5 re-imports the ontology on its next push. If re.8 is not live by then, those bases store 5 with every relation typed as a state, so after the apply the reset is `5 → 4`, not `4 → 3`. Out of scope: reviews, review comments, conversation comments, and tombstoning deleted issues, releases or tags. Checks run locally: vitest across libs/shared, apps/lore-api and apps/stations; `tsc --noEmit` on every package that imports shared; `eslint .` with 0 errors; `prettier --check`. Generated with [Claude Code](https://claude.com/claude-code)
The first tracker walk on lore, run right after #2802 deployed, pushed nothing. Its first item, `pull:1`, timed out after 30 seconds: Utopia's database was at its 4-core limit, with connections waiting 2 to 6 seconds. One walk pushes up to 2,000 items, and each is a job that database has to work through, so a first walk of a repo with a long history piles onto an already busy queue. This PR caps a walk at 200 entries, as #2801 did for commits. A long backlog now arrives in batches of 200, each resuming from `tracker_since`. A walk runs for every changed issue or pull request and once in the weekly backfill, so a busy repo like lore catches up within a day, and a quiet one over a few Sundays. Spec FR14.4 and the ADR-052 amendment now say 200. Generated with [Claude Code](https://claude.com/claude-code)
1 participant
A commit item named every file the commit changed, so a forty-file commit made Utopia resolve forty entities against the whole knowledge base. On 2026-10-17 Utopia's queue grew about twice as fast as its database could drain it, even after it was sized up to four CPUs (#2799). The weekly backfill would then have walked up to 2,000 commits for each of the 14 repos on Sunday.
getCommitper commit and no.lore/ingest.ymlread. A repo's history in Utopia starts 200 commits before its first walk, and later commits arrive through the push handler as they land. Webhook: a repo's hook carries push events #2796 putpushon the webhook's required events; the hooks get it with one ensure per repo after this deploys.changesstays declared in ontology v4, pushed by no item. Removing it would force a re-import of every base now, and Utopia's imports only add, so it would stay in the live bases anyway.Spec FR12.2–FR12.7 and the intro are rewritten; ADR-052 gains a revision. Locally the shared, lore-api and stations suites,
tscand the repo-wideeslintall pass.Related: #2790 (the commit mirror), #2796 (push events), #2799 (database sizing).
🤖 Generated with Claude Code