Repository navigation
feat(engine-mcp): a capability search finds a symbol by name and answers in one call (PRD-865) - #644
Conversation
44b8b89 to
3305634
Compare
3305634 to
8c7787d
Compare
|
The merge queue stopped on
Fix, push, re-enqueue. |
|
The merge queue stopped on
Fix, push, re-enqueue. |
|
The merge queue stopped on
Fix, push, re-enqueue. |
…ers in one call (PRD-865) The index read only `situations` and `aliases`, so an exact symbol name answered `verdict: none`: `godrays`, `traa` and `mergeParts` are exported by the manifest and were unfindable, while `engine_capability_detail` cost a call per hit for a row that lacked only `signature`, `overrides` and `deprecated`. - The tokenizer splits camelCase, and a request that spells a name as a run of whole words leads with that symbol: whole-name equality is case- and separator-insensitive (`mergeparts`, `characterbody3d`), a node name inside a sentence counts (`GPUParticles3D sparks`, `Area3D trigger`), and the lead scales with how much of the request the name accounts for, with a longer name winning a tie. A name mentioned in passing cannot outrank the mechanic the request is about. - Names carry their own index (digits stripped, `GPUParticles3D` reads as `gpu particles`) and the summary is scored over it at low weight, which is what makes a thinly described entry like `traa` findable. Keeping it apart from the prose index is deliberate: folding it in cost `SnowField` its row in the recall budget. - A row equals the detail minus `situations`/`aliases`. `AGENTS.md`, `agent-docs/kit-router.md`, the shared `threenative-capabilities` skill and all sixteen kit `AGENTS.md`/`CLAUDE.md` now say read every row and call the detail only for a symbol the search did not return. - MCP text is compact JSON; indentation cost 11.8% of every response and the shell command re-indents its own copy. - A frozen gold set and a two-arm A/B: 150 real queries from 26 `.threenative/engine-mcp.log` files, 100 dev / 50 held out, held-out sha256 in the fixture and the PRD, 20 exact-symbol. Labels came from a fresh opus process that read only the logs and the manifest. `search-gold.spec.ts` locks the held-out floor; `scripts/capability-search-ab.ts` extracts the base source with `git archive`, measures both arms and runs a live `claude -p` arm. Against ecd2a1a on the frozen held-out half: recall@5 0.580 -> 0.780, recall@1 0.400 -> 0.620, 0 rows lost from the top 5, modelled tokens -47%. Live: MCP calls -56..62%, MCP response bytes -30..36%, correct-symbol rate unchanged, 0 failed runs. Two independent opus reviews; every finding fixed, including the sixteen scaffold trees re-measured on the merged tree. Also lands the develop red this PR was queued behind: the platformer kit gets the worker entry pair the starter shipped in #574 (`index.html` -> `src/worker-page.ts`, `main.html` for the page-thread arm, `src/worker.ts`, and the two-entry build). Its golden-path leg boots `index.html` and asserts threads on, so without the worker page the scaffolded game never booted and the leg reported `TN_PLAYTEST_BRIDGE_MISSING`. The platformer's game code is worker-safe (`render/loading.ts` guards its canvas behind `typeof document`).
8c7787d to
4a78221
Compare
|
Merged as One correction to the commit message, for the record: the platformer worker-entry port in this commit ( So the page boots and the engine reports ready; what is missing is |
PRD:
docs/PRDs/done/PRD-865-capability-search-finds-by-name-and-answers-in-one-call.md(100%: 3/3 phases, 4/4 acceptance criteria)One commit on
origin/develop(e897bf827). Rebased after #633–#645 landed: the previous head carried nine pre-squash doc commits that are now on develop, which is what made the PR conflicting.Also lands the develop red this PR was queued behind (AGENTS.md: fix a red on develop in the PR already in flight).
develop's board has been red since #639/#640, which is why every merge group — including this PR's three attempts — ejected:golden-path-template (platformer)failed withTN_PLAYTEST_BRIDGE_MISSINGbecause the platformer kit still booted/src/main.tson the page thread while the golden path servesindex.htmland asserts threads on. The kit now has the worker entry pair the starter shipped in #574:index.html→src/worker-page.ts,src/worker.ts,main.htmlfor the page-thread arm, and the two-entry build. Its game code is worker-safe (render/loading.tsguards its canvas behindtypeof document). That is the only reason this PR is not strictly PRD-865's scope.What changed
engine_search_capabilitiesnow finds a capability by its exported symbol name and answers in one call.mergeparts,characterbody3d), including a node named inside a sentence (GPUParticles3D sparks,Area3D trigger). The lead scales with how much of the request the name accounts for and a longer name wins a tie, soGPUParticles3Dbeats the genericParticleswhilefirst person view with a weapon held in the handstill answers the camera, not theViewsymbol.GPUParticles3Dreads asgpu particles) and the summary is scored over it at low weight — that is what makes a thinly described entry (traa) findable. It stays apart from the prose index on purpose: folding it in costSnowFieldits row in the recall budget.ICapabilitySearchResultgainssignature,overridesanddeprecated: a row is the detail minussituations/aliases.AGENTS.md,agent-docs/kit-router.md, the sharedthreenative-capabilitiesskill and all sixteen kitAGENTS.md/CLAUDE.mdnow say read every row and callengine_capability_detailonly for a symbol the search did not return.threenative engine_* …from a shell re-indents its own copy..threenative/engine-mcp.logfiles, 100dev/ 50heldout, 20 exact-symbol, held-out sha256 frozen in the fixture and in the PRD. Labels were written by a fresh opus process that read only the logs and the manifest; 58 rows it judged engine-unowned were dropped before selection.A/B against
ecd2a1a56(arm A), frozen held-out set, measured on this merged treeLive arm, 10 briefs × 2 runs per arm (
claude -p --model sonnet --strict-mcp-config, the arm's rule as the prompt prefix), three runs because the base arm varies with how many details the model opens:The split is recorded in the PRD's Decisions: the row change is the smaller half (details no longer opened ≈ 1.65 KB per search, five complete rows ≈ 4.5 KB, because a row is ~900 B at p50, not the 416 B the PRD assumed), and compact serialization adds 11.8% on every response including the reference stubs. Both figures print on every harness run.
Review
Two independent opus reviews of this branch; every finding fixed.
engine-mcp-tools.jsondescription, the sharedthreenative-capabilitiesskill, the trackedpackages/core/mcp/engine-server.mjsbundle, and the sixteen scaffold trees (re-measured on the merged tree).physicsansweredafterPhysics); the next shape was too strict (whole-query equality missed a node named in a sentence, and a rarity floor hiduse rapier). The final rule has no rarity constant: spelling plus a share-scaled lead plus a specificity tie-break, all 20 gold exact-symbol rows and both reviewers' probe sets green.startertwo words over its 1,200-word instruction budget (the line is nowrows answer; engine_capability_detail only for omitted symbols, zero delta, full detail in thethreenative-capabilitiesskill), and seven Unreal-vocabulary aliases are now recalled with the aliases stripped because their capabilities' summaries name the same words — flaggedbaselineAlreadyRecalledwith a comment rather than deleted.Gates
pnpm tsx scripts/capability-search-ab.ts --base ecd2a1a56 --live --runs 2— exit 0, every gate passes; artifactartifacts/capability-search-ab.json(local, gitignored).pnpm exec vitest run packages/engine-mcp— 153 passed, 1 skipped: the gold spec (20/20 names at rank 1, locked held-out floor 0.60 / 0.76), the 86 search cases, and both reviewers' probes.pnpm exec vitest run scripts/__tests__/kit-conventions.spec.ts scripts/__tests__/mcp-tool-schemas.spec.ts scripts/__tests__/verify-registry-install.spec.ts packages/core/__tests__/mcp-install.spec.ts packages/create-threenative/__tests__/scaffold-mcp.spec.ts scripts/__tests__/sync-agent-docs.spec.ts— 166 passed;scaffold.spec.tsgreen with the merged-tree hashes.pnpm caps:recall— 0.938053, unchanged (an earlier shape of this change costsnow.refill).pnpm typecheck,pnpm lint(0 errors),pnpm check:docs,pnpm sync:agents --check,pnpm quality: green.ci:fast: lint and drift pass.pnpm buildon this worktree fails only inexamples/spell-lab:TN_WASM_ENGINE_MISSING(a fresh worktree has no Wasm build; packages all build).