Demo book layer: a private entity served as gists, built keyless-testably

[lukefwalton]

What & why

Refs #24, #28 (the keyless half; #28 closes when the keyed build lands on this branch). The demo gains the contract's other exposure: The Wonderful Wizard of Oz (Project Gutenberg #55) as one private entity, one fragment per chapter, served as gists (exposure: semantic). Private is the same layer assignment the sermons carry; what the layer shows is that a chapter reaches a consumer as provenance plus a released description and never as text.

• demo/build-lib.ts (new): the demo build on the schema-4 pipeline as pure functions (adapters → collectEntities → gist drafting into demo/corpus/projections.json → policy resolution → embed → writeIndex). Vectors are reused by content hash and gold-query vectors by id, so adding the book never re-embeds a Smith and the headline cannot move. demo/build.ts only supplies the real client and drafter. demo/build.test.ts runs the same functions keyless with a fake embedder and a scripted drafter: three layers from markdown, the book drafted and resolved, the served view stripped and verified, the sweep clean, a no-op rebuild embedding nothing, an edit re-embedding one fragment, a quoting draft failing the lint twice and resolving to locator, a canary in a released gist named by index.
• demo/run.ts: --natural+book (and --natural+synthetic+book) unions demo/corpus/index.book.json and loads demo/gold.book.yaml, after a keyless check of the layer's served view (validateServedIndex(toServedIndex(...))) and a canary sweep over every gist it would release; a hit fails the run. The headline is untouched: --natural 7/7 at rho 1.0000, the spire 9/9, int4 still rejected 7/9.
• demo/artifacts.test.ts reads schema 4 (entity allowlist; hashes through embedStringFor, which is the same invariant stated in v4 terms). demo/book.test.ts holds the committed layer to one fragment per chapter, every chapter served as a gist, no chapter text in the served view, and a clean sweep; it skips until the keyed build has run.
• demo/gold.book.yaml: two route cases pinned to the related-material template (#ch8, #ch1), a verbatim demand (oz-verbatim, the twin of eval/gold.yaml q12/q13), and five canaries from the novel for the sweep.
• scripts/prepare-gutenberg-book.mjs: turns the Gutenberg text into the book file without rewriting a word (both heading layouts; header, preface, illustrations, license dropped; --expect 24; --check-canaries names a canary that does not occur, by index), with a spawn test in test/prepare-gutenberg-book.test.ts.
• Docs: demo/README.md (new section), demo/corpus/README.md (provenance row, authored choice 8), build-handoff.md §6 (the owner's steps), docs/CONTRACT.md §12, .github/STANDARDS.md (the committed-artifacts exception now names the layer), CHANGELOG.md.

What is not in this PR, on purpose: the novel's text, its gists, and its vectors. Gutenberg and its mirrors are unreachable from the sandbox. The steps are build-handoff §6: run the converter on 55-0.txt, npm run demo:build (keyed; drafts 24 gists, embeds 24 chapters and 3 queries, touches nothing else), npm run demo:run -- --natural+book, commit demo/corpus/private-books/…, index.book.json, projections.json, query-vectors.json on this branch. Two things to read from that run: the unservable lines (a chapter whose draft failed the lint twice is served as a locator until its draft is edited) and the --check-canaries report (a canary the text does not contain guards nothing).

Checklist

• npm test passes without an API key (the offline CI gate) — 116 pass, 2 skipped by design until the layer is built, no live calls
• npm run typecheck passes
• No change that makes the eval pass by special-casing a question — the natural and synthetic gold are untouched; the book gold is new and pinned to behaviour, not facts
• A boundary stays structural — nothing on the served path changes; the layer is served through the same toServedIndex / project() the Smiths use, and the test asserts no chapter phrase survives the strip
• Touched the prompt / retrieval / validation / repair? No. npm run test:dist passes; demo:run holds all three headlines
• Read against .github/STANDARDS.md — the converter and the runner name pieces and indexes, never a canary or a gist

🤖 Generated with Claude Code

https://claude.ai/code/session_01F2ti456guy1bXVmjA3fBTV

---

Generated by Claude Code

[surmado-code-review]

Automated Checks (advisory, non-blocking)

✅ All checks passed.

---

Standards Compliance

I don’t see a boundary-integrity regression in the served path here. The new book-layer preflight in demo/run.ts:

const served = validateServedIndex(toServedIndex(bookFile), `${BOOK_INDEX} (served view)`);
const sweep = sweepCanaries(served, canaries);

is well aligned with the repo standards around validating the stripped served view before use and sweeping every released gist keylessly. Likewise, demo/build-lib.ts keeps disclosure resolution at build time via resolveDisclosure(...), and the stale-artifact guards from the previous revision (assertLayerNotOrphaned(...), assertNoOrphanedProjections(...)) are a good fail-closed fix.

One standards-adjacent gap is still open in scripts/prepare-gutenberg-book.mjs:

const missing = canaries.map((c, i) => (new RegExp(c, 'i').test(chapterText) ? null : i)).filter((i) => i !== null);

The surrounding comments/docs describe canaries as literal wording from the novel. Treating them as regexes means metacharacters change the meaning of the check, and malformed regex syntax will throw instead of reporting missing canaries by index. That weakens the handoff guard that is supposed to verify the canaries actually exist in the prepared text.

Summary

This PR extracts the keyed demo build into demo/build-lib.ts, adds the optional private-book demo layer served as semantic gists, and hardens the demo workflow with orphan-artifact checks plus query-vector content hashes. Most of the change is in demo build/run tooling and the Gutenberg prep path rather than the core served-path contract types.
Reviewer: most of the risk is in demo/build-lib.ts’s rebuild/orphan invariants and scripts/prepare-gutenberg-book.mjs’s canary verification semantics; the served-path checks themselves look aligned.

What to pay attention to

• demo/build-lib.ts — this now decides when artifacts are reused, redrafted, re-embedded, or refused. It’s the main place where partial checkouts or stale committed files could otherwise create misleading demo state.
• demo/run.ts — the keyless runner now has two important safety gates: served-view validation/canary sweep for the book layer, and query-vector hash freshness checks for edited gold queries.
• scripts/prepare-gutenberg-book.mjs — this is the human handoff tool that turns Gutenberg text into the committed book source, so small parsing/checking semantics here can cascade into confusing build or review outcomes.

Things I noticed

• 🟡 scripts/prepare-gutenberg-book.mjs still treats canaries as regexes, not literal phrases.
  This line:

  new RegExp(c, 'i').test(chapterText)

  means a canary containing ., ?, (, [, etc. will be interpreted as a pattern rather than the literal text the gold comments describe. It also means malformed regex syntax would throw instead of cleanly reporting missing canaries by index. So --check-canaries could pass, fail, or crash for reasons unrelated to whether the literal phrase is present.

Good patterns

• The validateServedIndex(toServedIndex(...)) + sweepCanaries(...) preflight in demo/run.ts is exactly the kind of explicit served-path check this repo wants.
• Adding contentHash to committed query vectors and refusing stale ids in demo/run.ts is a strong, keyless regression guard for gold-query edits.

Suggested improvements

1. In scripts/prepare-gutenberg-book.mjs, switch --check-canaries to a case-insensitive literal containment check, or escape canaries before building a regex.
2. Add one test case with a metacharacter-bearing canary so this behavior stays locked down.
3. Consider making goldForBuild() fail loudly when a layer has sources but its corresponding gold file is missing, instead of silently skipping those query vectors.

Questions for the author

• Was the regex interpretation in --check-canaries intentional? The comments in demo/gold.book.yaml and the script read as though these are meant to be literal phrase checks.

---

Surmado Code Review (v1.5-mt, review #5 on 1046dc7) is an automated review, designed to work alongside human judgment.

Want to change your STANDARDS.md or YML? Edit it directly, or tune it with our AI agent Scout.

Comment /rerun-review on this PR to re-review the current code with your latest STANDARDS.md and config — costs 1 additional PR credit (limit 3/hour per PR).

[lukefwalton]

Both points, in order.

Stale query vectors: real, fixed in 3338a28. Reuse by id alone would have let an edited gold query be judged against the embedding of its old text. demo/corpus/query-vectors.json is now schema 2: each entry carries queryContentHash of the text it was embedded from. demo:build reuses a vector only when the hash matches the gold text in tree and re-embeds the rest; demo:run refuses a missing or stale vector by id before the gate runs; demo/artifacts.test.ts holds every committed vector to the gold text, keylessly, so an edit without a rebuild fails CI. The committed file was stamped by scripts/stamp-query-vectors.ts (one-off, keyless, kept as provenance like the index migration): git history shows no query: line changed since the vectors were written in 166ad22, only notes and labels, so the stamp is honest and no vector changed (checked byte for byte). demo/build.test.ts now also edits a query under its id and asserts exactly that query is re-embedded while the others keep their vectors and hashes.

First build from empty: already handled. readIndex() returns an empty schema-4 index for a missing file (src/store.ts, "A missing file is an empty index"), and readProjections() returns an empty map for a missing file. The first test in demo/build.test.ts builds all three layers from an empty output directory, which is the path the keyed book build will take.

---

Generated by Claude Code

[lukefwalton]

All three in 4563015.

• Red, the swallowed error: the script checked for an absent gold file by catching loadGold, which would have hidden a malformed file behind "absent". It now checks existsSync and lets a file that exists and fails to load throw its own message.
• The {{author}} question: you read it right. loadGold returns substituted text, so the guard could never fire, and stamping with an empty author would have been wrong for a templated query. The script now substitutes config.authorName from demo/config.ts, exactly as goldForBuild does, and the guard is gone because the substitution is the fix. The demo's queries carry no {{author}} (checked), so the hashes already in the committed file are identical under either substitution; nothing in query-vectors.json changed.
• The duplicate-id guard: assertUniqueGoldIds in demo/build-lib.ts is now the one guard, used by the build, by loadGoldSet in the runner, and by the stamp script, so none of them can judge or stamp two cases under one id.

---

Generated by Claude Code

[lukefwalton]

Refused, not cleared. The split you describe was real: an absent private-books/ on a partial checkout would have written an empty projections map over the author's gists while index.book.json sat stale. 64893c8 closes it before anything is written, in two checks that both name the remedy. assertLayerNotOrphaned refuses a layer whose index file exists but whose source directory is absent or empty (spire and book alike). assertNoOrphanedProjections refuses a stored gist whose fragment no current source produces, by fragment id (ids are safe to print), which also catches a re-cut that dropped or renamed a chapter. The new test builds, removes the book directory, asserts the rebuild rejects with the embedder never opened and every artifact byte-identical, then drops one chapter and asserts the orphaned gist is refused by id. build-handoff §6 states the contract; the follow-up commit carries that sentence.

So the intended model is the second one you named: once book artifacts exist, a build without the book is an error, not a state the build quietly reconciles.

---

Generated by Claude Code

[lukefwalton]

/rerun-review