Demo book layer: a private entity served as gists, built keyless-testably

.github/STANDARDS.md

@@ -54,7 +54,7 @@ Fail fast and name the problem and the remedy. Silent fallbacks hide bugs.
 Not a hardened production service, but a few things matter.
 
 - **Secrets: never commit `.env`; don't log prompts with keys.**
-- **Don't leak private embeddings, text, or projections into committed artifacts.** (`artifacts/` is gitignored for a reason: the index and the projections file are private material.) The one exception is `demo/`: it commits only the exact public-domain natural sources and flagged synthetic spire allowlisted in `demo/artifacts.test.ts`, on purpose, to reproduce the headline with no key. Some public-domain sources are routed through the private layer to exercise the boundary, but that is a demo layer assignment, not a secrecy claim. Do not generalize it to genuinely-private corpora.
+- **Don't leak private embeddings, text, or projections into committed artifacts.** (`artifacts/` is gitignored for a reason: the index and the projections file are private material.) The one exception is `demo/`: it commits only the exact public-domain natural sources and flagged synthetic spire allowlisted in `demo/artifacts.test.ts`, and the public-domain book layer with its machine-drafted gists held to `demo/book.test.ts`, on purpose, to reproduce the headline and the `semantic` exposure with no key. Some public-domain sources are routed through the private layer to exercise the boundary, but that is a demo layer assignment, not a secrecy claim. Do not generalize it to genuinely-private corpora.
 - **Custody is named, not overstated.** Private text reaches the embedding provider at ingest and, only where a gist was requested, the drafter. The boundary is disclosure on the served path, not confidentiality from the providers the operator chooses (`docs/CONTRACT.md` §10). Don't describe it as more.
 - **Performance: brute-force cosine is intentional at this scale.** Don't add pgvector, HTTP, or caching in a drive-by PR unless the README's "Where to take it" story is the explicit goal.
 

CHANGELOG.md

@@ -199,6 +199,27 @@ npm trusts this repository's `release.yml`.
   share a locator are refused by position and scheme. `npm run index` reads the books
   beside the records and the notes; a book's `semantic` request is drafted per
   fragment against the whole book's text, as CONTRACT §12 describes.
+- The demo's book layer (`npm run demo:run -- --natural+book`): *The Wonderful
+  Wizard of Oz* (Project Gutenberg #55) as one private entity, a fragment per
+  chapter, served as gists (`exposure: semantic`). `demo/build.ts` now runs the
+  schema-4 pipeline through `demo/build-lib.ts` (adapters, `collectEntities`,
+  gist drafting into `demo/corpus/projections.json`, policy resolution, vectors
+  reused by content hash so a Smith is never re-embedded, gold-query vectors
+  reused by id and by a hash of the query text, so an edited query is
+  re-embedded and `demo:run` refuses a stale one; a layer whose artifacts
+  exist but whose sources are gone, or a stored gist no source produces, is
+  refused before anything is written), and the tests run the same functions keyless with a fake
+  embedder and a scripted drafter (`demo/build.test.ts`). The runner checks the
+  layer's served view and sweeps every released gist against the canaries in
+  `demo/gold.book.yaml` before the gate; `demo/book.test.ts` holds the committed
+  layer to one fragment per chapter, every chapter served as a gist, and a
+  clean sweep (skipped until the keyed build has run).
+  `scripts/prepare-gutenberg-book.mjs` turns the Gutenberg text into the book
+  file without rewriting a word. `demo/corpus/query-vectors.json` is schema 2
+  (a text hash per entry); the committed file was stamped keylessly by
+  `scripts/stamp-query-vectors.ts` from the gold text in tree, which had not
+  changed since the vectors were written, and no vector changed. The headline
+  (`--natural`) does not include the book and is unchanged.
 
 ### Removed
 

demo/README.md

@@ -22,6 +22,7 @@ Run it:
 npm run demo:run                                  # int8, real corpus: the headline, keyless
 npm run demo:run -- --natural+synthetic           # add the spire and its gold
 npm run demo:run -- --natural+synthetic --bits 4  # int4: the gate rejects the spire route flips
+npm run demo:run -- --natural+book                # add the book layer: a private entity served as gists (checked and swept first)
 npm run demo:run -- --full                        # also run the answer-mode pass (needs a key)
 ```
 
@@ -78,6 +79,35 @@ Two facts make int8 admissible, and they differ in kind (the §6 split):
 The headline run is **keyless**: it reads committed full-precision vectors and
 committed gold-query vectors, so no embedding call is made. A key is needed only
 to regenerate the vectors (`demo:build`) or to run the `--full` answer pass.
+
+## The book layer: a private entity served as gists
+
+`--natural+book` adds one more layer beside the Smiths: *The Wonderful Wizard
+of Oz* (Project Gutenberg #55) as a **private** entity, one fragment per
+chapter, `exposure: semantic`. Private is the same layer assignment the
+sermons carry (`corpus/README.md` §2): the text is public domain and sits in
+the repo on purpose, and what the layer shows is the contract's other
+exposure. A chapter reaches a consumer as provenance (title, author, year,
+`ch. 8`) plus a one-paragraph description the build drafted and the author
+released, never as text. The descriptions are drafted at `demo:build` (keyed),
+linted so they cannot carry a run of five words from the chapter or from the
+whole book, stored in `corpus/projections.json` as the author-facing file
+(edit, veto, or mark one reviewed there), and committed because they derive
+from public-domain text; a genuinely private corpus commits neither its
+vectors nor its gists.
+
+The run is keyless and does two things before the gate. It checks the layer's
+served view (text blank, gists present, the strip verified by the store's own
+load-time check) and sweeps every gist it would release against the canaries
+in `gold.book.yaml`; a hit is named by index and fails the run. Then the gate
+runs the natural gold and the book's route cases over the union: a chapter
+question must win the top slot for its chapter, and the Smith questions must
+still resolve to their Smiths with the novel beside them. The headline
+(`--natural`) never includes the book; the layer is reported on its own lines.
+The text is not in the repo until someone runs
+`scripts/prepare-gutenberg-book.mjs` on the Gutenberg file
+([`build-handoff.md`](../docs/scaling-demo/build-handoff.md) §6); until then
+the flag says so and the tests that read the layer skip.
 That answer pass exercises route *selection*, which is what quantization moves;
 it does not touch A2, the answer model's confabulation residue, which the
 encoding never exercises.

demo/artifacts.test.ts

@@ -1,14 +1,19 @@
 import assert from 'node:assert/strict';
-import { createHash } from 'node:crypto';
+import { existsSync } from 'node:fs';
 import { test } from 'node:test';
 
-import { embedText, noteEmbedText } from '../src/corpus.js';
-import { truncateForEmbedding } from '../src/embedding.js';
+import { embedStringFor } from '../src/embed-string.js';
 import { loadGold } from '../src/evaluate.js';
-import { entrySourceId, readIndexFile } from '../src/store.js';
-import { readQueryVectors } from './query-vectors.js';
+import { readIndex } from '../src/store.js';
+import { contentHash } from './build-lib.js';
+import { queryContentHash, readQueryVectors } from './query-vectors.js';
 
-const EXPECTED_NATURAL_SOURCES = [
+const NATURAL = 'demo/corpus/index.json';
+const SYNTHETIC = 'demo/corpus/index.synthetic.json';
+const BOOK = 'demo/corpus/index.book.json';
+const AUTHOR = 'Smith Collection';
+
+const EXPECTED_NATURAL_ENTITIES = [
   'adam-smith:theory-of-moral-sentiments-justice',
   'adam-smith:theory-of-moral-sentiments-sympathy',
   'adam-smith:wealth-of-nations-division-of-labour',
@@ -23,53 +28,64 @@ const EXPECTED_NATURAL_SOURCES = [
 ].sort();
 
 test('committed demo index matches the public-domain source allowlist', () => {
-  const natural = readIndexFile('demo/corpus/index.json');
-  const actual = natural
-    .map((entry) => (entry.sourceType === 'record' ? entry.record.id : entry.note.id))
-    .sort();
+  const natural = readIndex(NATURAL);
+  const entities = natural.entities.map((e) => e.id).sort();
   assert.deepEqual(
-    actual,
-    EXPECTED_NATURAL_SOURCES,
-    `committed demo index source ids changed; update the public-domain provenance and allowlist deliberately\n` +
-      `actual: ${actual.join(', ')}`,
+    entities,
+    EXPECTED_NATURAL_ENTITIES,
+    `committed demo index entities changed; update the public-domain provenance and allowlist deliberately\n` +
+      `actual: ${entities.join(', ')}`,
   );
+  // Whole records and whole sermons: one fragment per entity, nothing split.
+  assert.deepEqual(natural.entries.map((e) => e.fragment.entityId).sort(), EXPECTED_NATURAL_ENTITIES);
 
-  const synthetic = readIndexFile('demo/corpus/index.synthetic.json');
+  const synthetic = readIndex(SYNTHETIC);
   assert.deepEqual(
-    synthetic.map((entry) => (entry.sourceType === 'note' ? entry.note.id : entry.record.id)),
+    synthetic.entities.map((e) => e.id),
     ['note:syn-amos-justice-margin'],
     'committed synthetic spire changed; keep it to the single flagged near-tie unless the demo is recalibrated',
   );
+  assert.deepEqual(synthetic.entries.map((e) => e.fragment.entityId), ['note:syn-amos-justice-margin']);
 });
 
-test('committed demo hashes match the current embed-text derivation', () => {
-  // Mirrors contentHash in demo/build.ts (that file runs its keyed build on
-  // import, so it can't be imported here). This pins the schema-v3 migration
-  // invariant: the committed vectors were embedded from label+body at v2, and
-  // today's derivation (noteEmbedText = title+body, embedText for records)
-  // must reproduce the exact bytes those hashes were taken over. Drift in the
-  // migration or the derivation fails here, keylessly — not at re-embed time.
-  const hash = (text: string) =>
-    createHash('sha1').update(truncateForEmbedding(text)).digest('hex').slice(0, 16);
-  for (const path of ['demo/corpus/index.json', 'demo/corpus/index.synthetic.json']) {
-    for (const entry of readIndexFile(path)) {
-      const text = entry.sourceType === 'note' ? noteEmbedText(entry.note) : embedText(entry.record);
+test('committed demo hashes match the current embed-string derivation', () => {
+  // The committed vectors were embedded from label+body at v2 and carried
+  // through the schema-3 and schema-4 migrations without re-embedding. This
+  // pins that invariant: the embed string the adapters produce today
+  // (src/embed-string.ts) must reproduce the exact bytes those hashes were
+  // taken over, so drift in an adapter or the derivation fails here,
+  // keylessly, not at re-embed time. The book layer, when built, is held to
+  // the same rule.
+  const layers = [NATURAL, SYNTHETIC, ...(existsSync(BOOK) ? [BOOK] : [])];
+  for (const path of layers) {
+    const file = readIndex(path);
+    const entities = new Map(file.entities.map((e) => [e.id, e]));
+    for (const entry of file.entries) {
+      const entity = entities.get(entry.fragment.entityId);
+      assert.ok(entity, `fragment '${entry.fragment.id}' names an unknown entity in ${path}`);
       assert.equal(
         entry.contentHash,
-        hash(text),
-        `contentHash drift for '${entrySourceId(entry)}' in ${path}`,
+        contentHash(embedStringFor(entry.fragment, entity)),
+        `contentHash drift for '${entry.fragment.id}' in ${path}`,
       );
     }
   }
 });
 
 test('committed demo query vectors match the gold suite ids', () => {
   const gold = [
-    ...loadGold('demo/gold.yaml', 'Smith Collection'),
-    ...loadGold('demo/gold.synthetic.yaml', 'Smith Collection'),
+    ...loadGold('demo/gold.yaml', AUTHOR),
+    ...loadGold('demo/gold.synthetic.yaml', AUTHOR),
+    ...(existsSync(BOOK) ? loadGold('demo/gold.book.yaml', AUTHOR) : []),
   ];
   const queryVectors = readQueryVectors('demo/corpus/query-vectors.json');
   assert.ok(queryVectors);
 
   assert.deepEqual([...queryVectors.byId.keys()].sort(), gold.map((g) => g.id).sort());
+  // And each vector is the embedding of the query text in tree: an edit to a
+  // gold query under the same id fails here, keylessly, until demo:build
+  // re-embeds it (demo:run refuses the same way).
+  for (const g of gold) {
+    assert.equal(queryVectors.hashes.get(g.id), queryContentHash(g.query), `query vector for '${g.id}' was embedded from other text`);
+  }
 });

demo/book.test.ts

@@ -0,0 +1,57 @@
+// The committed book layer, when it has been built (build-handoff.md §6):
+// one public-domain novel as a private entity, one fragment per chapter,
+// every chapter served as a gist, the served view stripped, the canaries
+// clean. Skipped, not failed, until the keyed build has run.
+
+import assert from 'node:assert/strict';
+import { existsSync } from 'node:fs';
+import { test } from 'node:test';
+
+import { loadGoldFile, sweepCanaries } from '../src/evaluate.js';
+import { readIndex, toServedIndex, validateServedIndex } from '../src/store.js';
+
+const BOOK = 'demo/corpus/index.book.json';
+const GOLD = 'demo/gold.book.yaml';
+const ENTITY = 'book:the-wonderful-wizard-of-oz';
+const CHAPTERS = 24;
+
+const skip = existsSync(BOOK) ? false : `${BOOK} is not built yet (build-handoff.md §6)`;
+
+test('committed book layer: one private entity, one fragment per chapter, every chapter served as a gist', { skip }, () => {
+  const file = readIndex(BOOK);
+  assert.deepEqual(file.entities.map((e) => e.id), [ENTITY]);
+  const entity = file.entities[0]!;
+  assert.deepEqual(entity.disclosure, { raw: 'private', exposure: 'semantic' });
+  assert.equal(entity.policy?.publicTitle, true);
+  assert.ok(entity.attribution.some((a) => a.name === 'L. Frank Baum'));
+
+  const chapters = file.entries.map((e) => e.fragment.locator);
+  assert.equal(chapters.length, CHAPTERS, `the novel has ${CHAPTERS} chapters; a sub-split chapter means maxFragmentChars was too low`);
+  assert.deepEqual(
+    chapters.map((l) => l.map((x) => `${x.scheme}:${x.value}`).join('.')).sort(),
+    Array.from({ length: CHAPTERS }, (_, i) => `chapter:${i + 1}`).sort(),
+  );
+  for (const entry of file.entries) {
+    assert.deepEqual(
+      entry.fragment.disclosure,
+      { raw: 'private', exposure: 'semantic' },
+      `${entry.fragment.id} is not served as a gist: edit its draft in demo/corpus/projections.json and rebuild`,
+    );
+  }
+});
+
+test('committed book layer: the served view carries gists and provenance, never the text; the sweep is clean', { skip }, () => {
+  const file = readIndex(BOOK);
+  const served = validateServedIndex(toServedIndex(file), `${BOOK} (served view)`);
+  const serialized = JSON.stringify(served);
+  for (const entry of file.entries) {
+    // The first sentence of every chapter is absent from the served view.
+    const opening = entry.fragment.text.split('\n\n').pop()!.slice(0, 60);
+    assert.ok(!serialized.includes(opening), `${entry.fragment.id}: chapter text reached the served view`);
+  }
+  const { canaries } = loadGoldFile(GOLD, 'Smith Collection');
+  assert.ok(canaries.length >= 3, 'the book gold carries canaries');
+  const sweep = sweepCanaries(served, canaries);
+  assert.deepEqual(sweep.issues, []);
+  assert.equal(sweep.gists, CHAPTERS);
+});