From c24875716d56216f6526f27f7dcdb1db12fbc205 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 7 Oct 2026 07:16:34 +0000 Subject: [PATCH 1/5] Private books in the teaching corpus; collectEntities compares by content - privateBooksDir (off by default): one markdown file per book, cut on its chapter headings or page markers into one private entity with many fragments (buildPrivateBooks, fromPrivateBook). The heading goes into the fragment's text and nowhere else; publicTitle and requireReview land on the entity's policy; the metadata lint runs over the whole book before npm run index drafts or embeds anything; two pieces that share a locator are refused by locator. The example content does not change. - collectEntities: two descriptions of one entity compare with object keys sorted at every level instead of by property insertion order (the review nit on #23), with a regression test; order inside a list is still content. - A bare year in frontmatter (YAML reads `date: 1900` as a number) is a date at year precision instead of a silently dropped field. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01F2ti456guy1bXVmjA3fBTV --- CHANGELOG.md | 27 ++++ README.md | 32 ++++- archive.config.ts | 4 + docs/CONTRACT.md | 14 +- scripts/smoke-dist.mjs | 2 + src/adapters/teaching.ts | 69 +++++++++- src/cli/build-index.ts | 19 ++- src/corpus.ts | 139 ++++++++++++++++++- src/ingest/collect.ts | 14 +- src/types.ts | 53 ++++++++ test/books.test.ts | 282 +++++++++++++++++++++++++++++++++++++++ test/contract.test.ts | 57 ++++++++ 12 files changed, 694 insertions(+), 18 deletions(-) create mode 100644 test/books.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index d37176f..050dcce 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -183,12 +183,39 @@ npm trusts this repository's `release.yml`. legacy view (which drafts nothing) resolves a requested `semantic` to `locator` itself. `projectionContentHash` moves to `src/ingest/projections.ts`, beside `projectionProblem`. +- Private books in the teaching corpus: `privateBooksDir` in `archive.config.ts` + (off by default; the example content does not change), one markdown file per + book. `buildPrivateBooks` (`src/corpus.ts`) reads the frontmatter (`title`, + `about`, and optionally `authors`, `identifiers`, `date`, `version`, `themes`, + `exposure`, `publicTitle`, `requireReview`, `fragmentBy`, `maxFragmentChars`, + `firstPage`) and cuts the body with `fragmentByHeadings` or + `fragmentByPageMarkers` into pieces whose locators are structural; a leading + `# Title` that repeats the frontmatter title is dropped as the title page. + `fromPrivateBook` (`src/adapters/teaching.ts`) yields one private entity with + one fragment per piece (`book:x#ch12`, `book:x#ch12.s3`, `book:x#p184`), the + heading inside the fragment's text and nowhere else, `publicTitle` and + `requireReview` on the entity's policy, and the metadata lint run over the + whole book before `npm run index` drafts or embeds anything; two pieces that + share a locator are refused by locator. `npm run index` reads the books + beside the records and the notes; a book's `semantic` request is drafted per + fragment against the whole book's text, as CONTRACT §12 describes. ### Removed - `scripts/build-blind-artifact.mjs`. Review is open, not blind; the script's anonymized tarball has no remaining use. +### Fixed + +- `collectEntities` compared two descriptions of one entity by + `JSON.stringify`, so the same entity emitted with its properties in another + insertion order (two adapters, or one adapter run twice) was refused as + "described two ways". It now compares by content, with object keys sorted at + every level; the order inside a list (creators, identifiers) is still + content. A regression test pins both. +- A bare year in frontmatter (`date: 1900`, which YAML reads as a number) was + dropped silently by every reader; it is now the date at year precision. + ### Breaking in the 3.0.0 line (summary) Recorded here in one place; `docs/CONTRACT.md` is the normative text. diff --git a/README.md b/README.md index 77cc2b3..ec780dc 100644 --- a/README.md +++ b/README.md @@ -65,8 +65,11 @@ notebook in `example-content/notebook/` is the private layer. Each note declares the public page it routes to (`about`), where the moment lives (`locator`), a linted display name (`label`), and, optionally, the `exposure` it wants; its `title` and body stay private, embedded so retrieval can find the -moment. The teaching adapters (`src/adapters/teaching.ts`) turn those files -into entities and fragments; a consumer writes its own. +moment. A private **book** (`privateBooksDir`, off by default) is one entity +with many fragments: the file is cut on its chapter headings or page markers, +and each piece travels as where it is (`ch. 12`, `p. 184`) or as its gist, +never as text. The teaching adapters (`src/adapters/teaching.ts`) turn those +files into entities and fragments; a consumer writes its own. > In production ([Ask the Archive](https://lukefwalton.com/ask/)), published > podcast passages are public fragments while unpublished transcript text is @@ -269,8 +272,29 @@ get `temperature: 0`). index it without ever serving it — see `docs/CONTRACT.md` §3 and §5. No private layer? Remove `privateNotesDir` from the config and the engine runs public-only. -4. Replace `example-content/` with your corpus and rerun `npm run index`. -5. Rewrite `eval/gold.yaml` for your corpus — keep the refusals. +4. A private book is one markdown file in `privateBooksDir` (set it in the + config; it is off by default). The frontmatter is the book's public + surface: `title` (required; checked against the text unless + `publicTitle: true` says it is public by construction), `about` (the public + URL a hit routes to, required), and optionally `authors` (names, or + `{ name, role }` entries), `identifiers` (`{ scheme, value }`: isbn, doi), + `date`, `version` ("manuscript"), `themes`, `exposure` (`locator` by + default; `semantic` has `npm run index` draft one gist per fragment for you + to authorize in `artifacts/projections.json`; `none`), and + `requireReview: true` to serve no gist you have not marked reviewed. The + body is the private text. It is cut into fragments on its `#`/`##` headings + (`fragmentBy: headings`, the default: "Chapter 12" is `ch. 12`, any other + heading is counted among them; a leading `# Title` that repeats the + frontmatter title is the title page, not a chapter) or on page markers + (`fragmentBy: pages`: a `<<>>` line, or a form feed with + `firstPage`); pieces longer than `maxFragmentChars` (6000) are sub-split on + paragraphs (`ch. 12, §3`). The locator is the only authored-looking thing a + hit carries about a chapter, and it is structural; the heading goes into the + embedding and nowhere else. The same build-time lint that checks a note's + label checks the book's title, version, and creators against the whole book + before anything is drafted or embedded. +5. Replace `example-content/` with your corpus and rerun `npm run index`. +6. Rewrite `eval/gold.yaml` for your corpus — keep the refusals. ## Commands diff --git a/archive.config.ts b/archive.config.ts index 500020e..9c435df 100644 --- a/archive.config.ts +++ b/archive.config.ts @@ -23,6 +23,10 @@ export const config: ArchiveConfig = { // production this is chunked podcast transcripts; here, notebook entries. // Remove the line to run public-only. privateNotesDir: './example-content/notebook', + // Private books: one markdown file per book, cut into fragments on its + // chapter headings (or page markers), served as where a moment is or as a + // gist the author authorizes, never as text. See README "Make it yours". + // privateBooksDir: './example-content/books', embeddingModel: 'text-embedding-3-large', // Default is a widely available chat model. Swap for any Responses-API model // you have access to (e.g. a reasoning model — see src/answer.ts). diff --git a/docs/CONTRACT.md b/docs/CONTRACT.md index 574a412..571e822 100644 --- a/docs/CONTRACT.md +++ b/docs/CONTRACT.md @@ -847,9 +847,17 @@ yields an entity with one `text` fragment whose locator is `[{ scheme: 'whole', value: '' }]`; `fromPrivateNote` yields a private entity with one `locator` fragment whose linted label becomes the entity title and whose locator string becomes `[{ scheme: 'note', value }]`, and whose frontmatter may ask for an -`exposure`. The example content does not change. The demo gains one entity that -exercises `semantic` in a following release: a public-domain novel with its -chapter gists committed beside the vectors, which needs a keyed build. +`exposure`. `fromPrivateBook` yields a private entity with one fragment per +piece the corpus reader cut (`buildPrivateBooks`: `fragmentByHeadings` or +`fragmentByPageMarkers`, then `splitLong`), ids `${entityId}#${locatorKey}`, +each fragment's text its heading and its piece joined as a note's title and +body are, the frontmatter's `publicTitle` and `requireReview` on the entity's +`policy`, and the section 6 metadata lint run over the whole book before the +build drafts or embeds anything; `collectEntities` then groups the pairs, comparing +two descriptions of one entity by content with keys sorted at every level, not +by property order. The example content does not change. The demo gains one +entity that exercises `semantic` in a following release: a public-domain novel +with its chapter gists committed beside the vectors, which needs a keyed build. ## 13. What remains owned rather than guaranteed diff --git a/scripts/smoke-dist.mjs b/scripts/smoke-dist.mjs index 1207254..4a84a99 100644 --- a/scripts/smoke-dist.mjs +++ b/scripts/smoke-dist.mjs @@ -35,6 +35,8 @@ for (const name of [ 'answerQuestion', 'fromArchiveRecord', 'fromPrivateNote', + 'fromPrivateBook', + 'buildPrivateBooks', ]) { assert.equal(typeof root[name], 'function', `missing export: ${name}`); } diff --git a/src/adapters/teaching.ts b/src/adapters/teaching.ts index d3a36be..3c1c188 100644 --- a/src/adapters/teaching.ts +++ b/src/adapters/teaching.ts @@ -11,10 +11,10 @@ // the same two shapes reconstructed losslessly, so today's retrieval, prompt, // eval and demo code keep running on a schema-4 index without changes. -import type { Entity, Fragment } from '../contract.js'; +import type { Entity, EntityPolicy, Fragment } from '../contract.js'; import { locatorKey, renderLocatorLabel } from '../locator.js'; -import { assertPublicSafeField } from '../public-safe.js'; -import type { ArchiveRecord, PrivateNote } from '../types.js'; +import { assertPublicSafeField, assertPublicSafeMetadata } from '../public-safe.js'; +import type { ArchiveRecord, PrivateBook, PrivateNote } from '../types.js'; export const WHOLE_LOCATOR = { scheme: 'whole', value: '' } as const; @@ -96,6 +96,69 @@ export function fromPrivateNote(note: PrivateNote): { entity: Entity; fragment: return { entity, fragment }; } +/** + * A private book is one entity with many fragments (CONTRACT.md §8, §12): one + * per piece the corpus reader cut, ids `${entityId}#${locatorKey}` (book:x#ch12, + * book:x#ch12.s3, book:x#p184), each fragment's text its heading and its piece + * joined as a note's title and body are, so a chapter title is in the + * embedding and never leaves `text`. The entity default is the exposure the + * frontmatter asked for, carried on every fragment as a request until the + * build resolves it against a projection. The frontmatter's `publicTitle` and + * `requireReview` become the entity's policy. + * + * The strings a hit on the book would carry (title, version, creators, + * themes, locator labels) go through the metadata lint here, against the whole + * book, before anything is drafted or embedded: a quoting title fails in the + * reader's output, not after a paid build. The store runs the same lint again + * at every load (CONTRACT.md §12). + */ +export function fromPrivateBook(book: PrivateBook): { entity: Entity; fragment: Fragment }[] { + const disclosure = { raw: 'private', exposure: book.exposure ?? 'locator' } as const; + const policy: EntityPolicy = { + ...(book.publicTitle ? { publicTitle: true } : {}), + ...(book.requireReview ? { requireReview: true } : {}), + }; + const entity: Entity = { + id: book.id, + type: book.type, + title: book.title, + attribution: book.attribution, + ...(book.date ? { date: book.date } : {}), + ...(book.version ? { version: book.version } : {}), + url: book.url, + identifiers: book.identifiers, + ...(book.themes.length > 0 ? { themes: book.themes } : {}), + disclosure, + ...(Object.keys(policy).length > 0 ? { policy } : {}), + }; + if (book.pieces.length === 0) { + throw new Error(`private book '${book.id}' has no pieces; a book is at least one fragment.`); + } + const fragments: Fragment[] = []; + const keys = new Set(); + for (const piece of book.pieces) { + const key = locatorKey(piece.locator); + if (keys.has(key)) { + // Two chapters numbered the same, or two pages marked the same: the + // locator is the id, so the author fixes the source, named by locator. + throw new Error( + `private book '${book.id}': two pieces share the locator '${renderLocatorLabel(piece.locator)}'; ` + + `each chapter heading or page marker must be distinct.`, + ); + } + keys.add(key); + fragments.push({ + id: `${book.id}#${key}`, + entityId: book.id, + locator: piece.locator, + text: joinNoteText(piece.heading ?? '', piece.text), + disclosure, + }); + } + assertPublicSafeMetadata(entity, fragments, { path: `private book '${book.id}'` }); + return fragments.map((fragment) => ({ entity, fragment })); +} + /** * Reverse of fromPrivateNote. The private title is the text before the first * blank line and the body is what follows: a note's title is one frontmatter diff --git a/src/cli/build-index.ts b/src/cli/build-index.ts index 36792df..e1535c0 100644 --- a/src/cli/build-index.ts +++ b/src/cli/build-index.ts @@ -1,4 +1,5 @@ -// npm run index — read both layers, draft the gists the author asked for, +// npm run index — read the public collections, the private notes, and the +// private books; draft the gists the author asked for, // resolve each fragment's exposure, embed what changed, and write the private // index (artifacts/index.json) and the author's projections file // (artifacts/projections.json). Both are private material and gitignored. @@ -25,9 +26,9 @@ import { createHash } from 'node:crypto'; import OpenAI from 'openai'; import { config } from '../../archive.config.js'; -import { fromArchiveRecord, fromPrivateNote } from '../adapters/teaching.js'; +import { fromArchiveRecord, fromPrivateBook, fromPrivateNote } from '../adapters/teaching.js'; import type { Entity, EntityPolicy, Fragment, SemanticProjection } from '../contract.js'; -import { buildCorpus, buildPrivateNotes } from '../corpus.js'; +import { buildCorpus, buildPrivateBooks, buildPrivateNotes } from '../corpus.js'; import { embedStringFor } from '../embed-string.js'; import { batchInputs, embedBatch, truncateForEmbedding } from '../embedding.js'; import { collectEntities } from '../ingest/collect.js'; @@ -107,9 +108,15 @@ async function main(): Promise { ); } const notes = buildPrivateNotes(config); - console.log(`Corpus: ${records.length} records, ${notes.length} private notes`); - - const sources: Source[] = [...records.map(fromArchiveRecord), ...notes.map(fromPrivateNote)]; + const books = buildPrivateBooks(config); + console.log(`Corpus: ${records.length} records, ${notes.length} private notes, ${books.length} private books`); + + // A record or a note is one fragment; a book is one entity with many. + const sources: Source[] = [ + ...records.map(fromArchiveRecord), + ...notes.map(fromPrivateNote), + ...books.flatMap(fromPrivateBook), + ]; const lint = lintPolicy(config.gist); if (lint) for (const { entity } of sources) entity.policy = { ...entity.policy, lint }; // Entities once, every fragment under its entity (CONTRACT.md §12). The diff --git a/src/corpus.ts b/src/corpus.ts index a930140..5ff4af1 100644 --- a/src/corpus.ts +++ b/src/corpus.ts @@ -8,8 +8,11 @@ import matter from 'gray-matter'; import { readdirSync, readFileSync } from 'node:fs'; import { join, resolve } from 'node:path'; +import type { Attribution, Identifier } from './contract.js'; +import { fragmentByHeadings, fragmentByPageMarkers } from './ingest/fragment.js'; +import type { FragmentPiece } from './ingest/fragment.js'; import { assertPublicSafeField } from './public-safe.js'; -import type { ArchiveConfig, ArchiveRecord, CollectionConfig, PrivateNote } from './types.js'; +import type { ArchiveConfig, ArchiveRecord, CollectionConfig, PrivateBook, PrivateNote } from './types.js'; /** Reduce markdown to plain text for indexing (link text kept, syntax dropped). */ export function stripMarkdown(body: string): string { @@ -38,6 +41,8 @@ function asThemes(data: Record): string[] { function asDate(value: unknown): string { if (value instanceof Date) return value.toISOString().slice(0, 10); + // YAML reads `date: 1900` as a number; a year is a date at its precision. + if (typeof value === 'number' && Number.isInteger(value) && value > 0) return String(value); if (typeof value === 'string' && value.trim()) return value.trim(); return ''; } @@ -182,6 +187,138 @@ export function buildPrivateNotes(config: ArchiveConfig): PrivateNote[] { return notes.sort((a, b) => a.id.localeCompare(b.id)); } +const ID_TOKEN = /^[A-Za-z0-9_-]+$/; + +/** Frontmatter `authors`: a list of names, or of `{ name, role? }` entries. + * Malformed entries name the file and the field, never their value. */ +function asAttribution(value: unknown, path: string): Attribution[] { + if (value === undefined) return []; + if (!Array.isArray(value)) throw new Error(`${path}: 'authors' must be a list of names or of { name, role } entries.`); + return value.map((entry): Attribution => { + if (typeof entry === 'string' && entry.trim()) return { name: entry.trim(), role: 'author' }; + if (entry && typeof entry === 'object') { + const { name, role } = entry as { name?: unknown; role?: unknown }; + if (typeof name !== 'string' || !name.trim()) throw new Error(`${path}: an 'authors' entry needs a 'name'.`); + if (role !== undefined && typeof role !== 'string') throw new Error(`${path}: an 'authors' entry's 'role' must be a string.`); + return { name: name.trim(), role: role?.trim() || 'author' }; + } + throw new Error(`${path}: 'authors' must be a list of names or of { name, role } entries.`); + }); +} + +/** Frontmatter `identifiers`: a list of `{ scheme, value }` entries. */ +function asIdentifiers(value: unknown, path: string): Identifier[] { + if (value === undefined) return []; + if (!Array.isArray(value)) throw new Error(`${path}: 'identifiers' must be a list of { scheme, value } entries.`); + return value.map((entry): Identifier => { + const { scheme, value: v } = (entry ?? {}) as { scheme?: unknown; value?: unknown }; + if (typeof scheme !== 'string' || !scheme.trim() || (typeof v !== 'string' && typeof v !== 'number')) { + throw new Error(`${path}: 'identifiers' must be a list of { scheme, value } entries.`); + } + return { scheme: scheme.trim(), value: String(v).trim() }; + }); +} + +function asOptionalBoolean(value: unknown, field: string, path: string): boolean | undefined { + if (value === undefined) return undefined; + if (typeof value !== 'boolean') throw new Error(`${path}: '${field}' must be true or false.`); + return value; +} + +function asOptionalPositiveInteger(value: unknown, field: string, path: string): number | undefined { + if (value === undefined) return undefined; + if (typeof value !== 'number' || !Number.isInteger(value) || value <= 0) { + throw new Error(`${path}: '${field}' must be a positive whole number.`); + } + return value; +} + +/** A leading `# Title` that repeats the frontmatter title is the title page, + * not a chapter: dropped, so it neither becomes a fragment of its own nor + * collides with the first chapter's ordinal. */ +function withoutTitleHeading(body: string, title: string): string { + const m = /^\s*#\s+(.+?)\s*#*\s*(?:\r?\n|$)/.exec(body); + if (!m || m[1]!.trim().toLowerCase() !== title.toLowerCase()) return body; + return body.slice(m[0].length); +} + +/** Read the private books. One markdown file per book; the frontmatter is the + * book's public surface (`title`, `about`, and optionally `authors`, + * `identifiers`, `date`, `version`, `themes`, `exposure`, `publicTitle`, + * `requireReview`) plus how to cut it (`fragmentBy: headings | pages`, + * `maxFragmentChars`, `firstPage`); the body is the private text, split + * here into pieces whose locators are structural (docs/CONTRACT.md §8). + * The pieces' text is embedded, never quoted; what the traveling strings may + * say about it is checked by the adapter (src/adapters/teaching.ts) before + * anything is drafted or embedded. */ +export function buildPrivateBooks(config: ArchiveConfig): PrivateBook[] { + if (!config.privateBooksDir) return []; + const dir = resolve(config.privateBooksDir); + const books: PrivateBook[] = []; + for (const { slug, path, data, content, title } of readMarkdownDir(dir, 'private books')) { + const url = firstString(data.about); + if (!url) { + throw new Error(`${path} needs 'about' (the public URL a hit on this book routes to) in its frontmatter.`); + } + const type = firstString(data.type) || 'book'; + if (!ID_TOKEN.test(type)) { + throw new Error(`${path}: 'type' must be a short token (letters, digits, '-' or '_'): it opens the entity id, as 'book' does.`); + } + const exposure = data.exposure; + if (exposure !== undefined && exposure !== 'semantic' && exposure !== 'locator' && exposure !== 'none') { + // Named, not echoed: authored frontmatter on a private book (STANDARDS §4). + throw new Error( + `${path}: 'exposure' must be semantic, locator, or none. ` + + `Private text is never exposed as text; see docs/CONTRACT.md §3.`, + ); + } + const fragmentBy = data.fragmentBy ?? 'headings'; + if (fragmentBy !== 'headings' && fragmentBy !== 'pages') { + throw new Error(`${path}: 'fragmentBy' must be headings (the default) or pages.`); + } + const maxFragmentChars = asOptionalPositiveInteger(data.maxFragmentChars, 'maxFragmentChars', path); + const firstPage = asOptionalPositiveInteger(data.firstPage, 'firstPage', path); + const publicTitle = asOptionalBoolean(data.publicTitle, 'publicTitle', path); + const requireReview = asOptionalBoolean(data.requireReview, 'requireReview', path); + const version = firstString(data.version); + const date = asDate(data.date); + + const body = withoutTitleHeading(content, title); + const options = { ...(maxFragmentChars !== undefined ? { maxFragmentChars } : {}) }; + const cut = + fragmentBy === 'pages' + ? fragmentByPageMarkers(body, { ...options, ...(firstPage !== undefined ? { firstPage } : {}) }) + : fragmentByHeadings(body, options); + const pieces: FragmentPiece[] = []; + for (const piece of cut) { + const text = stripMarkdown(piece.text); + if (!text) continue; + pieces.push({ locator: piece.locator, text, ...(piece.heading !== undefined ? { heading: piece.heading } : {}) }); + } + if (pieces.length === 0) { + throw new Error(`${path} has no text after its frontmatter; a private book needs a body to fragment.`); + } + + books.push({ + id: `${type}:${slug}`, + type, + slug, + title, + ...(publicTitle !== undefined ? { publicTitle } : {}), + url, + attribution: asAttribution(data.authors ?? data.creators, path), + identifiers: asIdentifiers(data.identifiers, path), + ...(date ? { date } : {}), + ...(version ? { version } : {}), + themes: asThemes(data), + ...(exposure !== undefined ? { exposure } : {}), + ...(requireReview !== undefined ? { requireReview } : {}), + pieces, + }); + } + return books.sort((a, b) => a.id.localeCompare(b.id)); +} + /** The text the embedding model sees. Themes are included so topic tags * improve retrieval, not just the answer prompt. */ export function embedText(record: ArchiveRecord): string { diff --git a/src/ingest/collect.ts b/src/ingest/collect.ts index 7f8da78..652c2a7 100644 --- a/src/ingest/collect.ts +++ b/src/ingest/collect.ts @@ -9,6 +9,18 @@ import type { Entity, Fragment } from '../contract.js'; +/** JSON with object keys sorted at every level, so two descriptions of one + * entity compare by content and not by the order an adapter happened to + * assign properties in. Arrays keep their order: two creators listed the + * other way round are two descriptions. */ +function canonical(value: unknown): string { + return JSON.stringify(value, (_key, v: unknown) => { + if (v === null || typeof v !== 'object' || Array.isArray(v)) return v; + const record = v as Record; + return Object.fromEntries(Object.keys(record).sort().map((k) => [k, record[k]])); + }); +} + export interface CollectedEntities { /** Entities by id, in first-seen order. */ entities: Map; @@ -24,7 +36,7 @@ export function collectEntities(sources: readonly { entity: Entity; fragment: Fr const seen = entities.get(entity.id); if (seen === undefined) { entities.set(entity.id, entity); - } else if (JSON.stringify(seen) !== JSON.stringify(entity)) { + } else if (canonical(seen) !== canonical(entity)) { throw new Error( `entity '${entity.id}' is described two ways by its sources; an id names one entity across collections, notes, and fragments.`, ); diff --git a/src/types.ts b/src/types.ts index ad3fee2..60c0503 100644 --- a/src/types.ts +++ b/src/types.ts @@ -2,6 +2,8 @@ // AnswerEvidence: RoutingHint has no field for private text, which makes // the privacy boundary a compile-time constraint rather than a review note. +import type { Attribution, Identifier } from './contract.js'; +import type { FragmentPiece } from './ingest/fragment.js'; import type { LintedGist } from './public-safe.js'; /** One piece of the PUBLIC archive — quotable, citable, body travels. */ @@ -64,6 +66,51 @@ export interface PrivateNote { exposure?: 'semantic' | 'locator' | 'none'; } +/** + * One book of the PRIVATE layer: a manuscript, a thesis, a novel the archive + * holds but may not quote (docs/CONTRACT.md §8). Unlike a note, it is one + * entity with many fragments: the markdown body is split on chapter headings + * (or page markers) at read time, and each piece becomes a fragment whose + * locator is structural ("ch. 12", "p. 184"), never authored prose. The text + * is embedded so retrieval can find the moment; what travels about it is the + * entity's exposure: where it is (`locator`, the default), a gist the author + * authorized (`semantic`), or nothing (`none`). + */ +export interface PrivateBook { + /** Stable id: `${type}:${slug}`; `type` is frontmatter `type`, default `book`. */ + id: string; + type: string; + slug: string; + /** Frontmatter `title`. It travels on every hit as the entity title, so it + * is checked against the book's text at build unless `publicTitle` says the + * title is public by construction (a published book's is). */ + title: string; + /** Frontmatter `publicTitle: true`: skip the quote check on the title (the + * one-line bound still holds). The author's declaration (CONTRACT.md §6). */ + publicTitle?: boolean; + /** The PUBLIC page a citation routes the reader to (frontmatter `about`). */ + url: string; + /** Frontmatter `authors`: names, or `{ name, role }` entries. */ + attribution: Attribution[]; + /** Frontmatter `identifiers`: `{ scheme, value }` entries (isbn, doi, ...). */ + identifiers: Identifier[]; + date?: string; + /** Frontmatter `version`: "manuscript", an edition, a draft number. */ + version?: string; + themes: string[]; + /** Frontmatter `exposure`; absent means `locator`. `text` is not an option + * for private material (CONTRACT.md §3). */ + exposure?: 'semantic' | 'locator' | 'none'; + /** Frontmatter `requireReview: true`: a gist is servable only once the + * author marks it reviewed in artifacts/projections.json. */ + requireReview?: boolean; + /** The pieces in source order, as the corpus reader cut them (a structural + * locator, the private text with markdown stripped, and the heading that + * opened the piece, kept apart so the adapter can put it in the embedding + * and nowhere else); never empty. */ + pieces: FragmentPiece[]; +} + /** * A hint: a private (or non-quotable) hit reduced to its routing surface for * the in-package synthesis consumer (src/evidence.ts). Deliberately has NO @@ -141,6 +188,12 @@ export interface ArchiveConfig { /** Directory of private notes (frontmatter: title, about, locator; body = * private text). Omit it and the engine runs public-only. */ privateNotesDir?: string; + /** Directory of private books, one markdown file per book (frontmatter: + * title, about, and optionally authors, identifiers, date, version, themes, + * exposure, publicTitle, requireReview, fragmentBy, maxFragmentChars, + * firstPage; body = the private text, split into fragments on its chapter + * headings or page markers). Omit it and no book is indexed. */ + privateBooksDir?: string; /** OpenAI embedding model. Changing it re-embeds everything on next index run. */ embeddingModel: string; /** OpenAI model that writes the answer. */ diff --git a/test/books.test.ts b/test/books.test.ts new file mode 100644 index 0000000..a1c3d36 --- /dev/null +++ b/test/books.test.ts @@ -0,0 +1,282 @@ +// Offline tests for private books: the corpus reader cuts one markdown file +// into pieces, the adapter makes one entity with many fragments, and the +// build-time lint runs on the strings a hit would carry before anything is +// drafted or embedded. The fixture is an invented novella written for this +// file. No API key, no network. + +import assert from 'node:assert/strict'; +import { mkdtempSync, writeFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { test } from 'node:test'; + +import { config } from '../archive.config.js'; +import { fromPrivateBook } from '../src/adapters/teaching.js'; +import type { Entity } from '../src/contract.js'; +import { buildPrivateBooks } from '../src/corpus.js'; +import { embedStringFor } from '../src/embed-string.js'; +import { collectEntities } from '../src/ingest/collect.js'; +import { resolveDisclosure } from '../src/ingest/disclosure.js'; +import { project } from '../src/no-leak.js'; +import { INDEX_SCHEMA_VERSION, readIndex, toServedIndex, writeIndex } from '../src/store.js'; +import type { FragmentEntry } from '../src/store.js'; + +const CHAPTER_ONE_A = + 'Mara kept the harbour lantern ledger in a tin box under the stairs, and every entry in it was a lie she could account for.'; +const CHAPTER_ONE_B = 'The box had belonged to her mother, who had kept a different kind of ledger.'; +/** Two paragraphs in the file; one whitespace-collapsed text once read. */ +const CHAPTER_ONE = `${CHAPTER_ONE_A} ${CHAPTER_ONE_B}`; +const CHAPTER_TWO = + 'The customs officer arrived on a Tuesday with a list of names and no appetite for breakfast. ' + + 'He read the list aloud twice, and the second time he left out her brother.'; +const INTERLUDE = 'A page torn from the ledger, folded twice, the ink still legible on one side.'; + +function front(extra: string[] = []): string { + return [ + '---', + 'title: The Lantern Ledger', + 'about: https://example.com/books/the-lantern-ledger/', + 'authors:', + ' - name: Person A', + ' role: author', + ' - Person B', + 'version: manuscript', + 'date: 2026-03', + 'identifiers:', + ' - scheme: isbn', + ' value: "978-0-00-000000-0"', + 'themes: [ledgers, harbours]', + 'exposure: semantic', + ...extra, + '---', + ].join('\n'); +} +const FRONT = front(); + +const BODY = [ + '# The Lantern Ledger', + '', + '## Chapter 1: The Tin Box', + '', + CHAPTER_ONE_A, + '', + CHAPTER_ONE_B, + '', + '## Chapter 2', + '', + `*${CHAPTER_TWO}*`, + '', + '## Interlude', + '', + INTERLUDE, + '', +].join('\n'); + +function bookDir(files: Record): string { + const dir = mkdtempSync(join(tmpdir(), 'ae-books-')); + for (const [name, content] of Object.entries(files)) writeFileSync(join(dir, name), content, 'utf8'); + return dir; +} + +function read(files: Record) { + return buildPrivateBooks({ ...config, privateBooksDir: bookDir(files) }); +} + +test('books: no privateBooksDir means no books; a missing directory fails loudly', () => { + assert.equal(config.privateBooksDir, undefined, 'the shipped config has no books'); + assert.deepEqual(buildPrivateBooks(config), []); + assert.throws( + () => buildPrivateBooks({ ...config, privateBooksDir: join(tmpdir(), 'ae-books-does-not-exist') }), + /cannot read private books at .*ae-books-does-not-exist/, + ); +}); + +test('books: the reader cuts one file on its headings; the title heading is the title page, not a chapter', () => { + const [book] = read({ 'the-lantern-ledger.md': `${FRONT}\n${BODY}` }); + assert.ok(book); + assert.equal(book.id, 'book:the-lantern-ledger'); + assert.equal(book.type, 'book'); + assert.equal(book.title, 'The Lantern Ledger'); + assert.equal(book.url, 'https://example.com/books/the-lantern-ledger/'); + assert.deepEqual(book.attribution, [ + { name: 'Person A', role: 'author' }, + { name: 'Person B', role: 'author' }, + ]); + assert.deepEqual(book.identifiers, [{ scheme: 'isbn', value: '978-0-00-000000-0' }]); + assert.equal(book.version, 'manuscript'); + assert.equal(book.date, '2026-03'); + assert.deepEqual(book.themes, ['ledgers', 'harbours']); + assert.equal(book.exposure, 'semantic'); + // Three pieces: two numbered chapters and one unnumbered heading counted + // among them. The `# The Lantern Ledger` line is not a fourth. + assert.deepEqual( + book.pieces.map((p) => [p.locator, p.heading]), + [ + [[{ scheme: 'chapter', value: '1' }], 'Chapter 1: The Tin Box'], + [[{ scheme: 'chapter', value: '2' }], 'Chapter 2'], + [[{ scheme: 'chapter', value: '3' }], 'Interlude'], + ], + ); + // Markdown is stripped from the text (the emphasis around chapter 2), and + // the paragraph break inside chapter 1 collapses as a note's body does. + assert.equal(book.pieces[0]!.text, CHAPTER_ONE); + assert.equal(book.pieces[1]!.text, CHAPTER_TWO); + assert.equal(book.pieces[2]!.text, INTERLUDE); +}); + +test('books: maxFragmentChars sub-splits a chapter; fragmentBy: pages reads page markers', () => { + // Chapter 1 is 200 characters of markdown in two paragraphs; chapter 2 is + // 169 in one. At 180, only chapter 1 is cut, on its paragraph boundary. + const [split] = read({ 'ledger.md': `${front(['maxFragmentChars: 180'])}\n${BODY}` }); + assert.ok(split); + assert.equal(split.pieces[0]!.text, CHAPTER_ONE_A); + assert.equal(split.pieces[1]!.text, CHAPTER_ONE_B); + assert.deepEqual( + split.pieces.map((p) => p.locator.map((l) => `${l.scheme}:${l.value}`).join('.')), + ['chapter:1.section:1', 'chapter:1.section:2', 'chapter:2', 'chapter:3'], + ); + + const paged = ['---', 'title: Typeset', 'about: https://example.com/t/', 'fragmentBy: pages', '---', '<<>>', 'First page.', '<<>>', 'Second page.', ''].join('\n'); + const [typeset] = read({ 'typeset.md': paged }); + assert.ok(typeset); + assert.equal(typeset.exposure, undefined); + assert.deepEqual( + typeset.pieces.map((p) => [p.locator, p.text]), + [ + [[{ scheme: 'page', value: '184' }], 'First page.'], + [[{ scheme: 'page', value: '185' }], 'Second page.'], + ], + ); + + const formFeed = ['---', 'title: Typeset', 'about: https://example.com/t/', 'fragmentBy: pages', 'firstPage: 40', '---', 'Page forty.\fPage forty-one.'].join('\n'); + const [ff] = read({ 'ff.md': formFeed }); + assert.deepEqual(ff!.pieces.map((p) => p.locator[0]!.value), ['40', '41']); + + // A bare year is a date at year precision, not a dropped field. + const [dated] = read({ 'd.md': '---\ntitle: D\nabout: https://example.com/d/\ndate: 1900\n---\n## Chapter 1\n\ntext\n' }); + assert.equal(dated!.date, '1900'); +}); + +test('books: malformed frontmatter names the file and the field, never a value', () => { + const file = (extra: string, body = '## Chapter 1\n\ntext\n') => + `---\ntitle: T\nabout: https://example.com/t/\n${extra}\n---\n${body}`; + assert.throws(() => read({ 'b.md': '---\ntitle: T\n---\n## Chapter 1\n\ntext\n' }), /b\.md needs 'about'/); + assert.throws( + () => read({ 'b.md': file('exposure: text or the first stanza') }), + (err: unknown) => + err instanceof Error && /b\.md: 'exposure' must be semantic, locator, or none\./.test(err.message) && !/stanza/.test(err.message), + ); + assert.throws(() => read({ 'b.md': file('fragmentBy: paragraphs') }), /b\.md: 'fragmentBy' must be headings/); + assert.throws(() => read({ 'b.md': file('maxFragmentChars: lots') }), /'maxFragmentChars' must be a positive whole number/); + assert.throws(() => read({ 'b.md': file('firstPage: 0') }), /'firstPage' must be a positive whole number/); + assert.throws(() => read({ 'b.md': file('publicTitle: yes please') }), /'publicTitle' must be true or false/); + assert.throws(() => read({ 'b.md': file('requireReview: 1') }), /'requireReview' must be true or false/); + assert.throws(() => read({ 'b.md': file('authors: Person A') }), /'authors' must be a list/); + assert.throws(() => read({ 'b.md': file('authors:\n - role: author') }), /an 'authors' entry needs a 'name'/); + assert.throws(() => read({ 'b.md': file('authors:\n - name: A\n role: 3') }), /'role' must be a string/); + assert.throws(() => read({ 'b.md': file('identifiers:\n - isbn') }), /'identifiers' must be a list of \{ scheme, value \}/); + assert.throws(() => read({ 'b.md': file('type: "a:b"') }), /'type' must be a short token/); + assert.throws(() => read({ 'b.md': file('', '\n\n') }), /b\.md has no text after its frontmatter/); + // A draft is skipped, as in every layer. + assert.deepEqual(read({ 'b.md': file('draft: true') }), []); +}); + +test('books: the adapter yields one entity with many fragments, and the metadata lint runs first', () => { + const [book] = read({ 'the-lantern-ledger.md': `${FRONT}\n${BODY}` }); + const pairs = fromPrivateBook(book!); + assert.equal(pairs.length, 3); + const entity = pairs[0]!.entity; + assert.ok(pairs.every((p) => p.entity === entity), 'every pair carries the one entity'); + assert.deepEqual(entity, { + id: 'book:the-lantern-ledger', + type: 'book', + title: 'The Lantern Ledger', + attribution: [ + { name: 'Person A', role: 'author' }, + { name: 'Person B', role: 'author' }, + ], + date: '2026-03', + version: 'manuscript', + url: 'https://example.com/books/the-lantern-ledger/', + identifiers: [{ scheme: 'isbn', value: '978-0-00-000000-0' }], + themes: ['ledgers', 'harbours'], + disclosure: { raw: 'private', exposure: 'semantic' }, + } satisfies Entity); + assert.deepEqual( + pairs.map((p) => p.fragment.id), + ['book:the-lantern-ledger#ch1', 'book:the-lantern-ledger#ch2', 'book:the-lantern-ledger#ch3'], + ); + // The heading is in the text, joined as a note's title is, and the + // embedding is the text alone: a private fragment embeds nothing else. + const first = pairs[0]!.fragment; + assert.equal(first.text, `Chapter 1: The Tin Box\n\n${CHAPTER_ONE}`); + assert.equal(embedStringFor(first, entity), first.text); + assert.deepEqual(first.locator, [{ scheme: 'chapter', value: '1' }]); + // Every fragment carries the request; the build resolves it. Without a + // projection, `semantic` resolves to `locator`. + for (const { fragment } of pairs) { + assert.deepEqual(fragment.disclosure, { raw: 'private', exposure: 'semantic' }); + assert.deepEqual(resolveDisclosure(entity, { exposure: fragment.disclosure.exposure }), { raw: 'private', exposure: 'locator' }); + } + // collectEntities groups the pairs under the one entity. + const grouped = collectEntities(pairs); + assert.deepEqual([...grouped.entities.keys()], ['book:the-lantern-ledger']); + assert.equal(grouped.fragments.get('book:the-lantern-ledger')!.length, 3); + + // The policy travels from the frontmatter. + const [reviewed] = read({ 'r.md': `---\ntitle: R\nabout: https://example.com/r/\nrequireReview: true\npublicTitle: true\n---\n## Chapter 1\n\ntext\n` }); + assert.deepEqual(fromPrivateBook(reviewed!)[0]!.entity.policy, { publicTitle: true, requireReview: true }); + assert.equal('policy' in entity, false); +}); + +test('books: a title that quotes the text is refused before any build, unless the author declares it public', () => { + const title = 'What the harbour keeps at night'; + const body = `## Chapter 1\n\nThe words what the harbour keeps at night were painted on the lid of the box.\n`; + const front = (extra = '') => `---\ntitle: ${title}\nabout: https://example.com/w/\n${extra}---\n${body}`; + const [quoting] = read({ 'w.md': front() }); + assert.throws( + () => fromPrivateBook(quoting!), + (err: unknown) => err instanceof Error && /private book 'book:w': entity 'book:w': 'title' quotes private text/.test(err.message) && !/painted/.test(err.message), + ); + const [declared] = read({ 'w.md': front('publicTitle: true\n') }); + assert.equal(fromPrivateBook(declared!).length, 1); + // A creator's name is checked the same way. + const [creator] = read({ 'w.md': front('publicTitle: true\nauthors: ["the harbour keeps at night, said"]\n') }); + assert.throws(() => fromPrivateBook(creator!), /'creator name' quotes private text/); + // Two chapters numbered the same would share an id; the reader's output is refused by locator. + const [twice] = read({ 'w.md': `---\ntitle: W\nabout: https://example.com/w/\n---\n## Chapter 3\n\none\n\n## Chapter 3\n\ntwo\n` }); + assert.throws(() => fromPrivateBook(twice!), /private book 'book:w': two pieces share the locator 'ch. 3'/); +}); + +test('books: through the store, a book serves locators and never its text', () => { + const [book] = read({ 'the-lantern-ledger.md': `${FRONT}\n${BODY}`.replace('exposure: semantic', 'exposure: locator') }); + const pairs = fromPrivateBook(book!); + const { entities, fragments } = collectEntities(pairs); + const entries: FragmentEntry[] = fragments.get(book!.id)!.map((fragment, i) => ({ + model: 'test-embedding', + dimensions: 2, + vector: [1, i], + contentHash: `hash-${i}`, + fragment: { ...fragment, disclosure: resolveDisclosure(entities.get(book!.id)!, { exposure: fragment.disclosure.exposure }) }, + })); + const path = join(mkdtempSync(join(tmpdir(), 'ae-books-index-')), 'index.json'); + writeIndex({ version: INDEX_SCHEMA_VERSION, entities: [...entities.values()], entries }, path); + const loaded = readIndex(path); + assert.equal(loaded.entries.length, 3); + const served = toServedIndex(loaded); + const serialized = JSON.stringify(served); + for (const phrase of ['tin box under the stairs', 'no appetite for breakfast', 'folded twice', 'The Tin Box']) { + assert.ok(!serialized.includes(phrase), `served index must not carry: ${phrase}`); + } + assert.ok(served.entries.every((e) => e.fragment.text === '' && e.fragment.projection === undefined)); + // And a hit on it carries where, not what. + const entry = loaded.entries[1]!; + const hit = project({ fragment: entry.fragment, entity: entities.get(book!.id)!, cosine: 0.87, score: 0.87, breakdown: { cosine: 0.87 } }); + assert.equal(hit.exposure, 'locator'); + assert.equal(hit.locatorLabel, 'ch. 2'); + assert.equal(hit.entity.title, 'The Lantern Ledger'); + assert.equal(hit.entity.url, 'https://example.com/books/the-lantern-ledger/'); + assert.deepEqual(hit.entity.identifiers, [{ scheme: 'isbn', value: '978-0-00-000000-0' }]); + assert.equal(hit.score, 0.85); + assert.ok(!JSON.stringify(hit).includes('breakfast')); +}); diff --git a/test/contract.test.ts b/test/contract.test.ts index da93ae0..4fa4e8a 100644 --- a/test/contract.test.ts +++ b/test/contract.test.ts @@ -488,3 +488,60 @@ test('collectEntities: entities once, many fragments; an id described two ways o /fragment 'book:x#p1' names entity 'book:x' but was produced under 'essay:y'/, ); }); + +test('collectEntities: one entity in two property orders is one entity; array order is content', () => { + const page = (n: number): Fragment => ({ + id: `book:x#p${n}`, + entityId: 'book:x', + locator: [{ scheme: 'page', value: String(n) }], + text: `page ${n}`, + disclosure: { raw: 'private', exposure: 'locator' }, + }); + // The same description, assigned in another order at every level (the + // entity, its policy, the lint window, a creator, an identifier): two + // adapters, or one adapter run twice, may well differ in insertion order. + const first: Entity = { + id: 'book:x', + type: 'book', + title: 'X', + attribution: [{ name: 'A. Author', role: 'author', identifiers: [{ scheme: 'orcid', value: '0000' }] }], + url: 'https://example.com/x/', + identifiers: [{ scheme: 'isbn', value: '1' }], + disclosure: { raw: 'private', exposure: 'locator' }, + policy: { requireReview: true, lint: { ngramWords: 5, gistMaxChars: 300 } }, + }; + const second: Entity = { + policy: { lint: { gistMaxChars: 300, ngramWords: 5 }, requireReview: true }, + disclosure: { exposure: 'locator', raw: 'private' }, + identifiers: [{ value: '1', scheme: 'isbn' }], + url: 'https://example.com/x/', + attribution: [{ identifiers: [{ value: '0000', scheme: 'orcid' }], role: 'author', name: 'A. Author' }], + title: 'X', + type: 'book', + id: 'book:x', + }; + assert.notEqual(JSON.stringify(first), JSON.stringify(second), 'the fixture must differ in order'); + const grouped = collectEntities([ + { entity: first, fragment: page(1) }, + { entity: second, fragment: page(2) }, + ]); + assert.equal(grouped.entities.get('book:x'), first, 'the first description is the one kept'); + assert.deepEqual(grouped.fragments.get('book:x')!.map((f) => f.id), ['book:x#p1', 'book:x#p2']); + + // Order inside a list is content: creators listed the other way round, or + // identifiers, describe the entity differently and are still refused. + const creators = (names: string[]): Entity => ({ ...first, attribution: names.map((name) => ({ name, role: 'author' })) }); + assert.throws( + () => collectEntities([{ entity: creators(['A', 'B']), fragment: page(1) }, { entity: creators(['B', 'A']), fragment: page(2) }]), + /entity 'book:x' is described two ways/, + ); + // And a real difference under a reordered key is still a difference. + assert.throws( + () => + collectEntities([ + { entity: first, fragment: page(1) }, + { entity: { ...second, policy: { lint: { gistMaxChars: 300, ngramWords: 4 }, requireReview: true } }, fragment: page(2) }, + ]), + /entity 'book:x' is described two ways/, + ); +}); From b57167d3867861e5dd66abaee19adcf23b8ba136 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 7 Oct 2026 07:25:28 +0000 Subject: [PATCH 2/5] Name positions and scheme, not the value, when a book repeats a locator A chapter value is a parsed number or an ordinal, but a page value is the author's own marker token, so the duplicate-locator refusal echoed an authored value. It now names the two positions and the locator scheme; the test asserts the token never reaches the message. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01F2ti456guy1bXVmjA3fBTV --- CHANGELOG.md | 2 +- src/adapters/teaching.ts | 19 ++++++++++++------- test/books.test.ts | 14 ++++++++++++-- 3 files changed, 25 insertions(+), 10 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 050dcce..97e93cb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -196,7 +196,7 @@ npm trusts this repository's `release.yml`. heading inside the fragment's text and nowhere else, `publicTitle` and `requireReview` on the entity's policy, and the metadata lint run over the whole book before `npm run index` drafts or embeds anything; two pieces that - share a locator are refused by locator. `npm run index` reads the books + share a locator are refused by position and scheme. `npm run index` reads the books beside the records and the notes; a book's `semantic` request is drafted per fragment against the whole book's text, as CONTRACT §12 describes. diff --git a/src/adapters/teaching.ts b/src/adapters/teaching.ts index 3c1c188..f850e9f 100644 --- a/src/adapters/teaching.ts +++ b/src/adapters/teaching.ts @@ -135,18 +135,23 @@ export function fromPrivateBook(book: PrivateBook): { entity: Entity; fragment: throw new Error(`private book '${book.id}' has no pieces; a book is at least one fragment.`); } const fragments: Fragment[] = []; - const keys = new Set(); - for (const piece of book.pieces) { + const keys = new Map(); + for (const [index, piece] of book.pieces.entries()) { const key = locatorKey(piece.locator); - if (keys.has(key)) { + const first = keys.get(key); + if (first !== undefined) { // Two chapters numbered the same, or two pages marked the same: the - // locator is the id, so the author fixes the source, named by locator. + // locator is the id, so the author fixes the source. Named by position + // and scheme, never by value: a chapter value is a parsed number or an + // ordinal, but a page value is the author's own marker token, and this + // message is what `npm run index` prints (STANDARDS §4). + const schemes = piece.locator.map((l) => l.scheme).join('.'); throw new Error( - `private book '${book.id}': two pieces share the locator '${renderLocatorLabel(piece.locator)}'; ` + - `each chapter heading or page marker must be distinct.`, + `private book '${book.id}': pieces ${first + 1} and ${index + 1} of ${book.pieces.length} share a ` + + `'${schemes}' locator; each chapter heading or page marker must be distinct.`, ); } - keys.add(key); + keys.set(key, index); fragments.push({ id: `${book.id}#${key}`, entityId: book.id, diff --git a/test/books.test.ts b/test/books.test.ts index a1c3d36..222bbd0 100644 --- a/test/books.test.ts +++ b/test/books.test.ts @@ -243,9 +243,19 @@ test('books: a title that quotes the text is refused before any build, unless th // A creator's name is checked the same way. const [creator] = read({ 'w.md': front('publicTitle: true\nauthors: ["the harbour keeps at night, said"]\n') }); assert.throws(() => fromPrivateBook(creator!), /'creator name' quotes private text/); - // Two chapters numbered the same would share an id; the reader's output is refused by locator. + // Two chapters numbered the same would share an id; the reader's output is + // refused by position and scheme. A page marker's token is the author's own + // text, so the message never carries a locator value. const [twice] = read({ 'w.md': `---\ntitle: W\nabout: https://example.com/w/\n---\n## Chapter 3\n\none\n\n## Chapter 3\n\ntwo\n` }); - assert.throws(() => fromPrivateBook(twice!), /private book 'book:w': two pieces share the locator 'ch. 3'/); + assert.throws(() => fromPrivateBook(twice!), /private book 'book:w': pieces 1 and 2 of 2 share a 'chapter' locator/); + const [marked] = read({ + 'w.md': `---\ntitle: W\nabout: https://example.com/w/\nfragmentBy: pages\n---\n<<>>\none\n<<>>\ntwo\n`, + }); + assert.throws( + () => fromPrivateBook(marked!), + (err: unknown) => + err instanceof Error && /pieces 1 and 2 of 2 share a 'page' locator/.test(err.message) && !/unguessable/.test(err.message), + ); }); test('books: through the store, a book serves locators and never its text', () => { From cb6a066cc7c7d507b4505d3c79b634e85cb69c21 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 7 Oct 2026 07:44:48 +0000 Subject: [PATCH 3/5] Say why a book fragment's id is safe to print, and pin it The id carries the locator values; a locator is public surface by contract and every value goes through the metadata lint before the id goes anywhere. A comment at the id states that, and the test shows a page marker's token refused by the lint, named by field and never echoed. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01F2ti456guy1bXVmjA3fBTV --- src/adapters/teaching.ts | 6 ++++++ test/books.test.ts | 13 +++++++++++++ 2 files changed, 19 insertions(+) diff --git a/src/adapters/teaching.ts b/src/adapters/teaching.ts index f850e9f..1b4a68e 100644 --- a/src/adapters/teaching.ts +++ b/src/adapters/teaching.ts @@ -152,6 +152,12 @@ export function fromPrivateBook(book: PrivateBook): { entity: Entity; fragment: ); } keys.set(key, index); + // The key carries the locator values (ch12, p184, p184.s2). A locator is + // public surface by contract (CONTRACT.md §1: structural, on every hit as + // the label), and the metadata lint below checks every value and label + // against the whole book before this id goes anywhere, so an id is safe to + // print by the same construction as the hit's locatorLabel. A page + // marker's token is one such value: bounded and linted, not trusted. fragments.push({ id: `${book.id}#${key}`, entityId: book.id, diff --git a/test/books.test.ts b/test/books.test.ts index 222bbd0..e174c40 100644 --- a/test/books.test.ts +++ b/test/books.test.ts @@ -256,6 +256,19 @@ test('books: a title that quotes the text is refused before any build, unless th (err: unknown) => err instanceof Error && /pieces 1 and 2 of 2 share a 'page' locator/.test(err.message) && !/unguessable/.test(err.message), ); + // A page marker's token becomes a locator value and so part of the fragment + // id. It is public surface by contract and goes through the metadata lint + // before the id goes anywhere: a token the lint bounds out is refused here, + // named by field and never echoed. + const oversized = 'x'.repeat(130); + const [bounded] = read({ 'w.md': `---\ntitle: W\nabout: https://example.com/w/\nfragmentBy: pages\n---\n<<>>\none\n` }); + assert.equal(bounded!.pieces[0]!.locator[0]!.value, oversized); + assert.throws( + () => fromPrivateBook(bounded!), + (err: unknown) => err instanceof Error && /'locator value \(page\)'/.test(err.message) && !/xxxxxxxx/.test(err.message), + ); + const [plain] = read({ 'w.md': `---\ntitle: W\nabout: https://example.com/w/\nfragmentBy: pages\n---\n<<>>\none\n` }); + assert.equal(fromPrivateBook(plain!)[0]!.fragment.id, 'book:w#p184'); }); test('books: through the store, a book serves locators and never its text', () => { From 963c78462c028a0034690d967d2fbb4a30158c89 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 7 Oct 2026 07:45:57 +0000 Subject: [PATCH 4/5] Bound a book's page markers to short labels, so ids are structural by construction A page marker's token becomes a locator value and part of the fragment id, both printed by every refusal that names a fragment. The reader now refuses a marker that is not a short label of letters, digits, '.' and '-', by piece position and never echoed. The earlier push's assertion was wrong for the same reason it was worth having: the lint's own message names the id. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01F2ti456guy1bXVmjA3fBTV --- CHANGELOG.md | 4 ++++ src/adapters/teaching.ts | 3 ++- src/corpus.ts | 16 +++++++++++++++- test/books.test.ts | 31 +++++++++++++++++++------------ 4 files changed, 40 insertions(+), 14 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 97e93cb..98a085a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -213,6 +213,10 @@ npm trusts this repository's `release.yml`. "described two ways". It now compares by content, with object keys sorted at every level; the order inside a list (creators, identifiers) is still content. A regression test pins both. +- A private book's page marker (`<<>>`) is bounded by the reader to a + short label of letters, digits, `.` and `-`, so a page locator value, and + the fragment id built from it, is structural by construction; a marker + outside that is refused by piece position, never echoed. - A bare year in frontmatter (`date: 1900`, which YAML reads as a number) was dropped silently by every reader; it is now the date at year precision. diff --git a/src/adapters/teaching.ts b/src/adapters/teaching.ts index 1b4a68e..d3ba258 100644 --- a/src/adapters/teaching.ts +++ b/src/adapters/teaching.ts @@ -157,7 +157,8 @@ export function fromPrivateBook(book: PrivateBook): { entity: Entity; fragment: // the label), and the metadata lint below checks every value and label // against the whole book before this id goes anywhere, so an id is safe to // print by the same construction as the hit's locatorLabel. A page - // marker's token is one such value: bounded and linted, not trusted. + // marker's token is one such value, bounded to a short label by the + // reader (src/corpus.ts) before it reaches this id. fragments.push({ id: `${book.id}#${key}`, entityId: book.id, diff --git a/src/corpus.ts b/src/corpus.ts index 5ff4af1..c69c89c 100644 --- a/src/corpus.ts +++ b/src/corpus.ts @@ -188,6 +188,8 @@ export function buildPrivateNotes(config: ArchiveConfig): PrivateNote[] { } const ID_TOKEN = /^[A-Za-z0-9_-]+$/; +const PAGE_LABEL_MAX = 32; +const PAGE_LABEL = /^[\p{L}\p{N}.-]{1,32}$/u; /** Frontmatter `authors`: a list of names, or of `{ name, role? }` entries. * Malformed entries name the file and the field, never their value. */ @@ -290,7 +292,19 @@ export function buildPrivateBooks(config: ArchiveConfig): PrivateBook[] { ? fragmentByPageMarkers(body, { ...options, ...(firstPage !== undefined ? { firstPage } : {}) }) : fragmentByHeadings(body, options); const pieces: FragmentPiece[] = []; - for (const piece of cut) { + for (const [i, piece] of cut.entries()) { + // A page marker's token becomes a locator value and so part of the + // fragment id, both of which are printed by every refusal that names a + // fragment. It is bounded here to a short label of label characters, so + // an id is structural by construction; the message names the piece by + // position, never the token (STANDARDS §4). + for (const l of piece.locator) { + if (l.scheme === 'page' && !PAGE_LABEL.test(l.value)) { + throw new Error( + `${path}: piece ${i + 1}'s page marker is not a short label (letters, digits, '.', '-', at most ${PAGE_LABEL_MAX} characters).`, + ); + } + } const text = stripMarkdown(piece.text); if (!text) continue; pieces.push({ locator: piece.locator, text, ...(piece.heading !== undefined ? { heading: piece.heading } : {}) }); diff --git a/test/books.test.ts b/test/books.test.ts index e174c40..6aefe93 100644 --- a/test/books.test.ts +++ b/test/books.test.ts @@ -257,18 +257,25 @@ test('books: a title that quotes the text is refused before any build, unless th err instanceof Error && /pieces 1 and 2 of 2 share a 'page' locator/.test(err.message) && !/unguessable/.test(err.message), ); // A page marker's token becomes a locator value and so part of the fragment - // id. It is public surface by contract and goes through the metadata lint - // before the id goes anywhere: a token the lint bounds out is refused here, - // named by field and never echoed. - const oversized = 'x'.repeat(130); - const [bounded] = read({ 'w.md': `---\ntitle: W\nabout: https://example.com/w/\nfragmentBy: pages\n---\n<<>>\none\n` }); - assert.equal(bounded!.pieces[0]!.locator[0]!.value, oversized); - assert.throws( - () => fromPrivateBook(bounded!), - (err: unknown) => err instanceof Error && /'locator value \(page\)'/.test(err.message) && !/xxxxxxxx/.test(err.message), - ); - const [plain] = read({ 'w.md': `---\ntitle: W\nabout: https://example.com/w/\nfragmentBy: pages\n---\n<<>>\none\n` }); - assert.equal(fromPrivateBook(plain!)[0]!.fragment.id, 'book:w#p184'); + // id, both printed by every refusal that names a fragment. The reader bounds + // it to a short label before either exists, named by piece position and + // never echoed, so an id is structural by construction. + const pages = (marker: string) => `---\ntitle: W\nabout: https://example.com/w/\nfragmentBy: pages\n---\n<<>>\none\n`; + for (const marker of ['x'.repeat(33), 'a_b', 'p;1', '184>>>']) { + assert.throws( + () => read({ 'w.md': pages(marker) }), + (err: unknown) => + err instanceof Error && /w\.md: piece 1's page marker is not a short label/.test(err.message) && !err.message.includes(marker), + ); + } + for (const [marker, key] of [ + ['184', 'p184'], + ['xii', 'pxii'], + ['A-12', 'pA-12'], + ['3.2', 'p3.2'], + ]) { + assert.equal(fromPrivateBook(read({ 'w.md': pages(marker!) })[0]!)[0]!.fragment.id, `book:w#${key}`); + } }); test('books: through the store, a book serves locators and never its text', () => { From 58b00458e890ed9b2a7e03ae303258ea77f73749 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 7 Oct 2026 07:49:20 +0000 Subject: [PATCH 5/5] Say "linted, not structural" about a book's page markers MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The two comments called a bounded page token "structural by construction". It is an authored string in the linted bucket (STANDARDS §1, §4): the adapter's metadata lint checks it against the book; the reader's bound only gives the lint's own refusal a safe error surface. The comments and the changelog line now say that. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01F2ti456guy1bXVmjA3fBTV --- CHANGELOG.md | 8 +++++--- src/adapters/teaching.ts | 16 +++++++++------- src/corpus.ts | 9 ++++++--- 3 files changed, 20 insertions(+), 13 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 98a085a..c35285a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -214,9 +214,11 @@ npm trusts this repository's `release.yml`. every level; the order inside a list (creators, identifiers) is still content. A regression test pins both. - A private book's page marker (`<<>>`) is bounded by the reader to a - short label of letters, digits, `.` and `-`, so a page locator value, and - the fragment id built from it, is structural by construction; a marker - outside that is refused by piece position, never echoed. + short label of letters, digits, `.` and `-` before it becomes a locator + value and part of a fragment id, so the refusals that name a fragment have + a safe error surface; the value stays linted metadata, checked against the + book by the adapter's metadata lint. A marker outside the bound is refused + by piece position, never echoed. - A bare year in frontmatter (`date: 1900`, which YAML reads as a number) was dropped silently by every reader; it is now the date at year precision. diff --git a/src/adapters/teaching.ts b/src/adapters/teaching.ts index d3ba258..c5c9047 100644 --- a/src/adapters/teaching.ts +++ b/src/adapters/teaching.ts @@ -152,13 +152,15 @@ export function fromPrivateBook(book: PrivateBook): { entity: Entity; fragment: ); } keys.set(key, index); - // The key carries the locator values (ch12, p184, p184.s2). A locator is - // public surface by contract (CONTRACT.md §1: structural, on every hit as - // the label), and the metadata lint below checks every value and label - // against the whole book before this id goes anywhere, so an id is safe to - // print by the same construction as the hit's locatorLabel. A page - // marker's token is one such value, bounded to a short label by the - // reader (src/corpus.ts) before it reaches this id. + // The key carries the locator values (ch12, p184, p184.s2). A chapter or + // section value is a parsed number or an ordinal; a page value is the + // author's marker token, an authored string in the linted bucket + // (STANDARDS §1, §4). The metadata lint below checks every locator value + // and label against the whole book before this id goes anywhere, so an id + // is safe to print for the same reason the hit's locatorLabel is: linted, + // not trusted. The reader (src/corpus.ts) additionally bounds a page token + // to a short label, so the lint's own refusal, which names the id, has a + // safe error surface. fragments.push({ id: `${book.id}#${key}`, entityId: book.id, diff --git a/src/corpus.ts b/src/corpus.ts index c69c89c..13b7801 100644 --- a/src/corpus.ts +++ b/src/corpus.ts @@ -295,9 +295,12 @@ export function buildPrivateBooks(config: ArchiveConfig): PrivateBook[] { for (const [i, piece] of cut.entries()) { // A page marker's token becomes a locator value and so part of the // fragment id, both of which are printed by every refusal that names a - // fragment. It is bounded here to a short label of label characters, so - // an id is structural by construction; the message names the piece by - // position, never the token (STANDARDS §4). + // fragment. It is an authored string, so it stays in the linted bucket + // (STANDARDS §1, §4): the adapter's metadata lint checks it against the + // book before any public surface exists. What this bound adds is a safe + // error surface: a token the lint would name in a message is short and + // made of label characters, and one that is not is refused here by + // piece position, never echoed. for (const l of piece.locator) { if (l.scheme === 'page' && !PAGE_LABEL.test(l.value)) { throw new Error(