diff --git a/CHANGELOG.md b/CHANGELOG.md index d37176f..c35285a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -183,12 +183,45 @@ npm trusts this repository's `release.yml`. legacy view (which drafts nothing) resolves a requested `semantic` to `locator` itself. `projectionContentHash` moves to `src/ingest/projections.ts`, beside `projectionProblem`. +- Private books in the teaching corpus: `privateBooksDir` in `archive.config.ts` + (off by default; the example content does not change), one markdown file per + book. `buildPrivateBooks` (`src/corpus.ts`) reads the frontmatter (`title`, + `about`, and optionally `authors`, `identifiers`, `date`, `version`, `themes`, + `exposure`, `publicTitle`, `requireReview`, `fragmentBy`, `maxFragmentChars`, + `firstPage`) and cuts the body with `fragmentByHeadings` or + `fragmentByPageMarkers` into pieces whose locators are structural; a leading + `# Title` that repeats the frontmatter title is dropped as the title page. + `fromPrivateBook` (`src/adapters/teaching.ts`) yields one private entity with + one fragment per piece (`book:x#ch12`, `book:x#ch12.s3`, `book:x#p184`), the + heading inside the fragment's text and nowhere else, `publicTitle` and + `requireReview` on the entity's policy, and the metadata lint run over the + whole book before `npm run index` drafts or embeds anything; two pieces that + share a locator are refused by position and scheme. `npm run index` reads the books + beside the records and the notes; a book's `semantic` request is drafted per + fragment against the whole book's text, as CONTRACT §12 describes. ### Removed - `scripts/build-blind-artifact.mjs`. Review is open, not blind; the script's anonymized tarball has no remaining use. +### Fixed + +- `collectEntities` compared two descriptions of one entity by + `JSON.stringify`, so the same entity emitted with its properties in another + insertion order (two adapters, or one adapter run twice) was refused as + "described two ways". It now compares by content, with object keys sorted at + every level; the order inside a list (creators, identifiers) is still + content. A regression test pins both. +- A private book's page marker (`<<>>`) is bounded by the reader to a + short label of letters, digits, `.` and `-` before it becomes a locator + value and part of a fragment id, so the refusals that name a fragment have + a safe error surface; the value stays linted metadata, checked against the + book by the adapter's metadata lint. A marker outside the bound is refused + by piece position, never echoed. +- A bare year in frontmatter (`date: 1900`, which YAML reads as a number) was + dropped silently by every reader; it is now the date at year precision. + ### Breaking in the 3.0.0 line (summary) Recorded here in one place; `docs/CONTRACT.md` is the normative text. diff --git a/README.md b/README.md index 77cc2b3..ec780dc 100644 --- a/README.md +++ b/README.md @@ -65,8 +65,11 @@ notebook in `example-content/notebook/` is the private layer. Each note declares the public page it routes to (`about`), where the moment lives (`locator`), a linted display name (`label`), and, optionally, the `exposure` it wants; its `title` and body stay private, embedded so retrieval can find the -moment. The teaching adapters (`src/adapters/teaching.ts`) turn those files -into entities and fragments; a consumer writes its own. +moment. A private **book** (`privateBooksDir`, off by default) is one entity +with many fragments: the file is cut on its chapter headings or page markers, +and each piece travels as where it is (`ch. 12`, `p. 184`) or as its gist, +never as text. The teaching adapters (`src/adapters/teaching.ts`) turn those +files into entities and fragments; a consumer writes its own. > In production ([Ask the Archive](https://lukefwalton.com/ask/)), published > podcast passages are public fragments while unpublished transcript text is @@ -269,8 +272,29 @@ get `temperature: 0`). index it without ever serving it — see `docs/CONTRACT.md` §3 and §5. No private layer? Remove `privateNotesDir` from the config and the engine runs public-only. -4. Replace `example-content/` with your corpus and rerun `npm run index`. -5. Rewrite `eval/gold.yaml` for your corpus — keep the refusals. +4. A private book is one markdown file in `privateBooksDir` (set it in the + config; it is off by default). The frontmatter is the book's public + surface: `title` (required; checked against the text unless + `publicTitle: true` says it is public by construction), `about` (the public + URL a hit routes to, required), and optionally `authors` (names, or + `{ name, role }` entries), `identifiers` (`{ scheme, value }`: isbn, doi), + `date`, `version` ("manuscript"), `themes`, `exposure` (`locator` by + default; `semantic` has `npm run index` draft one gist per fragment for you + to authorize in `artifacts/projections.json`; `none`), and + `requireReview: true` to serve no gist you have not marked reviewed. The + body is the private text. It is cut into fragments on its `#`/`##` headings + (`fragmentBy: headings`, the default: "Chapter 12" is `ch. 12`, any other + heading is counted among them; a leading `# Title` that repeats the + frontmatter title is the title page, not a chapter) or on page markers + (`fragmentBy: pages`: a `<<>>` line, or a form feed with + `firstPage`); pieces longer than `maxFragmentChars` (6000) are sub-split on + paragraphs (`ch. 12, §3`). The locator is the only authored-looking thing a + hit carries about a chapter, and it is structural; the heading goes into the + embedding and nowhere else. The same build-time lint that checks a note's + label checks the book's title, version, and creators against the whole book + before anything is drafted or embedded. +5. Replace `example-content/` with your corpus and rerun `npm run index`. +6. Rewrite `eval/gold.yaml` for your corpus — keep the refusals. ## Commands diff --git a/archive.config.ts b/archive.config.ts index 500020e..9c435df 100644 --- a/archive.config.ts +++ b/archive.config.ts @@ -23,6 +23,10 @@ export const config: ArchiveConfig = { // production this is chunked podcast transcripts; here, notebook entries. // Remove the line to run public-only. privateNotesDir: './example-content/notebook', + // Private books: one markdown file per book, cut into fragments on its + // chapter headings (or page markers), served as where a moment is or as a + // gist the author authorizes, never as text. See README "Make it yours". + // privateBooksDir: './example-content/books', embeddingModel: 'text-embedding-3-large', // Default is a widely available chat model. Swap for any Responses-API model // you have access to (e.g. a reasoning model — see src/answer.ts). diff --git a/docs/CONTRACT.md b/docs/CONTRACT.md index 574a412..571e822 100644 --- a/docs/CONTRACT.md +++ b/docs/CONTRACT.md @@ -847,9 +847,17 @@ yields an entity with one `text` fragment whose locator is `[{ scheme: 'whole', value: '' }]`; `fromPrivateNote` yields a private entity with one `locator` fragment whose linted label becomes the entity title and whose locator string becomes `[{ scheme: 'note', value }]`, and whose frontmatter may ask for an -`exposure`. The example content does not change. The demo gains one entity that -exercises `semantic` in a following release: a public-domain novel with its -chapter gists committed beside the vectors, which needs a keyed build. +`exposure`. `fromPrivateBook` yields a private entity with one fragment per +piece the corpus reader cut (`buildPrivateBooks`: `fragmentByHeadings` or +`fragmentByPageMarkers`, then `splitLong`), ids `${entityId}#${locatorKey}`, +each fragment's text its heading and its piece joined as a note's title and +body are, the frontmatter's `publicTitle` and `requireReview` on the entity's +`policy`, and the section 6 metadata lint run over the whole book before the +build drafts or embeds anything; `collectEntities` then groups the pairs, comparing +two descriptions of one entity by content with keys sorted at every level, not +by property order. The example content does not change. The demo gains one +entity that exercises `semantic` in a following release: a public-domain novel +with its chapter gists committed beside the vectors, which needs a keyed build. ## 13. What remains owned rather than guaranteed diff --git a/scripts/smoke-dist.mjs b/scripts/smoke-dist.mjs index 1207254..4a84a99 100644 --- a/scripts/smoke-dist.mjs +++ b/scripts/smoke-dist.mjs @@ -35,6 +35,8 @@ for (const name of [ 'answerQuestion', 'fromArchiveRecord', 'fromPrivateNote', + 'fromPrivateBook', + 'buildPrivateBooks', ]) { assert.equal(typeof root[name], 'function', `missing export: ${name}`); } diff --git a/src/adapters/teaching.ts b/src/adapters/teaching.ts index d3a36be..c5c9047 100644 --- a/src/adapters/teaching.ts +++ b/src/adapters/teaching.ts @@ -11,10 +11,10 @@ // the same two shapes reconstructed losslessly, so today's retrieval, prompt, // eval and demo code keep running on a schema-4 index without changes. -import type { Entity, Fragment } from '../contract.js'; +import type { Entity, EntityPolicy, Fragment } from '../contract.js'; import { locatorKey, renderLocatorLabel } from '../locator.js'; -import { assertPublicSafeField } from '../public-safe.js'; -import type { ArchiveRecord, PrivateNote } from '../types.js'; +import { assertPublicSafeField, assertPublicSafeMetadata } from '../public-safe.js'; +import type { ArchiveRecord, PrivateBook, PrivateNote } from '../types.js'; export const WHOLE_LOCATOR = { scheme: 'whole', value: '' } as const; @@ -96,6 +96,83 @@ export function fromPrivateNote(note: PrivateNote): { entity: Entity; fragment: return { entity, fragment }; } +/** + * A private book is one entity with many fragments (CONTRACT.md §8, §12): one + * per piece the corpus reader cut, ids `${entityId}#${locatorKey}` (book:x#ch12, + * book:x#ch12.s3, book:x#p184), each fragment's text its heading and its piece + * joined as a note's title and body are, so a chapter title is in the + * embedding and never leaves `text`. The entity default is the exposure the + * frontmatter asked for, carried on every fragment as a request until the + * build resolves it against a projection. The frontmatter's `publicTitle` and + * `requireReview` become the entity's policy. + * + * The strings a hit on the book would carry (title, version, creators, + * themes, locator labels) go through the metadata lint here, against the whole + * book, before anything is drafted or embedded: a quoting title fails in the + * reader's output, not after a paid build. The store runs the same lint again + * at every load (CONTRACT.md §12). + */ +export function fromPrivateBook(book: PrivateBook): { entity: Entity; fragment: Fragment }[] { + const disclosure = { raw: 'private', exposure: book.exposure ?? 'locator' } as const; + const policy: EntityPolicy = { + ...(book.publicTitle ? { publicTitle: true } : {}), + ...(book.requireReview ? { requireReview: true } : {}), + }; + const entity: Entity = { + id: book.id, + type: book.type, + title: book.title, + attribution: book.attribution, + ...(book.date ? { date: book.date } : {}), + ...(book.version ? { version: book.version } : {}), + url: book.url, + identifiers: book.identifiers, + ...(book.themes.length > 0 ? { themes: book.themes } : {}), + disclosure, + ...(Object.keys(policy).length > 0 ? { policy } : {}), + }; + if (book.pieces.length === 0) { + throw new Error(`private book '${book.id}' has no pieces; a book is at least one fragment.`); + } + const fragments: Fragment[] = []; + const keys = new Map(); + for (const [index, piece] of book.pieces.entries()) { + const key = locatorKey(piece.locator); + const first = keys.get(key); + if (first !== undefined) { + // Two chapters numbered the same, or two pages marked the same: the + // locator is the id, so the author fixes the source. Named by position + // and scheme, never by value: a chapter value is a parsed number or an + // ordinal, but a page value is the author's own marker token, and this + // message is what `npm run index` prints (STANDARDS §4). + const schemes = piece.locator.map((l) => l.scheme).join('.'); + throw new Error( + `private book '${book.id}': pieces ${first + 1} and ${index + 1} of ${book.pieces.length} share a ` + + `'${schemes}' locator; each chapter heading or page marker must be distinct.`, + ); + } + keys.set(key, index); + // The key carries the locator values (ch12, p184, p184.s2). A chapter or + // section value is a parsed number or an ordinal; a page value is the + // author's marker token, an authored string in the linted bucket + // (STANDARDS §1, §4). The metadata lint below checks every locator value + // and label against the whole book before this id goes anywhere, so an id + // is safe to print for the same reason the hit's locatorLabel is: linted, + // not trusted. The reader (src/corpus.ts) additionally bounds a page token + // to a short label, so the lint's own refusal, which names the id, has a + // safe error surface. + fragments.push({ + id: `${book.id}#${key}`, + entityId: book.id, + locator: piece.locator, + text: joinNoteText(piece.heading ?? '', piece.text), + disclosure, + }); + } + assertPublicSafeMetadata(entity, fragments, { path: `private book '${book.id}'` }); + return fragments.map((fragment) => ({ entity, fragment })); +} + /** * Reverse of fromPrivateNote. The private title is the text before the first * blank line and the body is what follows: a note's title is one frontmatter diff --git a/src/cli/build-index.ts b/src/cli/build-index.ts index 36792df..e1535c0 100644 --- a/src/cli/build-index.ts +++ b/src/cli/build-index.ts @@ -1,4 +1,5 @@ -// npm run index — read both layers, draft the gists the author asked for, +// npm run index — read the public collections, the private notes, and the +// private books; draft the gists the author asked for, // resolve each fragment's exposure, embed what changed, and write the private // index (artifacts/index.json) and the author's projections file // (artifacts/projections.json). Both are private material and gitignored. @@ -25,9 +26,9 @@ import { createHash } from 'node:crypto'; import OpenAI from 'openai'; import { config } from '../../archive.config.js'; -import { fromArchiveRecord, fromPrivateNote } from '../adapters/teaching.js'; +import { fromArchiveRecord, fromPrivateBook, fromPrivateNote } from '../adapters/teaching.js'; import type { Entity, EntityPolicy, Fragment, SemanticProjection } from '../contract.js'; -import { buildCorpus, buildPrivateNotes } from '../corpus.js'; +import { buildCorpus, buildPrivateBooks, buildPrivateNotes } from '../corpus.js'; import { embedStringFor } from '../embed-string.js'; import { batchInputs, embedBatch, truncateForEmbedding } from '../embedding.js'; import { collectEntities } from '../ingest/collect.js'; @@ -107,9 +108,15 @@ async function main(): Promise { ); } const notes = buildPrivateNotes(config); - console.log(`Corpus: ${records.length} records, ${notes.length} private notes`); - - const sources: Source[] = [...records.map(fromArchiveRecord), ...notes.map(fromPrivateNote)]; + const books = buildPrivateBooks(config); + console.log(`Corpus: ${records.length} records, ${notes.length} private notes, ${books.length} private books`); + + // A record or a note is one fragment; a book is one entity with many. + const sources: Source[] = [ + ...records.map(fromArchiveRecord), + ...notes.map(fromPrivateNote), + ...books.flatMap(fromPrivateBook), + ]; const lint = lintPolicy(config.gist); if (lint) for (const { entity } of sources) entity.policy = { ...entity.policy, lint }; // Entities once, every fragment under its entity (CONTRACT.md §12). The diff --git a/src/corpus.ts b/src/corpus.ts index a930140..13b7801 100644 --- a/src/corpus.ts +++ b/src/corpus.ts @@ -8,8 +8,11 @@ import matter from 'gray-matter'; import { readdirSync, readFileSync } from 'node:fs'; import { join, resolve } from 'node:path'; +import type { Attribution, Identifier } from './contract.js'; +import { fragmentByHeadings, fragmentByPageMarkers } from './ingest/fragment.js'; +import type { FragmentPiece } from './ingest/fragment.js'; import { assertPublicSafeField } from './public-safe.js'; -import type { ArchiveConfig, ArchiveRecord, CollectionConfig, PrivateNote } from './types.js'; +import type { ArchiveConfig, ArchiveRecord, CollectionConfig, PrivateBook, PrivateNote } from './types.js'; /** Reduce markdown to plain text for indexing (link text kept, syntax dropped). */ export function stripMarkdown(body: string): string { @@ -38,6 +41,8 @@ function asThemes(data: Record): string[] { function asDate(value: unknown): string { if (value instanceof Date) return value.toISOString().slice(0, 10); + // YAML reads `date: 1900` as a number; a year is a date at its precision. + if (typeof value === 'number' && Number.isInteger(value) && value > 0) return String(value); if (typeof value === 'string' && value.trim()) return value.trim(); return ''; } @@ -182,6 +187,155 @@ export function buildPrivateNotes(config: ArchiveConfig): PrivateNote[] { return notes.sort((a, b) => a.id.localeCompare(b.id)); } +const ID_TOKEN = /^[A-Za-z0-9_-]+$/; +const PAGE_LABEL_MAX = 32; +const PAGE_LABEL = /^[\p{L}\p{N}.-]{1,32}$/u; + +/** Frontmatter `authors`: a list of names, or of `{ name, role? }` entries. + * Malformed entries name the file and the field, never their value. */ +function asAttribution(value: unknown, path: string): Attribution[] { + if (value === undefined) return []; + if (!Array.isArray(value)) throw new Error(`${path}: 'authors' must be a list of names or of { name, role } entries.`); + return value.map((entry): Attribution => { + if (typeof entry === 'string' && entry.trim()) return { name: entry.trim(), role: 'author' }; + if (entry && typeof entry === 'object') { + const { name, role } = entry as { name?: unknown; role?: unknown }; + if (typeof name !== 'string' || !name.trim()) throw new Error(`${path}: an 'authors' entry needs a 'name'.`); + if (role !== undefined && typeof role !== 'string') throw new Error(`${path}: an 'authors' entry's 'role' must be a string.`); + return { name: name.trim(), role: role?.trim() || 'author' }; + } + throw new Error(`${path}: 'authors' must be a list of names or of { name, role } entries.`); + }); +} + +/** Frontmatter `identifiers`: a list of `{ scheme, value }` entries. */ +function asIdentifiers(value: unknown, path: string): Identifier[] { + if (value === undefined) return []; + if (!Array.isArray(value)) throw new Error(`${path}: 'identifiers' must be a list of { scheme, value } entries.`); + return value.map((entry): Identifier => { + const { scheme, value: v } = (entry ?? {}) as { scheme?: unknown; value?: unknown }; + if (typeof scheme !== 'string' || !scheme.trim() || (typeof v !== 'string' && typeof v !== 'number')) { + throw new Error(`${path}: 'identifiers' must be a list of { scheme, value } entries.`); + } + return { scheme: scheme.trim(), value: String(v).trim() }; + }); +} + +function asOptionalBoolean(value: unknown, field: string, path: string): boolean | undefined { + if (value === undefined) return undefined; + if (typeof value !== 'boolean') throw new Error(`${path}: '${field}' must be true or false.`); + return value; +} + +function asOptionalPositiveInteger(value: unknown, field: string, path: string): number | undefined { + if (value === undefined) return undefined; + if (typeof value !== 'number' || !Number.isInteger(value) || value <= 0) { + throw new Error(`${path}: '${field}' must be a positive whole number.`); + } + return value; +} + +/** A leading `# Title` that repeats the frontmatter title is the title page, + * not a chapter: dropped, so it neither becomes a fragment of its own nor + * collides with the first chapter's ordinal. */ +function withoutTitleHeading(body: string, title: string): string { + const m = /^\s*#\s+(.+?)\s*#*\s*(?:\r?\n|$)/.exec(body); + if (!m || m[1]!.trim().toLowerCase() !== title.toLowerCase()) return body; + return body.slice(m[0].length); +} + +/** Read the private books. One markdown file per book; the frontmatter is the + * book's public surface (`title`, `about`, and optionally `authors`, + * `identifiers`, `date`, `version`, `themes`, `exposure`, `publicTitle`, + * `requireReview`) plus how to cut it (`fragmentBy: headings | pages`, + * `maxFragmentChars`, `firstPage`); the body is the private text, split + * here into pieces whose locators are structural (docs/CONTRACT.md §8). + * The pieces' text is embedded, never quoted; what the traveling strings may + * say about it is checked by the adapter (src/adapters/teaching.ts) before + * anything is drafted or embedded. */ +export function buildPrivateBooks(config: ArchiveConfig): PrivateBook[] { + if (!config.privateBooksDir) return []; + const dir = resolve(config.privateBooksDir); + const books: PrivateBook[] = []; + for (const { slug, path, data, content, title } of readMarkdownDir(dir, 'private books')) { + const url = firstString(data.about); + if (!url) { + throw new Error(`${path} needs 'about' (the public URL a hit on this book routes to) in its frontmatter.`); + } + const type = firstString(data.type) || 'book'; + if (!ID_TOKEN.test(type)) { + throw new Error(`${path}: 'type' must be a short token (letters, digits, '-' or '_'): it opens the entity id, as 'book' does.`); + } + const exposure = data.exposure; + if (exposure !== undefined && exposure !== 'semantic' && exposure !== 'locator' && exposure !== 'none') { + // Named, not echoed: authored frontmatter on a private book (STANDARDS §4). + throw new Error( + `${path}: 'exposure' must be semantic, locator, or none. ` + + `Private text is never exposed as text; see docs/CONTRACT.md §3.`, + ); + } + const fragmentBy = data.fragmentBy ?? 'headings'; + if (fragmentBy !== 'headings' && fragmentBy !== 'pages') { + throw new Error(`${path}: 'fragmentBy' must be headings (the default) or pages.`); + } + const maxFragmentChars = asOptionalPositiveInteger(data.maxFragmentChars, 'maxFragmentChars', path); + const firstPage = asOptionalPositiveInteger(data.firstPage, 'firstPage', path); + const publicTitle = asOptionalBoolean(data.publicTitle, 'publicTitle', path); + const requireReview = asOptionalBoolean(data.requireReview, 'requireReview', path); + const version = firstString(data.version); + const date = asDate(data.date); + + const body = withoutTitleHeading(content, title); + const options = { ...(maxFragmentChars !== undefined ? { maxFragmentChars } : {}) }; + const cut = + fragmentBy === 'pages' + ? fragmentByPageMarkers(body, { ...options, ...(firstPage !== undefined ? { firstPage } : {}) }) + : fragmentByHeadings(body, options); + const pieces: FragmentPiece[] = []; + for (const [i, piece] of cut.entries()) { + // A page marker's token becomes a locator value and so part of the + // fragment id, both of which are printed by every refusal that names a + // fragment. It is an authored string, so it stays in the linted bucket + // (STANDARDS §1, §4): the adapter's metadata lint checks it against the + // book before any public surface exists. What this bound adds is a safe + // error surface: a token the lint would name in a message is short and + // made of label characters, and one that is not is refused here by + // piece position, never echoed. + for (const l of piece.locator) { + if (l.scheme === 'page' && !PAGE_LABEL.test(l.value)) { + throw new Error( + `${path}: piece ${i + 1}'s page marker is not a short label (letters, digits, '.', '-', at most ${PAGE_LABEL_MAX} characters).`, + ); + } + } + const text = stripMarkdown(piece.text); + if (!text) continue; + pieces.push({ locator: piece.locator, text, ...(piece.heading !== undefined ? { heading: piece.heading } : {}) }); + } + if (pieces.length === 0) { + throw new Error(`${path} has no text after its frontmatter; a private book needs a body to fragment.`); + } + + books.push({ + id: `${type}:${slug}`, + type, + slug, + title, + ...(publicTitle !== undefined ? { publicTitle } : {}), + url, + attribution: asAttribution(data.authors ?? data.creators, path), + identifiers: asIdentifiers(data.identifiers, path), + ...(date ? { date } : {}), + ...(version ? { version } : {}), + themes: asThemes(data), + ...(exposure !== undefined ? { exposure } : {}), + ...(requireReview !== undefined ? { requireReview } : {}), + pieces, + }); + } + return books.sort((a, b) => a.id.localeCompare(b.id)); +} + /** The text the embedding model sees. Themes are included so topic tags * improve retrieval, not just the answer prompt. */ export function embedText(record: ArchiveRecord): string { diff --git a/src/ingest/collect.ts b/src/ingest/collect.ts index 7f8da78..652c2a7 100644 --- a/src/ingest/collect.ts +++ b/src/ingest/collect.ts @@ -9,6 +9,18 @@ import type { Entity, Fragment } from '../contract.js'; +/** JSON with object keys sorted at every level, so two descriptions of one + * entity compare by content and not by the order an adapter happened to + * assign properties in. Arrays keep their order: two creators listed the + * other way round are two descriptions. */ +function canonical(value: unknown): string { + return JSON.stringify(value, (_key, v: unknown) => { + if (v === null || typeof v !== 'object' || Array.isArray(v)) return v; + const record = v as Record; + return Object.fromEntries(Object.keys(record).sort().map((k) => [k, record[k]])); + }); +} + export interface CollectedEntities { /** Entities by id, in first-seen order. */ entities: Map; @@ -24,7 +36,7 @@ export function collectEntities(sources: readonly { entity: Entity; fragment: Fr const seen = entities.get(entity.id); if (seen === undefined) { entities.set(entity.id, entity); - } else if (JSON.stringify(seen) !== JSON.stringify(entity)) { + } else if (canonical(seen) !== canonical(entity)) { throw new Error( `entity '${entity.id}' is described two ways by its sources; an id names one entity across collections, notes, and fragments.`, ); diff --git a/src/types.ts b/src/types.ts index ad3fee2..60c0503 100644 --- a/src/types.ts +++ b/src/types.ts @@ -2,6 +2,8 @@ // AnswerEvidence: RoutingHint has no field for private text, which makes // the privacy boundary a compile-time constraint rather than a review note. +import type { Attribution, Identifier } from './contract.js'; +import type { FragmentPiece } from './ingest/fragment.js'; import type { LintedGist } from './public-safe.js'; /** One piece of the PUBLIC archive — quotable, citable, body travels. */ @@ -64,6 +66,51 @@ export interface PrivateNote { exposure?: 'semantic' | 'locator' | 'none'; } +/** + * One book of the PRIVATE layer: a manuscript, a thesis, a novel the archive + * holds but may not quote (docs/CONTRACT.md §8). Unlike a note, it is one + * entity with many fragments: the markdown body is split on chapter headings + * (or page markers) at read time, and each piece becomes a fragment whose + * locator is structural ("ch. 12", "p. 184"), never authored prose. The text + * is embedded so retrieval can find the moment; what travels about it is the + * entity's exposure: where it is (`locator`, the default), a gist the author + * authorized (`semantic`), or nothing (`none`). + */ +export interface PrivateBook { + /** Stable id: `${type}:${slug}`; `type` is frontmatter `type`, default `book`. */ + id: string; + type: string; + slug: string; + /** Frontmatter `title`. It travels on every hit as the entity title, so it + * is checked against the book's text at build unless `publicTitle` says the + * title is public by construction (a published book's is). */ + title: string; + /** Frontmatter `publicTitle: true`: skip the quote check on the title (the + * one-line bound still holds). The author's declaration (CONTRACT.md §6). */ + publicTitle?: boolean; + /** The PUBLIC page a citation routes the reader to (frontmatter `about`). */ + url: string; + /** Frontmatter `authors`: names, or `{ name, role }` entries. */ + attribution: Attribution[]; + /** Frontmatter `identifiers`: `{ scheme, value }` entries (isbn, doi, ...). */ + identifiers: Identifier[]; + date?: string; + /** Frontmatter `version`: "manuscript", an edition, a draft number. */ + version?: string; + themes: string[]; + /** Frontmatter `exposure`; absent means `locator`. `text` is not an option + * for private material (CONTRACT.md §3). */ + exposure?: 'semantic' | 'locator' | 'none'; + /** Frontmatter `requireReview: true`: a gist is servable only once the + * author marks it reviewed in artifacts/projections.json. */ + requireReview?: boolean; + /** The pieces in source order, as the corpus reader cut them (a structural + * locator, the private text with markdown stripped, and the heading that + * opened the piece, kept apart so the adapter can put it in the embedding + * and nowhere else); never empty. */ + pieces: FragmentPiece[]; +} + /** * A hint: a private (or non-quotable) hit reduced to its routing surface for * the in-package synthesis consumer (src/evidence.ts). Deliberately has NO @@ -141,6 +188,12 @@ export interface ArchiveConfig { /** Directory of private notes (frontmatter: title, about, locator; body = * private text). Omit it and the engine runs public-only. */ privateNotesDir?: string; + /** Directory of private books, one markdown file per book (frontmatter: + * title, about, and optionally authors, identifiers, date, version, themes, + * exposure, publicTitle, requireReview, fragmentBy, maxFragmentChars, + * firstPage; body = the private text, split into fragments on its chapter + * headings or page markers). Omit it and no book is indexed. */ + privateBooksDir?: string; /** OpenAI embedding model. Changing it re-embeds everything on next index run. */ embeddingModel: string; /** OpenAI model that writes the answer. */ diff --git a/test/books.test.ts b/test/books.test.ts new file mode 100644 index 0000000..6aefe93 --- /dev/null +++ b/test/books.test.ts @@ -0,0 +1,312 @@ +// Offline tests for private books: the corpus reader cuts one markdown file +// into pieces, the adapter makes one entity with many fragments, and the +// build-time lint runs on the strings a hit would carry before anything is +// drafted or embedded. The fixture is an invented novella written for this +// file. No API key, no network. + +import assert from 'node:assert/strict'; +import { mkdtempSync, writeFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { test } from 'node:test'; + +import { config } from '../archive.config.js'; +import { fromPrivateBook } from '../src/adapters/teaching.js'; +import type { Entity } from '../src/contract.js'; +import { buildPrivateBooks } from '../src/corpus.js'; +import { embedStringFor } from '../src/embed-string.js'; +import { collectEntities } from '../src/ingest/collect.js'; +import { resolveDisclosure } from '../src/ingest/disclosure.js'; +import { project } from '../src/no-leak.js'; +import { INDEX_SCHEMA_VERSION, readIndex, toServedIndex, writeIndex } from '../src/store.js'; +import type { FragmentEntry } from '../src/store.js'; + +const CHAPTER_ONE_A = + 'Mara kept the harbour lantern ledger in a tin box under the stairs, and every entry in it was a lie she could account for.'; +const CHAPTER_ONE_B = 'The box had belonged to her mother, who had kept a different kind of ledger.'; +/** Two paragraphs in the file; one whitespace-collapsed text once read. */ +const CHAPTER_ONE = `${CHAPTER_ONE_A} ${CHAPTER_ONE_B}`; +const CHAPTER_TWO = + 'The customs officer arrived on a Tuesday with a list of names and no appetite for breakfast. ' + + 'He read the list aloud twice, and the second time he left out her brother.'; +const INTERLUDE = 'A page torn from the ledger, folded twice, the ink still legible on one side.'; + +function front(extra: string[] = []): string { + return [ + '---', + 'title: The Lantern Ledger', + 'about: https://example.com/books/the-lantern-ledger/', + 'authors:', + ' - name: Person A', + ' role: author', + ' - Person B', + 'version: manuscript', + 'date: 2026-03', + 'identifiers:', + ' - scheme: isbn', + ' value: "978-0-00-000000-0"', + 'themes: [ledgers, harbours]', + 'exposure: semantic', + ...extra, + '---', + ].join('\n'); +} +const FRONT = front(); + +const BODY = [ + '# The Lantern Ledger', + '', + '## Chapter 1: The Tin Box', + '', + CHAPTER_ONE_A, + '', + CHAPTER_ONE_B, + '', + '## Chapter 2', + '', + `*${CHAPTER_TWO}*`, + '', + '## Interlude', + '', + INTERLUDE, + '', +].join('\n'); + +function bookDir(files: Record): string { + const dir = mkdtempSync(join(tmpdir(), 'ae-books-')); + for (const [name, content] of Object.entries(files)) writeFileSync(join(dir, name), content, 'utf8'); + return dir; +} + +function read(files: Record) { + return buildPrivateBooks({ ...config, privateBooksDir: bookDir(files) }); +} + +test('books: no privateBooksDir means no books; a missing directory fails loudly', () => { + assert.equal(config.privateBooksDir, undefined, 'the shipped config has no books'); + assert.deepEqual(buildPrivateBooks(config), []); + assert.throws( + () => buildPrivateBooks({ ...config, privateBooksDir: join(tmpdir(), 'ae-books-does-not-exist') }), + /cannot read private books at .*ae-books-does-not-exist/, + ); +}); + +test('books: the reader cuts one file on its headings; the title heading is the title page, not a chapter', () => { + const [book] = read({ 'the-lantern-ledger.md': `${FRONT}\n${BODY}` }); + assert.ok(book); + assert.equal(book.id, 'book:the-lantern-ledger'); + assert.equal(book.type, 'book'); + assert.equal(book.title, 'The Lantern Ledger'); + assert.equal(book.url, 'https://example.com/books/the-lantern-ledger/'); + assert.deepEqual(book.attribution, [ + { name: 'Person A', role: 'author' }, + { name: 'Person B', role: 'author' }, + ]); + assert.deepEqual(book.identifiers, [{ scheme: 'isbn', value: '978-0-00-000000-0' }]); + assert.equal(book.version, 'manuscript'); + assert.equal(book.date, '2026-03'); + assert.deepEqual(book.themes, ['ledgers', 'harbours']); + assert.equal(book.exposure, 'semantic'); + // Three pieces: two numbered chapters and one unnumbered heading counted + // among them. The `# The Lantern Ledger` line is not a fourth. + assert.deepEqual( + book.pieces.map((p) => [p.locator, p.heading]), + [ + [[{ scheme: 'chapter', value: '1' }], 'Chapter 1: The Tin Box'], + [[{ scheme: 'chapter', value: '2' }], 'Chapter 2'], + [[{ scheme: 'chapter', value: '3' }], 'Interlude'], + ], + ); + // Markdown is stripped from the text (the emphasis around chapter 2), and + // the paragraph break inside chapter 1 collapses as a note's body does. + assert.equal(book.pieces[0]!.text, CHAPTER_ONE); + assert.equal(book.pieces[1]!.text, CHAPTER_TWO); + assert.equal(book.pieces[2]!.text, INTERLUDE); +}); + +test('books: maxFragmentChars sub-splits a chapter; fragmentBy: pages reads page markers', () => { + // Chapter 1 is 200 characters of markdown in two paragraphs; chapter 2 is + // 169 in one. At 180, only chapter 1 is cut, on its paragraph boundary. + const [split] = read({ 'ledger.md': `${front(['maxFragmentChars: 180'])}\n${BODY}` }); + assert.ok(split); + assert.equal(split.pieces[0]!.text, CHAPTER_ONE_A); + assert.equal(split.pieces[1]!.text, CHAPTER_ONE_B); + assert.deepEqual( + split.pieces.map((p) => p.locator.map((l) => `${l.scheme}:${l.value}`).join('.')), + ['chapter:1.section:1', 'chapter:1.section:2', 'chapter:2', 'chapter:3'], + ); + + const paged = ['---', 'title: Typeset', 'about: https://example.com/t/', 'fragmentBy: pages', '---', '<<>>', 'First page.', '<<>>', 'Second page.', ''].join('\n'); + const [typeset] = read({ 'typeset.md': paged }); + assert.ok(typeset); + assert.equal(typeset.exposure, undefined); + assert.deepEqual( + typeset.pieces.map((p) => [p.locator, p.text]), + [ + [[{ scheme: 'page', value: '184' }], 'First page.'], + [[{ scheme: 'page', value: '185' }], 'Second page.'], + ], + ); + + const formFeed = ['---', 'title: Typeset', 'about: https://example.com/t/', 'fragmentBy: pages', 'firstPage: 40', '---', 'Page forty.\fPage forty-one.'].join('\n'); + const [ff] = read({ 'ff.md': formFeed }); + assert.deepEqual(ff!.pieces.map((p) => p.locator[0]!.value), ['40', '41']); + + // A bare year is a date at year precision, not a dropped field. + const [dated] = read({ 'd.md': '---\ntitle: D\nabout: https://example.com/d/\ndate: 1900\n---\n## Chapter 1\n\ntext\n' }); + assert.equal(dated!.date, '1900'); +}); + +test('books: malformed frontmatter names the file and the field, never a value', () => { + const file = (extra: string, body = '## Chapter 1\n\ntext\n') => + `---\ntitle: T\nabout: https://example.com/t/\n${extra}\n---\n${body}`; + assert.throws(() => read({ 'b.md': '---\ntitle: T\n---\n## Chapter 1\n\ntext\n' }), /b\.md needs 'about'/); + assert.throws( + () => read({ 'b.md': file('exposure: text or the first stanza') }), + (err: unknown) => + err instanceof Error && /b\.md: 'exposure' must be semantic, locator, or none\./.test(err.message) && !/stanza/.test(err.message), + ); + assert.throws(() => read({ 'b.md': file('fragmentBy: paragraphs') }), /b\.md: 'fragmentBy' must be headings/); + assert.throws(() => read({ 'b.md': file('maxFragmentChars: lots') }), /'maxFragmentChars' must be a positive whole number/); + assert.throws(() => read({ 'b.md': file('firstPage: 0') }), /'firstPage' must be a positive whole number/); + assert.throws(() => read({ 'b.md': file('publicTitle: yes please') }), /'publicTitle' must be true or false/); + assert.throws(() => read({ 'b.md': file('requireReview: 1') }), /'requireReview' must be true or false/); + assert.throws(() => read({ 'b.md': file('authors: Person A') }), /'authors' must be a list/); + assert.throws(() => read({ 'b.md': file('authors:\n - role: author') }), /an 'authors' entry needs a 'name'/); + assert.throws(() => read({ 'b.md': file('authors:\n - name: A\n role: 3') }), /'role' must be a string/); + assert.throws(() => read({ 'b.md': file('identifiers:\n - isbn') }), /'identifiers' must be a list of \{ scheme, value \}/); + assert.throws(() => read({ 'b.md': file('type: "a:b"') }), /'type' must be a short token/); + assert.throws(() => read({ 'b.md': file('', '\n\n') }), /b\.md has no text after its frontmatter/); + // A draft is skipped, as in every layer. + assert.deepEqual(read({ 'b.md': file('draft: true') }), []); +}); + +test('books: the adapter yields one entity with many fragments, and the metadata lint runs first', () => { + const [book] = read({ 'the-lantern-ledger.md': `${FRONT}\n${BODY}` }); + const pairs = fromPrivateBook(book!); + assert.equal(pairs.length, 3); + const entity = pairs[0]!.entity; + assert.ok(pairs.every((p) => p.entity === entity), 'every pair carries the one entity'); + assert.deepEqual(entity, { + id: 'book:the-lantern-ledger', + type: 'book', + title: 'The Lantern Ledger', + attribution: [ + { name: 'Person A', role: 'author' }, + { name: 'Person B', role: 'author' }, + ], + date: '2026-03', + version: 'manuscript', + url: 'https://example.com/books/the-lantern-ledger/', + identifiers: [{ scheme: 'isbn', value: '978-0-00-000000-0' }], + themes: ['ledgers', 'harbours'], + disclosure: { raw: 'private', exposure: 'semantic' }, + } satisfies Entity); + assert.deepEqual( + pairs.map((p) => p.fragment.id), + ['book:the-lantern-ledger#ch1', 'book:the-lantern-ledger#ch2', 'book:the-lantern-ledger#ch3'], + ); + // The heading is in the text, joined as a note's title is, and the + // embedding is the text alone: a private fragment embeds nothing else. + const first = pairs[0]!.fragment; + assert.equal(first.text, `Chapter 1: The Tin Box\n\n${CHAPTER_ONE}`); + assert.equal(embedStringFor(first, entity), first.text); + assert.deepEqual(first.locator, [{ scheme: 'chapter', value: '1' }]); + // Every fragment carries the request; the build resolves it. Without a + // projection, `semantic` resolves to `locator`. + for (const { fragment } of pairs) { + assert.deepEqual(fragment.disclosure, { raw: 'private', exposure: 'semantic' }); + assert.deepEqual(resolveDisclosure(entity, { exposure: fragment.disclosure.exposure }), { raw: 'private', exposure: 'locator' }); + } + // collectEntities groups the pairs under the one entity. + const grouped = collectEntities(pairs); + assert.deepEqual([...grouped.entities.keys()], ['book:the-lantern-ledger']); + assert.equal(grouped.fragments.get('book:the-lantern-ledger')!.length, 3); + + // The policy travels from the frontmatter. + const [reviewed] = read({ 'r.md': `---\ntitle: R\nabout: https://example.com/r/\nrequireReview: true\npublicTitle: true\n---\n## Chapter 1\n\ntext\n` }); + assert.deepEqual(fromPrivateBook(reviewed!)[0]!.entity.policy, { publicTitle: true, requireReview: true }); + assert.equal('policy' in entity, false); +}); + +test('books: a title that quotes the text is refused before any build, unless the author declares it public', () => { + const title = 'What the harbour keeps at night'; + const body = `## Chapter 1\n\nThe words what the harbour keeps at night were painted on the lid of the box.\n`; + const front = (extra = '') => `---\ntitle: ${title}\nabout: https://example.com/w/\n${extra}---\n${body}`; + const [quoting] = read({ 'w.md': front() }); + assert.throws( + () => fromPrivateBook(quoting!), + (err: unknown) => err instanceof Error && /private book 'book:w': entity 'book:w': 'title' quotes private text/.test(err.message) && !/painted/.test(err.message), + ); + const [declared] = read({ 'w.md': front('publicTitle: true\n') }); + assert.equal(fromPrivateBook(declared!).length, 1); + // A creator's name is checked the same way. + const [creator] = read({ 'w.md': front('publicTitle: true\nauthors: ["the harbour keeps at night, said"]\n') }); + assert.throws(() => fromPrivateBook(creator!), /'creator name' quotes private text/); + // Two chapters numbered the same would share an id; the reader's output is + // refused by position and scheme. A page marker's token is the author's own + // text, so the message never carries a locator value. + const [twice] = read({ 'w.md': `---\ntitle: W\nabout: https://example.com/w/\n---\n## Chapter 3\n\none\n\n## Chapter 3\n\ntwo\n` }); + assert.throws(() => fromPrivateBook(twice!), /private book 'book:w': pieces 1 and 2 of 2 share a 'chapter' locator/); + const [marked] = read({ + 'w.md': `---\ntitle: W\nabout: https://example.com/w/\nfragmentBy: pages\n---\n<<>>\none\n<<>>\ntwo\n`, + }); + assert.throws( + () => fromPrivateBook(marked!), + (err: unknown) => + err instanceof Error && /pieces 1 and 2 of 2 share a 'page' locator/.test(err.message) && !/unguessable/.test(err.message), + ); + // A page marker's token becomes a locator value and so part of the fragment + // id, both printed by every refusal that names a fragment. The reader bounds + // it to a short label before either exists, named by piece position and + // never echoed, so an id is structural by construction. + const pages = (marker: string) => `---\ntitle: W\nabout: https://example.com/w/\nfragmentBy: pages\n---\n<<>>\none\n`; + for (const marker of ['x'.repeat(33), 'a_b', 'p;1', '184>>>']) { + assert.throws( + () => read({ 'w.md': pages(marker) }), + (err: unknown) => + err instanceof Error && /w\.md: piece 1's page marker is not a short label/.test(err.message) && !err.message.includes(marker), + ); + } + for (const [marker, key] of [ + ['184', 'p184'], + ['xii', 'pxii'], + ['A-12', 'pA-12'], + ['3.2', 'p3.2'], + ]) { + assert.equal(fromPrivateBook(read({ 'w.md': pages(marker!) })[0]!)[0]!.fragment.id, `book:w#${key}`); + } +}); + +test('books: through the store, a book serves locators and never its text', () => { + const [book] = read({ 'the-lantern-ledger.md': `${FRONT}\n${BODY}`.replace('exposure: semantic', 'exposure: locator') }); + const pairs = fromPrivateBook(book!); + const { entities, fragments } = collectEntities(pairs); + const entries: FragmentEntry[] = fragments.get(book!.id)!.map((fragment, i) => ({ + model: 'test-embedding', + dimensions: 2, + vector: [1, i], + contentHash: `hash-${i}`, + fragment: { ...fragment, disclosure: resolveDisclosure(entities.get(book!.id)!, { exposure: fragment.disclosure.exposure }) }, + })); + const path = join(mkdtempSync(join(tmpdir(), 'ae-books-index-')), 'index.json'); + writeIndex({ version: INDEX_SCHEMA_VERSION, entities: [...entities.values()], entries }, path); + const loaded = readIndex(path); + assert.equal(loaded.entries.length, 3); + const served = toServedIndex(loaded); + const serialized = JSON.stringify(served); + for (const phrase of ['tin box under the stairs', 'no appetite for breakfast', 'folded twice', 'The Tin Box']) { + assert.ok(!serialized.includes(phrase), `served index must not carry: ${phrase}`); + } + assert.ok(served.entries.every((e) => e.fragment.text === '' && e.fragment.projection === undefined)); + // And a hit on it carries where, not what. + const entry = loaded.entries[1]!; + const hit = project({ fragment: entry.fragment, entity: entities.get(book!.id)!, cosine: 0.87, score: 0.87, breakdown: { cosine: 0.87 } }); + assert.equal(hit.exposure, 'locator'); + assert.equal(hit.locatorLabel, 'ch. 2'); + assert.equal(hit.entity.title, 'The Lantern Ledger'); + assert.equal(hit.entity.url, 'https://example.com/books/the-lantern-ledger/'); + assert.deepEqual(hit.entity.identifiers, [{ scheme: 'isbn', value: '978-0-00-000000-0' }]); + assert.equal(hit.score, 0.85); + assert.ok(!JSON.stringify(hit).includes('breakfast')); +}); diff --git a/test/contract.test.ts b/test/contract.test.ts index da93ae0..4fa4e8a 100644 --- a/test/contract.test.ts +++ b/test/contract.test.ts @@ -488,3 +488,60 @@ test('collectEntities: entities once, many fragments; an id described two ways o /fragment 'book:x#p1' names entity 'book:x' but was produced under 'essay:y'/, ); }); + +test('collectEntities: one entity in two property orders is one entity; array order is content', () => { + const page = (n: number): Fragment => ({ + id: `book:x#p${n}`, + entityId: 'book:x', + locator: [{ scheme: 'page', value: String(n) }], + text: `page ${n}`, + disclosure: { raw: 'private', exposure: 'locator' }, + }); + // The same description, assigned in another order at every level (the + // entity, its policy, the lint window, a creator, an identifier): two + // adapters, or one adapter run twice, may well differ in insertion order. + const first: Entity = { + id: 'book:x', + type: 'book', + title: 'X', + attribution: [{ name: 'A. Author', role: 'author', identifiers: [{ scheme: 'orcid', value: '0000' }] }], + url: 'https://example.com/x/', + identifiers: [{ scheme: 'isbn', value: '1' }], + disclosure: { raw: 'private', exposure: 'locator' }, + policy: { requireReview: true, lint: { ngramWords: 5, gistMaxChars: 300 } }, + }; + const second: Entity = { + policy: { lint: { gistMaxChars: 300, ngramWords: 5 }, requireReview: true }, + disclosure: { exposure: 'locator', raw: 'private' }, + identifiers: [{ value: '1', scheme: 'isbn' }], + url: 'https://example.com/x/', + attribution: [{ identifiers: [{ value: '0000', scheme: 'orcid' }], role: 'author', name: 'A. Author' }], + title: 'X', + type: 'book', + id: 'book:x', + }; + assert.notEqual(JSON.stringify(first), JSON.stringify(second), 'the fixture must differ in order'); + const grouped = collectEntities([ + { entity: first, fragment: page(1) }, + { entity: second, fragment: page(2) }, + ]); + assert.equal(grouped.entities.get('book:x'), first, 'the first description is the one kept'); + assert.deepEqual(grouped.fragments.get('book:x')!.map((f) => f.id), ['book:x#p1', 'book:x#p2']); + + // Order inside a list is content: creators listed the other way round, or + // identifiers, describe the entity differently and are still refused. + const creators = (names: string[]): Entity => ({ ...first, attribution: names.map((name) => ({ name, role: 'author' })) }); + assert.throws( + () => collectEntities([{ entity: creators(['A', 'B']), fragment: page(1) }, { entity: creators(['B', 'A']), fragment: page(2) }]), + /entity 'book:x' is described two ways/, + ); + // And a real difference under a reordered key is still a difference. + assert.throws( + () => + collectEntities([ + { entity: first, fragment: page(1) }, + { entity: { ...second, policy: { lint: { gistMaxChars: 300, ngramWords: 4 }, requireReview: true } }, fragment: page(2) }, + ]), + /entity 'book:x' is described two ways/, + ); +});