diff --git a/CHANGELOG.md b/CHANGELOG.md index 1a98032..2a9ab13 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,26 @@ and this project follows [Semantic Versioning](https://semver.org/spec/v2.0.0.ht ## [Unreleased] +### Added +- An OpenTimestamps proof parser, `src/audit/ots-proof.ts`, implementing the proof grammar of + `docs/audit-format.md`. It is bounded against hostile input with caps on argument size, total + size, fork depth, operation count, and working message length, because the file being verified is + attacker-influenced by definition. RIPEMD-160 and Keccak-256 operations are declined rather than + evaluated, matching the independent Go verifier. + +### Changed +- The bundled verifier's `anchored` layer judges an anchor record by the evidence instead of by what + the record says about itself. It recomputes each record's digest from the checkpoint the record + embeds and reports `digest-mismatch` when they differ, requires non-empty proof bytes behind any + submission that reached a calendar and reports `proof-missing` when they are absent, and parses the + proof against the submitted digest, reporting `proof-parse-error` when the container does not + parse. An anchor claiming `confirmed` with no proof file behind it fails the layer. A submission + recorded with an `error` is exempt: it never reached a calendar, so it has no proof to point at and + is already counted as failed. +- `agentwall anchor` parses a calendar response before keeping it, and treats one that does not parse + as a submission failure rather than writing it as a proof. A broken or hostile answer therefore + leaves a recorded gap instead of a file that a later verify reports as corrupt evidence. + ## [0.2.0] - 2026-08-05 The first tagged release. It freezes the on-disk evidence format and makes that format diff --git a/README.md b/README.md index f4645b0..ee920f7 100644 --- a/README.md +++ b/README.md @@ -354,23 +354,14 @@ node scripts/conformance.js The run prints one line per case. Its tail: ``` -DIVERGENCE b9-anchor-digest-altered - format expects exit=1 chained=true linked=true anchored=false - typescript returns exit=0 chained=true linked=true anchored=true - it reports the digest the record claims was submitted and never recomputes one from the checkpoint the record embeds -DIVERGENCE b10-proof-truncated - format expects exit=1 chained=true linked=true anchored=false - typescript returns exit=0 chained=true linked=true anchored=true - it never opens a proof file, so a proof that cannot be parsed still counts as an anchor +ok b9-anchor-digest-altered exit=1 chained=true linked=true anchored=false +ok b10-proof-truncated exit=1 chained=true linked=true anchored=false DIVERGENCE b11-torn-tail format expects exit=1 chained=true linked=true anchored=false typescript returns exit=1 chained=false linked=true anchored=false it reports a partial final line as a broken chain rather than as the torn tail a hard kill leaves behind ok b12-duplicate-key-shadowed exit=1 chained=false linked=true anchored=false -DIVERGENCE b13-confirmed-without-proof - format expects exit=1 chained=true linked=true anchored=false - typescript returns exit=0 chained=true linked=true anchored=true - it counts the status field, so an anchor claiming confirmation passes with no proof bytes behind it +ok b13-confirmed-without-proof exit=1 chained=true linked=true anchored=false ok b14-submission-never-reached-calendar exit=1 chained=true linked=true anchored=false ok b15-sealed-segment-rewritten exit=1 chained=true linked=false anchored=true ok b16-live-tail-rewritten-after-checkpoint exit=1 chained=true linked=true anchored=false @@ -378,7 +369,7 @@ ok b17-sealed-segment-missing exit=1 chained=true linked=false anchored ok l1-confirmed-with-pending-proof exit=0 chained=true linked=true anchored=true ok l2-legacy-canon-unmarked exit=1 chained=false linked=true anchored=false -26 cases, typescript and go: 22 agreed, 4 declared divergence(s), 0 failure(s) +26 cases, typescript and go: 25 agreed, 1 declared divergence(s), 0 failure(s) ``` Each case is copied to a temp directory before it runs, so a verifier cannot alter what it checks. @@ -392,23 +383,23 @@ That prints nothing, because the regenerated tree is byte identical to the commi ### Where the two verifiers disagree today -Four corpus cases get different verdicts from the two verifiers. In three of them the bundled -TypeScript verifier accepts evidence the format rejects, which makes the Go verifier the stricter of -the two today. In the fourth both reject the file, and the bundled verifier blames the chain instead -of naming the torn tail: +One corpus case gets different verdicts from the two verifiers. Both reject the file, and the +bundled verifier blames the chain instead of naming the torn tail: | Case | The edit | Bundled TypeScript verifier | Go verifier | | --- | --- | --- | --- | -| `b9-anchor-digest-altered` | the anchor record's `digest` field altered | `anchored` PASS, exit 0. It reports the digest the record claims was submitted and never recomputes one from the checkpoint the record embeds | `digest-mismatch`, `anchored` FAIL, exit 1 | -| `b10-proof-truncated` | the OTS proof truncated inside a length prefix | `anchored` PASS, exit 0. It never opens a proof file, so a proof that cannot be parsed still counts as an anchor | `proof-parse-error`, `anchored` FAIL, exit 1 | | `b11-torn-tail` | a partial final line, as a hard kill leaves behind | `chained` FAIL, exit 1. It condemns the whole chain over one partial write | `torn-tail` reported distinctly, `chained` PASS, exit 1 because nothing is anchored | -| `b13-confirmed-without-proof` | an anchor claiming `confirmed` with its proof file deleted | `anchored` PASS, exit 0. It counts the status field, so a claim of confirmation passes with no proof bytes behind it | `proof-missing`, `anchored` FAIL, exit 1 | -The three acceptance gaps are limits of the bundled verifier as it ships today. The harness prints -every entry in this list on each run and fails if one of them starts agreeing -([`scripts/conformance.js:40-66`](scripts/conformance.js)), so the list cannot rot into a set of -excuses, and it is why the summary line above reports four declared divergences instead of agreement -on every case. +That naming gap is a limit of the bundled verifier as it ships today. The harness prints every entry +in this list on each run and fails if one of them starts agreeing +([`scripts/conformance.js:40-51`](scripts/conformance.js)), so the list cannot rot into a set of +excuses, and it is why the summary line above reports one declared divergence instead of agreement on +every case. + +The bundled verifier recomputes each anchor record's digest from the checkpoint the record embeds, +requires a non-empty proof file behind any submission that reached a calendar, and parses that proof +against the submitted digest. So an altered digest, a deleted proof, and a truncated proof all fail +the `anchored` layer in both verifiers rather than in one. ### What verification does not prove diff --git a/docs/audit-format.md b/docs/audit-format.md index 3f3108f..4261175 100644 --- a/docs/audit-format.md +++ b/docs/audit-format.md @@ -810,7 +810,8 @@ The bundled implementation of this format is in [`src/audit/file-sink.ts`](../src/audit/file-sink.ts) for the writer and the per-file chain walk, [`src/audit/rotation.ts`](../src/audit/rotation.ts) for the manifest, [`src/audit/signing.ts`](../src/audit/signing.ts) for checkpoints, -[`src/audit/anchor.ts`](../src/audit/anchor.ts) for anchor records and proof persistence, and +[`src/audit/anchor.ts`](../src/audit/anchor.ts) for anchor records and proof persistence, +[`src/audit/ots-proof.ts`](../src/audit/ots-proof.ts) for the proof grammar above, and [`src/audit/anchor-service.ts`](../src/audit/anchor-service.ts) for the three-layer verify. Where that code and this document disagree, this document is correct and the code has a bug. diff --git a/scripts/conformance.js b/scripts/conformance.js index f9f5489..499abfe 100755 --- a/scripts/conformance.js +++ b/scripts/conformance.js @@ -43,26 +43,11 @@ const SKIP_GO = process.env.CONFORMANCE_SKIP_GO === "1"; * the bundled verifier, not opinions about the corpus. */ const DIVERGENCES = { - "b9-anchor-digest-altered": { - exit: 0, - layers: { anchored: true }, - why: "it reports the digest the record claims was submitted and never recomputes one from the checkpoint the record embeds", - }, - "b10-proof-truncated": { - exit: 0, - layers: { anchored: true }, - why: "it never opens a proof file, so a proof that cannot be parsed still counts as an anchor", - }, "b11-torn-tail": { exit: 1, layers: { chained: false }, why: "it reports a partial final line as a broken chain rather than as the torn tail a hard kill leaves behind", }, - "b13-confirmed-without-proof": { - exit: 0, - layers: { anchored: true }, - why: "it counts the status field, so an anchor claiming confirmation passes with no proof bytes behind it", - }, }; function fail(message) { diff --git a/src/audit/anchor-service.ts b/src/audit/anchor-service.ts index 513f54a..f70201f 100644 --- a/src/audit/anchor-service.ts +++ b/src/audit/anchor-service.ts @@ -1,9 +1,10 @@ import { createHash } from "crypto"; -import { appendFileSync, existsSync, mkdirSync, readFileSync } from "fs"; -import { dirname, join, resolve } from "path"; -import { anchorToOpenTimestamps, fetchPoster, type AnchorRecord } from "./anchor"; +import { appendFileSync, existsSync, mkdirSync, readFileSync, statSync } from "fs"; +import { basename, dirname, isAbsolute, join, resolve } from "path"; +import { anchorDigest, anchorToOpenTimestamps, fetchPoster, type AnchorRecord } from "./anchor"; import { verifyChainFile } from "./file-sink"; import { chainAuditEvent, findDuplicateKey } from "./chain"; +import { OtsParseError, parseOtsProofFile } from "./ots-proof"; import { loadOrCreateKeys, signCheckpoint, verifyCheckpoint, type Checkpoint } from "./signing"; import { adoptExistingSegments, @@ -269,6 +270,26 @@ function countIndexReuse(path: string): { distinct: number; worst: number } | nu } } +/** + * Find the proof file an anchor record names. + * + * `proofPath` holds whatever the producer wrote, relative to a working directory this + * verifier neither knows nor needs to share, because an evidence directory gets copied + * between hosts. So a short fixed candidate list is tried and the first file that exists + * wins. Deriving the name from the digest instead is refused: naming a proof after its + * digest is a writer convention rather than a rule of the format, and the recorded path + * is the only thing that finds a proof named any other way. Nothing here writes. + */ +function resolveProofPath(proofPath: string, r: ResolvedPaths): string | null { + const candidates = isAbsolute(proofPath) ? [proofPath] : []; + candidates.push( + join(r.proofDir, proofPath), + join(r.proofDir, basename(proofPath)), + join(dirname(r.anchorLogPath), proofPath), + ); + return candidates.find((c) => existsSync(c)) ?? null; +} + export interface LayerVerdict { name: string; ok: boolean; @@ -402,6 +423,8 @@ export function runVerify(paths: AnchorPaths): VerifyReport { let pending = 0; let confirmed = 0; let failed = 0; + let calendarAttestations = 0; + let bitcoinAttestations = 0; if (existsSync(r.anchorLogPath) && existsSync(r.keyPath)) { const manifestEntries = readManifest(r.manifestPath); // One composite set per distinct sealed-segment count. An anchor log holds many @@ -434,6 +457,19 @@ export function runVerify(paths: AnchorPaths): VerifyReport { if (rec.checkpoint) { const v = verifyCheckpoint(rec.checkpoint, keys.publicKey.export({ type: "spki", format: "der" }).toString("base64")); if (!v.ok) anchorProblems.push(`checkpoint ${rec.checkpoint.chainIndex}: ${v.problem}`); + // Recompute the digest from the checkpoint the record carries, rather than + // reporting the one the record states. Taken on trust, `digest` lets a forger + // point a record at a checkpoint its proof never covered: the proof still + // parses, because nothing tied the two together. Recomputing is what makes an + // off-box timestamp attest to THIS checkpoint. + const submitted = anchorDigest(rec.checkpoint); + if (rec.digest !== submitted) { + anchorProblems.push( + `checkpoint ${rec.checkpoint.chainIndex}: digest-mismatch, the record says it submitted ` + + `${String(rec.digest).slice(0, 16)} and the checkpoint it embeds hashes to ` + + `${submitted.slice(0, 16)}, so the proof does not attest to this checkpoint`, + ); + } const composites = compositesFor(rec.checkpoint.chainIndex); if (!composites) { anchorProblems.push( @@ -448,6 +484,11 @@ export function runVerify(paths: AnchorPaths): VerifyReport { "no longer describes the evidence on disk", ); } + } else { + anchorProblems.push( + "digest-mismatch, an anchor record embeds no checkpoint, so there is nothing to " + + "recompute its digest from and nothing for its proof to be about", + ); } // A total submission failure is recorded as status "pending" WITH an error, // because the record is written either way. Counting that as pending would @@ -456,6 +497,48 @@ export function runVerify(paths: AnchorPaths): VerifyReport { if (rec.error) failed++; else if (rec.status === "confirmed") confirmed++; else if (rec.status === "pending") pending++; + + // `status` is what the record says about itself, and a record is exactly as + // trustworthy as the host that wrote it. The calendar's response IS the proof, so + // an anchor that reached a calendar has proof bytes behind it and one that did not + // carries `error` and is already counted failed. Without this, "confirmed" with an + // empty proof directory verifies, which is the overclaim this layer exists to + // refuse: it would report Bitcoin-grade evidence for a line of JSON. + if (!rec.error) { + const named = typeof rec.proofPath === "string" ? rec.proofPath : ""; + const found = named ? resolveProofPath(named, r) : null; + if (!found || statSync(found).size === 0) { + anchorProblems.push( + `anchor ${rec.chainIndex}: proof-missing, it records status "${rec.status}" and ` + + (named + ? `the proof it names (${basename(named)}) is absent or empty` + : "names no proof file") + + ", so no off-box bytes stand behind the claim", + ); + continue; + } + // Parse it. Unopened, a proof is a file name: truncate the bytes and the anchor + // still counts, which reduces the whole layer to trusting that an HTTP request + // once happened. + let parseProblem: string | null = null; + try { + const attestations = parseOtsProofFile(found, Buffer.from(String(rec.digest), "hex")); + if (attestations.length === 0) parseProblem = "it parses but reaches no attestation"; + for (const a of attestations) { + if (a.kind === "pending") calendarAttestations++; + else bitcoinAttestations++; + } + } catch (err) { + parseProblem = + err instanceof OtsParseError ? err.message : `unreadable: ${(err as Error).message}`; + } + if (parseProblem) { + anchorProblems.push( + `anchor ${rec.chainIndex}: proof-parse-error, ${basename(found)} ${parseProblem}, ` + + "so the bytes on disk are not the timestamp the record claims", + ); + } + } } } const attempted = confirmed + pending + failed; @@ -466,7 +549,9 @@ export function runVerify(paths: AnchorPaths): VerifyReport { attempted === 0 ? "nothing anchored off-box yet" : `${confirmed} confirmed, ${pending} pending a Bitcoin block` + - (failed ? `, ${failed} FAILED to reach a calendar` : ""), + (failed ? `, ${failed} FAILED to reach a calendar` : "") + + `; proofs carry ${calendarAttestations} calendar and ${bitcoinAttestations} ` + + "bitcoin attestation(s), neither kind confirmation on its own", problems: anchorProblems, }); diff --git a/src/audit/anchor.ts b/src/audit/anchor.ts index 2d56137..51d6c45 100644 --- a/src/audit/anchor.ts +++ b/src/audit/anchor.ts @@ -1,6 +1,7 @@ import { createHash } from "crypto"; import { existsSync, mkdirSync, readFileSync, writeFileSync } from "fs"; import { join } from "path"; +import { parseOtsProof } from "./ots-proof"; import type { Checkpoint } from "./signing"; /** @@ -155,6 +156,21 @@ export async function anchorToOpenTimestamps( failures.push(`${cal}: empty proof body`); continue; } + // Parse the body before it is kept. A response that does not lead from the + // digest we submitted to an attestation is not a timestamp, whether the + // calendar is broken or an on-path attacker answered for it. Writing it anyway + // would put bytes on disk that a later verify reports as a proof parse error, + // so the operator would be told their evidence is corrupt when in truth it was + // never a proof. Try the next calendar instead. + try { + if (parseOtsProof(res.body, Buffer.from(digest, "hex")).length === 0) { + failures.push(`${cal}: proof reaches no attestation`); + continue; + } + } catch (err) { + failures.push(`${cal}: unparseable proof, ${(err as Error).message}`); + continue; + } let proofPath: string | undefined; if (proofDir) { mkdirSync(proofDir, { recursive: true, mode: 0o700 }); diff --git a/src/audit/ots-proof.ts b/src/audit/ots-proof.ts new file mode 100644 index 0000000..ebfcacf --- /dev/null +++ b/src/audit/ots-proof.ts @@ -0,0 +1,281 @@ +import { createHash } from "crypto"; +import { readFileSync, statSync } from "fs"; + +/** + * OpenTimestamps proof parsing, per section 3.7 of docs/audit-format.md. + * + * WHY A PARSER AT ALL + * + * The calendar's response body IS the timestamp: the operations that lead from the digest + * we submitted up to an attestation. A verifier that only checks the file exists accepts + * a proof consisting of arbitrary bytes, so an anchor degrades into a claim that some + * file is on disk. Parsing is what turns those bytes back into the statement they carry. + * + * WHY EVERY LENGTH HERE IS CAPPED + * + * The proof is attacker-influenced by definition: it arrives over the network and lands + * in a directory on the host whose history is in dispute. An adversary who can drop a + * file there can hand the verifier a varint claiming a gigabyte-long append, a fork + * chain nested a million deep, or an endless run of continuation bytes. Any of those + * suppress the verdict by exhausting the process instead of by forging anything, so a + * verifier that its own input can wedge is itself the attack surface. Each cap below + * turns that class of input into a fast, named parse failure. + * + * WHAT A PARSED PROOF DOES AND DOES NOT ESTABLISH + * + * A pending attestation says a calendar accepted the submission. It is not proof of + * anything being timestamped and MUST NOT be reported as such. A Bitcoin attestation + * yields a block height and the value the operations derive; confirming inclusion means + * comparing that value against the block's real merkle root, which needs a Bitcoin + * source this offline path does not fetch. + */ + +/** + * Bounds on what one proof file may ask the verifier to do. Sized so that every proof a + * calendar legitimately produces fits with room to spare: a real merkle path is a few + * dozen operations over arguments of tens of bytes. + */ +export const OTS_LIMITS = { + /** Total proof bytes. A merkle path is short; anything near this is not one. */ + maxFileBytes: 1 << 20, + /** One append, prepend, or attestation payload. */ + maxVarBytes: 4096, + /** Fork and operation nesting, bounding recursion depth. */ + maxDepth: 256, + /** Operations across all branches, bounding total work. */ + maxOps: 4096, + /** Attestations collected, bounding output growth. */ + maxAttestations: 256, + /** Working message length, bounding append and hexlify growth. */ + maxMessageBytes: 1 << 16, + /** Varint bytes, so a run of continuation bytes cannot spin forever. */ + maxVarintBytes: 9, +} as const; + +/** The 31 magic bytes that open a full `.ots` file. */ +const OTS_MAGIC = Buffer.from([ + 0x00, 0x4f, 0x70, 0x65, 0x6e, 0x54, 0x69, 0x6d, 0x65, 0x73, 0x74, 0x61, 0x6d, 0x70, 0x73, 0x00, + 0x00, 0x50, 0x72, 0x6f, 0x6f, 0x66, 0x00, 0xbf, 0x89, 0xe2, 0xe8, 0x84, 0xe8, 0x92, 0x94, +]); + +const TAG_PENDING = Buffer.from([0x83, 0xdf, 0xe3, 0x0d, 0x2e, 0xf9, 0x0c, 0x8e]); +const TAG_BITCOIN = Buffer.from([0x05, 0x88, 0x96, 0x0d, 0x73, 0xd7, 0x19, 0x01]); + +/** What a proof leads to. Pending and Bitcoin carry different facts, so they differ in shape. */ +export type OtsAttestation = + | { kind: "pending"; uri: string } + | { kind: "bitcoin"; height: number; value: Buffer }; + +/** + * A proof that does not parse. Distinct from every other failure so a caller can report + * "these bytes are not a proof" rather than blaming the evidence they describe. + */ +export class OtsParseError extends Error { + constructor(message: string) { + super(message); + this.name = "OtsParseError"; + } +} + +/** Cursor over proof bytes. Every read is bounds-checked; nothing here trusts a length. */ +class Reader { + private pos = 0; + private ops = 0; + private attestations = 0; + + constructor(private readonly data: Buffer) {} + + get offset(): number { + return this.pos; + } + + get exhausted(): boolean { + return this.pos === this.data.length; + } + + seek(pos: number): void { + this.pos = pos; + } + + byte(): number { + if (this.pos >= this.data.length) throw new OtsParseError("unexpected end of proof"); + return this.data[this.pos++]; + } + + bytes(n: number): Buffer { + if (n < 0 || this.pos + n > this.data.length) throw new OtsParseError("proof truncated"); + const out = this.data.subarray(this.pos, this.pos + n); + this.pos += n; + return out; + } + + /** + * Unsigned little-endian base 128, high bit as continuation. + * + * The byte count is capped so a file made entirely of continuation bytes terminates, + * and a value past exact integer range is refused rather than rounded: a verifier that + * reported an approximate block height would be stating a fact it does not hold. + */ + varint(): number { + let value = 0; + let scale = 1; + for (let i = 0; ; i++) { + if (i >= OTS_LIMITS.maxVarintBytes) throw new OtsParseError("varint exceeds length cap"); + const b = this.byte(); + value += (b & 0x7f) * scale; + if (!Number.isSafeInteger(value)) throw new OtsParseError("varint exceeds exact integer range"); + if ((b & 0x80) === 0) return value; + scale *= 128; + } + } + + /** A varint length then that many bytes, with the length capped before it is trusted. */ + varbytes(): Buffer { + const n = this.varint(); + if (n > OTS_LIMITS.maxVarBytes) { + throw new OtsParseError(`varbytes length ${n} exceeds cap ${OTS_LIMITS.maxVarBytes}`); + } + return this.bytes(n); + } + + /** + * One timestamp node: a run of fork edges, each prefixed `FF`, then one final edge. + * A fork branch gets its own copy of the message, so one branch cannot alter what a + * sibling sees. + */ + timestamp(msg: Buffer, depth: number, out: OtsAttestation[]): void { + if (depth > OTS_LIMITS.maxDepth) throw new OtsParseError("proof nesting exceeds depth cap"); + for (;;) { + const tag = this.byte(); + if (tag !== 0xff) { + this.edge(tag, msg, depth, out); + return; + } + this.edge(this.byte(), Buffer.from(msg), depth, out); + } + } + + /** One edge: an attestation leaf, or an operation whose result feeds a child timestamp. */ + private edge(tag: number, msg: Buffer, depth: number, out: OtsAttestation[]): void { + if (tag === 0x00) { + this.attestation(msg, out); + return; + } + this.timestamp(this.applyOp(tag, msg), depth + 1, out); + } + + /** One operation against the current message. Growth is capped after every step. */ + private applyOp(tag: number, msg: Buffer): Buffer { + if (++this.ops > OTS_LIMITS.maxOps) throw new OtsParseError("proof exceeds operation cap"); + switch (tag) { + case 0xf0: + return capMessage(Buffer.concat([msg, this.varbytes()])); + case 0xf1: + return capMessage(Buffer.concat([this.varbytes(), msg])); + case 0xf2: + return Buffer.from(msg).reverse(); + case 0xf3: + return capMessage(Buffer.from(msg.toString("hex"), "utf8")); + case 0x02: + return createHash("sha1").update(msg).digest(); + case 0x08: + return createHash("sha256").update(msg).digest(); + case 0x03: + case 0x67: + // RIPEMD-160 and Keccak-256 are declined rather than evaluated. Neither is + // reliably present in this runtime's crypto provider, so accepting them would + // make a verdict depend on how the host's OpenSSL was built, and hand-rolling + // a hash primitive inside the component whose whole value is being trustworthy + // trades away more than it buys. A proof needing one is unverifiable here, and + // the independent Go verifier declines them too, so both agree. + throw new OtsParseError(`op 0x${tag.toString(16)} is not supported by this verifier`); + default: + throw new OtsParseError(`unknown op tag 0x${tag.toString(16).padStart(2, "0")}`); + } + } + + /** + * One attestation: eight tag bytes then a varbytes payload. An unrecognized tag is + * skipped using that length, so a type added later is ignored rather than treated as + * corruption. It is neither proof nor failure. + */ + private attestation(msg: Buffer, out: OtsAttestation[]): void { + if (++this.attestations > OTS_LIMITS.maxAttestations) { + throw new OtsParseError("proof exceeds attestation cap"); + } + const tag = this.bytes(8); + const payload = this.varbytes(); + if (tag.equals(TAG_PENDING)) { + out.push({ kind: "pending", uri: new Reader(payload).varbytes().toString("utf8") }); + } else if (tag.equals(TAG_BITCOIN)) { + out.push({ kind: "bitcoin", height: new Reader(payload).varint(), value: Buffer.from(msg) }); + } + } +} + +function capMessage(b: Buffer): Buffer { + if (b.length > OTS_LIMITS.maxMessageBytes) { + throw new OtsParseError(`proof message grew past ${OTS_LIMITS.maxMessageBytes} bytes`); + } + return b; +} + +/** + * Parse proof bytes against the digest the anchor record says was submitted. + * + * Both container shapes carry the same operations stream and both apply it to that + * digest, so a full `.ots` file's own embedded digest is skipped rather than believed: + * the anchor record is what ties the proof to a checkpoint, and a header that disagreed + * with it would otherwise let a proof for one digest vouch for another. + * + * Trailing bytes after the stream are a failure. A proof is exactly the operations that + * reach an attestation, and unconsumed bytes mean the file is not the thing it claims to + * be, which is also what catches a proof with a second stream appended. + * + * @throws OtsParseError when the container does not parse or exceeds a cap. + */ +export function parseOtsProof(data: Buffer, digest: Buffer): OtsAttestation[] { + if (data.length > OTS_LIMITS.maxFileBytes) { + throw new OtsParseError(`proof is ${data.length} bytes, exceeding the ${OTS_LIMITS.maxFileBytes} byte cap`); + } + const r = new Reader(data); + if (data.length >= OTS_MAGIC.length && data.subarray(0, OTS_MAGIC.length).equals(OTS_MAGIC)) { + r.seek(OTS_MAGIC.length); + r.varint(); // format version + // File hash-op tag, then the digest it introduces. SHA-1 and RIPEMD-160 are 20 bytes, + // SHA-256 is 32; the bytes are skipped, not evaluated, so an op unsupported for + // message hashing is still a legal header here. + const fileHashOp = r.byte(); + const digestLength = fileHashOp === 0x08 ? 32 : fileHashOp === 0x02 || fileHashOp === 0x03 ? 20 : 0; + if (digestLength === 0) { + throw new OtsParseError(`unknown file hash-op tag 0x${fileHashOp.toString(16).padStart(2, "0")}`); + } + r.bytes(digestLength); + } + + const attestations: OtsAttestation[] = []; + r.timestamp(Buffer.from(digest), 0, attestations); + if (!r.exhausted) throw new OtsParseError(`${data.length - r.offset} trailing bytes after the timestamp`); + return attestations; +} + +/** + * Read and parse a proof file. + * + * The size is checked before the read, because reading a hostile file into memory to + * discover it is too large is the exhaustion this cap exists to prevent. + * + * @throws OtsParseError when the file is oversized, unreadable, or does not parse. + */ +export function parseOtsProofFile(path: string, digest: Buffer): OtsAttestation[] { + let size: number; + try { + size = statSync(path).size; + } catch (err) { + throw new OtsParseError(`proof unreadable: ${(err as Error).message}`); + } + if (size > OTS_LIMITS.maxFileBytes) { + throw new OtsParseError(`proof file is ${size} bytes, exceeding the ${OTS_LIMITS.maxFileBytes} byte cap`); + } + return parseOtsProof(readFileSync(path), digest); +} diff --git a/tests/audit-anchor-service.test.ts b/tests/audit-anchor-service.test.ts index 2249e8a..6f370eb 100644 --- a/tests/audit-anchor-service.test.ts +++ b/tests/audit-anchor-service.test.ts @@ -1,10 +1,11 @@ import { afterEach, describe, expect, it } from "@jest/globals"; -import { appendFileSync, existsSync, mkdtempSync, readFileSync, renameSync, rmSync, writeFileSync } from "fs"; +import { appendFileSync, cpSync, existsSync, mkdtempSync, readFileSync, renameSync, rmSync, writeFileSync } from "fs"; import { tmpdir } from "os"; import { join } from "path"; import { runAnchorPass, runVerify, resolvePaths } from "../src/audit/anchor-service"; import { readManifest } from "../src/audit/rotation"; import type { HttpPoster } from "../src/audit/anchor"; +import { pendingProof } from "./ots-fixtures"; /** * The wired anchor path. @@ -45,9 +46,15 @@ function write(path: string, count: number, seed = "h"): void { writeFileSync(path, Array.from({ length: count }, (_, i) => rec(i, seed)).join("")); } -/** A calendar that always answers with a plausible proof. */ +/** + * A calendar that answers with a genuine pending proof. + * + * The submitted digest is the request body, and every answer names a distinct calendar, so + * repeated passes produce distinct proof bytes and an overwrite would be visible. + */ +let calendarCall = 0; const okPoster: HttpPoster = { - post: async () => ({ status: 200, body: Buffer.from("proofbytes-ots") }), + post: async () => ({ status: 200, body: pendingProof(`https://calendar-${++calendarCall}.example.com`) }), }; const deadPoster: HttpPoster = { post: async () => { @@ -112,10 +119,11 @@ describe("anchor pass", () => { write(audit, 10); const r = resolvePaths({ auditPath: audit }); - // Distinct bodies, so an overwrite is detected rather than merely suspected. + // Distinct bodies, so an overwrite is detected rather than merely suspected. Each is a + // real proof, because the writer refuses a body it cannot parse. let call = 0; const distinctProofs: HttpPoster = { - post: async () => ({ status: 200, body: Buffer.from(`proof-${++call}`) }), + post: async () => ({ status: 200, body: pendingProof(`https://calendar-${++call}.example.com`) }), }; // Two passes six hours apart, the interval a scheduler would use. let t = Date.parse("2026-01-01T00:00:00.000Z"); @@ -135,8 +143,8 @@ describe("anchor pass", () => { expect(a).toBeDefined(); expect(b).toBeDefined(); expect(a).not.toBe(b); - expect(readFileSync(a, "utf8")).toBe("proof-1"); - expect(readFileSync(b, "utf8")).toBe("proof-2"); + expect(readFileSync(a)).toEqual(pendingProof("https://calendar-1.example.com")); + expect(readFileSync(b)).toEqual(pendingProof("https://calendar-2.example.com")); // Both log records still point at a file that is there, so either anchor can be // checked on its own. @@ -462,4 +470,114 @@ describe("verify", () => { expect(anchored?.ok).toBe(false); expect(anchored?.problems.join(" ")).toMatch(/live-tail-mismatch/); }); + + it("detects an anchor record whose digest does not describe the checkpoint it carries", async () => { + // One edit to a field nobody recomputed. The signature still verifies and the proof + // still parses, because nothing tied the proof to a particular checkpoint: the digest + // is that tie. Left unchecked, a forger points a record at a checkpoint whose state + // was never timestamped and keeps a proof that attests to something else entirely. + const d = tmp(); + const audit = join(d, "audit.jsonl"); + write(audit, 8); + await runAnchorPass({ auditPath: audit }, () => new Date(), okPoster); + + const r = resolvePaths({ auditPath: audit }); + const line = JSON.parse(readFileSync(r.anchorLogPath, "utf8").trim()); + const claimed = `0${line.digest.slice(1)}`; + writeFileSync(r.anchorLogPath, JSON.stringify({ ...line, digest: claimed }) + "\n"); + + const anchored = runVerify({ auditPath: audit }).layers.find((l) => l.name === "anchored"); + expect(anchored?.ok).toBe(false); + expect(anchored?.problems.join(" ")).toMatch(/digest-mismatch/); + }); + + it("refuses a confirmed claim with no proof bytes behind it", async () => { + // The worst of the acceptance gaps. `status` is what the record says about itself, so + // counting it means a line of JSON claiming Bitcoin-grade evidence passes with an + // empty proof directory. Trusting a self-reported status is the overclaim this whole + // layer exists to refuse. + const d = tmp(); + const audit = join(d, "audit.jsonl"); + write(audit, 8); + await runAnchorPass({ auditPath: audit }, () => new Date(), okPoster); + + const r = resolvePaths({ auditPath: audit }); + const line = JSON.parse(readFileSync(r.anchorLogPath, "utf8").trim()); + writeFileSync(r.anchorLogPath, JSON.stringify({ ...line, status: "confirmed" }) + "\n"); + rmSync(line.proofPath); + + const report = runVerify({ auditPath: audit }); + const anchored = report.layers.find((l) => l.name === "anchored"); + expect(report.confirmed).toBe(1); + expect(anchored?.ok).toBe(false); + expect(anchored?.problems.join(" ")).toMatch(/proof-missing/); + }); + + it("refuses a proof file that exists but is empty", async () => { + // Truncating a proof to nothing leaves the recorded path resolving, so an existence + // check alone passes. Zero bytes attest to nothing. + const d = tmp(); + const audit = join(d, "audit.jsonl"); + write(audit, 8); + const pass = await runAnchorPass({ auditPath: audit }, () => new Date(), okPoster); + writeFileSync(pass.records?.[0].proofPath as string, ""); + + const anchored = runVerify({ auditPath: audit }).layers.find((l) => l.name === "anchored"); + expect(anchored?.ok).toBe(false); + expect(anchored?.problems.join(" ")).toMatch(/proof-missing/); + }); + + it("refuses a proof whose bytes do not parse", async () => { + // Cut inside a length prefix, as the corpus forgery does. Unopened, a proof is a file + // name, and an anchor backed by a file name is a claim that an HTTP request happened. + const d = tmp(); + const audit = join(d, "audit.jsonl"); + write(audit, 8); + const pass = await runAnchorPass({ auditPath: audit }, () => new Date(), okPoster); + const proofPath = pass.records?.[0].proofPath as string; + const full = readFileSync(proofPath); + writeFileSync(proofPath, full.subarray(0, full.length - 5)); + + const anchored = runVerify({ auditPath: audit }).layers.find((l) => l.name === "anchored"); + expect(anchored?.ok).toBe(false); + expect(anchored?.problems.join(" ")).toMatch(/proof-parse-error/); + // A damaged proof is an anchoring failure and nothing else, so the linkage verdict it + // says nothing about must not move with it. + expect(runVerify({ auditPath: audit }).layers.find((l) => l.name === "linked")?.ok).toBe(true); + }); + + it("does not demand a proof from a submission that never reached a calendar", async () => { + // A recorded failure has no proof to point at, and it is already counted as failed. + // Reporting a missing proof on top would blame the operator's evidence for a third + // party being unreachable, and bury the failure that actually happened. + const d = tmp(); + const audit = join(d, "audit.jsonl"); + write(audit, 5); + await runAnchorPass({ auditPath: audit }, () => new Date(), deadPoster); + + const report = runVerify({ auditPath: audit }); + const anchored = report.layers.find((l) => l.name === "anchored"); + expect(report.failed).toBe(1); + expect(anchored?.problems.join(" ")).not.toMatch(/proof-missing/); + expect(anchored?.problems.join(" ")).not.toMatch(/proof-parse-error/); + }); + + it("finds a proof by its recorded base name after the evidence is copied elsewhere", async () => { + // An operator checks evidence on a machine that is not the one that wrote it, so the + // absolute path in the record points at a directory that does not exist here. The + // base name inside the proof directory is what survives the copy; failing here would + // report every anchor as unproven on any host but the original. + const origin = tmp(); + const copy = tmp(); + write(join(origin, "audit.jsonl"), 9); + await runAnchorPass({ auditPath: join(origin, "audit.jsonl") }, () => new Date(), okPoster); + cpSync(origin, copy, { recursive: true }); + + const anchored = runVerify({ auditPath: join(copy, "audit.jsonl") }).layers.find((l) => l.name === "anchored"); + expect(anchored?.problems).toEqual([]); + expect(anchored?.ok).toBe(true); + // And it read the proof rather than skipping it: the attestation count comes from the + // bytes, not from the record's status field. + expect(anchored?.detail).toMatch(/proofs carry 1 calendar and 0 bitcoin attestation\(s\)/); + }); }); diff --git a/tests/audit-anchor.test.ts b/tests/audit-anchor.test.ts index 4b56c2e..9568b89 100644 --- a/tests/audit-anchor.test.ts +++ b/tests/audit-anchor.test.ts @@ -1,5 +1,5 @@ import { afterEach, describe, expect, it } from "@jest/globals"; -import { appendFileSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "fs"; +import { appendFileSync, mkdtempSync, readdirSync, readFileSync, rmSync, writeFileSync } from "fs"; import { tmpdir } from "os"; import { join } from "path"; import { @@ -10,6 +10,7 @@ import { type HttpPoster, } from "../src/audit/anchor"; import { loadOrCreateKeys, signCheckpoint } from "../src/audit/signing"; +import { pendingProof } from "./ots-fixtures"; /** * Off-box anchoring. @@ -58,7 +59,7 @@ describe("anchoring", () => { // the aggregated root. The returned bytes ARE the proof and must be written, or the // anchor cannot later be verified or upgraded to a full attestation. const dir = tmp(); - const proofBytes = Buffer.from("f00891c256e9dade", "hex"); + const proofBytes = pendingProof("https://alice.btc.calendar.opentimestamps.org"); const poster: HttpPoster = { async post() { return { status: 200, body: proofBytes }; } }; const cp = checkpoint(); const r = await anchorToOpenTimestamps(cp, poster, () => new Date(), dir); @@ -79,6 +80,29 @@ describe("anchoring", () => { expect(r.proofPath).toBeUndefined(); }); + it("refuses a 200 whose body is not a parseable proof, instead of filing it as evidence", async () => { + // A broken calendar, or an on-path answer, returns bytes that lead nowhere. Keeping + // them would put a file on disk that verify later reports as a corrupt proof, telling + // the operator their evidence was damaged when it was never a proof. A recorded + // failure is the honest outcome, and nothing is written for a forger to point at. + const dir = tmp(); + const r = await anchorToOpenTimestamps(checkpoint(), ok("not a merkle path"), () => new Date(), dir); + expect(r.error).toMatch(/unparseable proof/); + expect(r.proofPath).toBeUndefined(); + expect(readdirSync(dir)).toEqual([]); + }); + + it("refuses a proof that parses but reaches no attestation", async () => { + // Operations alone attest to nothing. Accepting them would count an anchor whose + // proof names no calendar and no block. + const noAttestation: HttpPoster = { + async post() { return { status: 200, body: Buffer.from("f0081122334455667788", "hex") }; }, + }; + const r = await anchorToOpenTimestamps(checkpoint(), noAttestation, () => new Date(), tmp()); + expect(r.error).toMatch(/reaches no attestation|unparseable proof/); + expect(r.proofPath).toBeUndefined(); + }); + it("opentimestamps reports all calendars failing", async () => { const r = await anchorToOpenTimestamps(checkpoint(), failing(500)); expect(r.error).toMatch(/all calendars failed/); diff --git a/tests/audit-ots-proof.test.ts b/tests/audit-ots-proof.test.ts new file mode 100644 index 0000000..1344791 --- /dev/null +++ b/tests/audit-ots-proof.test.ts @@ -0,0 +1,191 @@ +import { describe, expect, it } from "@jest/globals"; +import { mkdtempSync, rmSync, writeFileSync } from "fs"; +import { tmpdir } from "os"; +import { join } from "path"; +import { afterEach } from "@jest/globals"; +import { + OTS_LIMITS, + OtsParseError, + parseOtsProof, + parseOtsProofFile, + type OtsAttestation, +} from "../src/audit/ots-proof"; +import { bitcoinProof, otsContainer, pendingProof } from "./ots-fixtures"; + +/** + * OpenTimestamps proof parsing. + * + * The property under test: proof bytes are read as the statement they carry, and nothing + * a hostile file can say makes the parser do unbounded work. Both halves matter. A parser + * that accepts anything turns an anchor into a claim that a file exists, and a parser that + * rejects everything is worse, because it passes every forgery test while breaking every + * real deployment. So the worked examples from the format are asserted alongside the caps. + */ + +const dirs: string[] = []; +function tmp(): string { + const d = mkdtempSync(join(tmpdir(), "aw-ots-")); + dirs.push(d); + return d; +} +afterEach(() => { + while (dirs.length) rmSync(dirs.pop() as string, { recursive: true, force: true }); +}); + +/** The digest of the worked example in section 3.6 of the format. */ +const DIGEST = Buffer.from("d39f84ad3463447e33fc23d9df11fe8052912c48e0d1c2f3eeb09bc3751edc94", "hex"); + +describe("ots proof parsing", () => { + it("reads the format's worked pending proof, including the calendar it names", () => { + // The bytes and the expected reading are both from the specification, so this fails + // if the parser drifts from the document rather than merely from itself. + const ops = Buffer.from( + "f0081122334455667788080083dfe30d2ef90c8e2e2d68747470733a2f2f616c6963652e6274632e" + + "63616c656e6461722e6f70656e74696d657374616d70732e6f7267", + "hex", + ); + expect(ops).toHaveLength(67); + expect(parseOtsProof(ops, DIGEST)).toEqual([ + { kind: "pending", uri: "https://alice.btc.calendar.opentimestamps.org" }, + ]); + }); + + it("reads a Bitcoin attestation as a height plus the value the operations derive", () => { + // The format's second worked example. The derived value is what an operator compares + // against a block merkle root, so getting it wrong would hand them the wrong number + // to check and a false sense that they checked something. + const attestations = parseOtsProof(Buffer.from("f008112233445566778808000588960d73d7190103d0f033", "hex"), DIGEST); + expect(attestations).toEqual([ + { + kind: "bitcoin", + height: 850000, + value: Buffer.from("a422fc26d26edb0ea1b4a0b2b421d0d0e7e8d60c814db3d654a5fa2130c0ae00", "hex"), + }, + ]); + }); + + it("reads a full .ots container and a bare calendar response identically", () => { + // A verifier distinguishes the two shapes by the magic bytes and applies the same + // operations to the anchor record's digest either way. + const ops = pendingProof("https://calendar.example.com"); + expect(parseOtsProof(otsContainer(DIGEST, ops), DIGEST)).toEqual(parseOtsProof(ops, DIGEST)); + }); + + it("applies the operations to the anchor record's digest, not to a digest the file supplies", () => { + // The record is what ties a proof to a checkpoint. A container header that could + // override the starting message would let a proof for one digest vouch for another. + const other = Buffer.alloc(32, 0xab); + const ops = bitcoinProof(700000); + const fromRecord = parseOtsProof(otsContainer(other, ops), DIGEST) as [ + Extract, + ]; + const fromHeader = parseOtsProof(ops, other) as [Extract]; + expect(fromRecord[0].value.equals(fromHeader[0].value)).toBe(false); + }); + + it("follows both sides of a fork from the message as it stands", () => { + // One submission can reach a calendar and a Bitcoin attestation on separate branches. + // A parser that followed one branch would silently drop half the evidence. + const proof = Buffer.concat([ + Buffer.from([0xff]), + pendingProof("https://alice.example.com"), + bitcoinProof(850000), + ]); + expect(parseOtsProof(proof, DIGEST).map((a) => a.kind)).toEqual(["pending", "bitcoin"]); + }); + + it("skips an attestation type it does not know rather than calling the proof corrupt", () => { + // Unknown attestations are neither proof nor failure, which is why the payload carries + // a length. Treating one as corruption would make a future OTS release unverifiable. + const unknown = Buffer.concat([ + Buffer.from([0x00]), + Buffer.from("0102030405060708", "hex"), + Buffer.from([0x04, 0xde, 0xad, 0xbe, 0xef]), + ]); + expect(parseOtsProof(unknown, DIGEST)).toEqual([]); + }); + + it("rejects a truncated proof instead of accepting the prefix it managed to read", () => { + // The forgery in the corpus: cut the file inside a length prefix and a verifier that + // stops at the first short read reports an anchor backed by a partial file. + const full = pendingProof("https://alice.btc.calendar.opentimestamps.org/timestamp"); + expect(() => parseOtsProof(full.subarray(0, full.length - 4), DIGEST)).toThrow(OtsParseError); + }); + + it("rejects trailing bytes after the timestamp", () => { + // A proof is exactly the operations that reach an attestation. Unconsumed bytes mean + // the file is not that, which is also how a second stream appended to a real proof is + // caught rather than ignored. + expect(() => parseOtsProof(Buffer.concat([bitcoinProof(1), Buffer.from([0x08])]), DIGEST)).toThrow( + /trailing bytes/, + ); + }); + + it("rejects an unknown operation tag", () => { + expect(() => parseOtsProof(Buffer.from([0x42]), DIGEST)).toThrow(/unknown op tag 0x42/); + }); + + it("declines ripemd160 and keccak256 rather than guessing at the message", () => { + // Declining is the honest answer when the primitive is not reliably present in this + // runtime, and it is what the independent verifier does, so the two agree. + expect(() => parseOtsProof(Buffer.from([0x03]), DIGEST)).toThrow(/not supported/); + expect(() => parseOtsProof(Buffer.from([0x67]), DIGEST)).toThrow(/not supported/); + }); + + it("stops a varint made of continuation bytes instead of reading forever", () => { + // A hostile file's cheapest attack: no payload at all, just a run of high bits. + const endless = Buffer.concat([Buffer.from([0xf0]), Buffer.alloc(64, 0x80)]); + expect(() => parseOtsProof(endless, DIGEST)).toThrow(/varint exceeds length cap/); + }); + + it("refuses a length prefix larger than any legitimate proof element", () => { + // Without the cap this asks for a 16 MiB read from a 4 byte file, which is a memory + // spike a forger gets for free by editing three bytes. + const huge = Buffer.concat([Buffer.from([0xf0]), Buffer.from([0x80, 0x80, 0x80, 0x08])]); + expect(() => parseOtsProof(huge, DIGEST)).toThrow(/exceeds cap/); + }); + + it("refuses a fork chain nested past the depth cap instead of exhausting the stack", () => { + // Parsing is recursive, so nesting depth is attacker-chosen call depth. A stack + // overflow in the verifier suppresses the verdict, which is what a forger wants. + const nested = Buffer.concat([Buffer.alloc(OTS_LIMITS.maxDepth + 2, 0x08), bitcoinProof(1)]); + expect(() => parseOtsProof(nested, DIGEST)).toThrow(/depth cap/); + }); + + it("refuses a proof whose message grows past the working cap", () => { + // Hexlify doubles the message. Repeated, it turns a small file into a large buffer. + const growing = Buffer.concat([Buffer.alloc(24, 0xf3), bitcoinProof(1)]); + expect(() => parseOtsProof(growing, DIGEST)).toThrow(/grew past/); + }); + + it("refuses a proof larger than the total size cap without reading it", () => { + const oversized = join(tmp(), "big.ots"); + writeFileSync(oversized, Buffer.alloc(OTS_LIMITS.maxFileBytes + 1)); + expect(() => parseOtsProofFile(oversized, DIGEST)).toThrow(/exceeding the/); + }); + + it("reports a missing proof file as a parse failure rather than throwing an fs error", () => { + // The caller reports on evidence, so an unreadable file has to arrive as the same + // kind of answer as an unparseable one instead of aborting the whole verify. + expect(() => parseOtsProofFile(join(tmp(), "absent.ots"), DIGEST)).toThrow(OtsParseError); + }); + + it("parses a real calendar response byte for byte", () => { + // Captured from an OpenTimestamps calendar: two aggregation appends, a 32 byte + // prepend, and a pending attestation. A parser that only handles the synthetic + // fixtures above would pass every test here and reject production evidence. + const real = Buffer.from( + "f0084934fe6351fdafa008f01005b72e01b9aae261174d3d82a375aaaa08f1201599b555e14e3656" + + "b345a00d8d3c17cd4d380f36545296227c7f049fb4a4611b08f1046a72e696f0083dfb21ca601a021" + + "10083dfe30d2ef90c8e2e2d68747470733a2f2f616c6963652e6274632e63616c656e6461722e6f70" + + "656e74696d657374616d70732e6f7267", + "hex", + ); + expect(real).toHaveLength(137); + const proofPath = join(tmp(), "real.ots"); + writeFileSync(proofPath, real); + expect(parseOtsProofFile(proofPath, Buffer.alloc(32, 0x11))).toEqual([ + { kind: "pending", uri: "https://alice.btc.calendar.opentimestamps.org" }, + ]); + }); +}); diff --git a/tests/ots-fixtures.ts b/tests/ots-fixtures.ts new file mode 100644 index 0000000..510bc6e --- /dev/null +++ b/tests/ots-fixtures.ts @@ -0,0 +1,62 @@ +/** + * OpenTimestamps proof bytes, hand-assembled from section 3.7 of docs/audit-format.md. + * + * Built byte by byte rather than by calling the parser's own helpers, so a test proves the + * parser reads the FORMAT and not merely its own round trip. Shared because three suites + * need the same shapes and three hand-copied byte strings would drift apart. + * + * The operations do not depend on the digest: it is the starting message, so the same + * bytes are a valid proof for any anchor record. + */ + +/** Unsigned little-endian base 128, least significant group first, high bit as continuation. */ +function varint(n: number): Buffer { + const out: number[] = []; + let v = n; + do { + out.push((v % 128) | (v >= 128 ? 0x80 : 0)); + v = Math.floor(v / 128); + } while (v > 0); + return Buffer.from(out); +} + +/** A varint length then the bytes, the `varbytes` of the grammar. */ +function varbytes(b: Buffer): Buffer { + return Buffer.concat([varint(b.length), b]); +} + +/** Append eight bytes, then sha256. The aggregation step every calendar answer starts with. */ +const AGGREGATE = Buffer.concat([ + Buffer.from([0xf0]), + varbytes(Buffer.from("1122334455667788", "hex")), + Buffer.from([0x08]), +]); + +const ATTESTATION = Buffer.from([0x00]); +const TAG_PENDING = Buffer.from("83dfe30d2ef90c8e", "hex"); +const TAG_BITCOIN = Buffer.from("0588960d73d71901", "hex"); + +/** What a calendar returns for a fresh submission: aggregation, then a pending attestation. */ +export function pendingProof(calendarUri: string): Buffer { + return Buffer.concat([ + AGGREGATE, + ATTESTATION, + TAG_PENDING, + varbytes(varbytes(Buffer.from(calendarUri, "utf8"))), + ]); +} + +/** An upgraded proof: the same aggregation, then a Bitcoin attestation naming a block. */ +export function bitcoinProof(height: number): Buffer { + return Buffer.concat([AGGREGATE, ATTESTATION, TAG_BITCOIN, varbytes(varint(height))]); +} + +/** The 31 magic bytes, version, and file hash-op header that wrap a full `.ots` file. */ +export function otsContainer(digest: Buffer, operations: Buffer): Buffer { + return Buffer.concat([ + Buffer.from("004f70656e54696d657374616d7073000050726f6f6600bf89e2e884e89294", "hex"), + Buffer.from([0x01, 0x08]), + digest, + operations, + ]); +} diff --git a/verifier/README.md b/verifier/README.md index 41369b0..f17674c 100644 --- a/verifier/README.md +++ b/verifier/README.md @@ -140,29 +140,28 @@ it checks. From the repository root: cd verifier && go build -o agentwall-verify . && cd .. node scripts/conformance.js - 26 cases, typescript and go: 22 agreed, 4 declared divergence(s), 0 failure(s) + 26 cases, typescript and go: 25 agreed, 1 declared divergence(s), 0 failure(s) `go test ./...` in this directory runs the unit tests plus a corpus walk that asserts every case's `expected.json` against this verifier alone. ## Where the two verifiers disagree today -Four corpus cases get different verdicts from the two verifiers. In three of them the bundled -TypeScript verifier accepts evidence the format rejects, which makes this verifier the stricter of -the two today. In the fourth both reject the file, and the bundled verifier blames the chain instead -of naming the torn tail: +One corpus case gets different verdicts from the two verifiers. Both reject the file, and the bundled +verifier blames the chain instead of naming the torn tail: | Case | The edit | Bundled TypeScript verifier | This verifier | | --- | --- | --- | --- | -| `b9-anchor-digest-altered` | the anchor record's `digest` field altered | `anchored` PASS, exit 0. It reports the digest the record claims was submitted and never recomputes one from the checkpoint the record embeds | `digest-mismatch`, `anchored` FAIL, exit 1 | -| `b10-proof-truncated` | the OTS proof truncated inside a length prefix | `anchored` PASS, exit 0. It never opens a proof file, so a proof that cannot be parsed still counts as an anchor | `proof-parse-error`, `anchored` FAIL, exit 1 | | `b11-torn-tail` | a partial final line, as a hard kill leaves behind | `chained` FAIL, exit 1. It condemns the whole chain over one partial write | `torn-tail` reported distinctly, `chained` PASS, exit 1 because nothing is anchored | -| `b13-confirmed-without-proof` | an anchor claiming `confirmed` with its proof file deleted | `anchored` PASS, exit 0. It counts the status field, so a claim of confirmation passes with no proof bytes behind it | `proof-missing`, `anchored` FAIL, exit 1 | -Those three acceptance gaps are limits of the bundled verifier as it ships today. The harness -declares all four in -`scripts/conformance.js`, prints them on every run, and fails if one of them starts agreeing, so the -list cannot rot into a set of excuses. +That naming gap is a limit of the bundled verifier as it ships today. The harness declares it in +`scripts/conformance.js`, prints it on every run, and fails if it starts agreeing, so the list cannot +rot into a set of excuses. + +On the `anchored` layer the two verifiers agree case for case. The bundled one recomputes each anchor +record's digest from the embedded checkpoint, requires non-empty proof bytes behind any submission +that reached a calendar, and parses the proof against the submitted digest under the same caps as this +one, so `digest-mismatch`, `proof-missing`, and `proof-parse-error` are reported by both. ## Exit codes