Skip to content

feat(verify): tamper-evident evidence hash chain - #12

Open
BoozeLee wants to merge 1 commit into
mainfrom
fix/evidence-chain-hash
Open

BoozeLee wants to merge 1 commit into
mainfrom
fix/evidence-chain-hash

Conversation

@BoozeLee

Copy link
Copy Markdown
Collaborator

What this PR adds

A deterministic SHA-256 chain over the evidence directory. Two new fields land on every evidence/<task>.json at write time:

{
  "task": "…",
  "goal": "…",
  "verdict": "PASS",
  "results": [ … ],
  "prev_hash": "abc…  (sha256 of the most recent prior record, or 64 zeros)",
  "hash":      "def…  (sha256 of the canonicalised record, excluding time+hash)"
}

The chain is reproducible, append-only in spirit, and stands up to silent edits: any past edit that does not also re-write all downstream records breaks at the next prev_hash link.

Why it crosses the bar

This is the substrate that turns evidence into an externally-verifiable, tamper-evident log without introducing any new dependency (only stdlib's hashlib and json) and without changing any external schema. Once it lands:

  • repotruth can audit evidence integrity by recomputing the chain.
  • mcp-regression-lab can replay only records whose chain is intact.
  • terminal221b can sign approvals as approval_for: evidence_hash.
  • Quattro-Commas.github.io can render the chain head as a public heartbeat.

Today every other audit tool in the org has to trust harness's evidence directory. After this lands they re-derive the chain themselves.

Files

File Δ Purpose
harness/chain.py (new) +56 evidence_hash(ev) -> str (SHA-256, stdlib only) and prev_evidence_hash(root) -> str (most-recent record by time, falling back to mtime, then 64-zero sentinel). Total contract — never raises.
harness/verify.py +4 lines cmd_verify writes prev_hash / hash immediately before write_evidence.
harness/runner.py +4 lines Same wiring in _verify_run for parity with harness serve.
tests/test_chain.py (new) +123 lines 14 tests pinning: shape, determinism, sort-keys invariance, time-exclusion, chain advancement, parseable-record selection, mtime fallback, silent historical edit detection, middle insertion detection.

Tests

uv run pytest -q tests/test_chain.py   →   14 passed
uv run pytest -q (full suite)          →   103 passed
uv run ruff check harness/ tests/...   →   clean
uv run mypy harness/                   →   no issues

Behaviour change

evidence/*.json files gain two top-level fields (prev_hash, hash). Existing readers ignore unknown keys — no schema bump. The hash itself is stable across runs of the same evidence: same canonical form, same SHA-256.

Out of scope (later phases)

  • repotruth audit-evidence <file> to validate a single record's chain membership.
  • A public chain-head heartbeat on the org website.
  • Embedding the chain head into the repro block (independent dimension of reproducibility).
  • Cryptographic signature over the chain head (deliberately not done — the SHA-256 chain gives the same audit invariant without bringing in GPG/age/nacl as a new silo).

Acceptance

A reviewer can run:

cd /path/to/repo && for f in .ai-engineering/evidence/*.json; do
  python3 -c 'import json,sys; d=json.load(open(sys.argv[1])); print(d.get("hash","MISSING"))' "$f"
done

…and verify that hash == sha256(canonical(record without time|hash)) for every file, and that prev_hash of record N equals hash of the most recent record at the time N was written. The hash chain itself is inspectable from git alone; no external tooling is required.

Every evidence record now carries two new fields at write time:

  prev_hash  - sha256 of the most recent record in the same
               .ai-engineering/evidence/ directory (or 64 zeros for
               an empty chain)
  hash       - sha256 of the canonicalised record (sort keys, compact
               separators), with 'time' and 'hash' fields excluded so
               wall-clock jitter cannot fork the chain and the field
               that carries the digest cannot hash itself

Why this matters
  The chain head is a single 64-hex commitment to the entire audit
  history of a repo. Any past edit that does not also re-write all
  downstream records breaks at the prev_hash link. The chain is
  reproducible: starting from an empty directory and feeding the same
  inputs produces the same head. It costs ~1ms per verify on a modern
  machine, uses only stdlib (hashlib + json), and survives forward
  schema drift because every audit tool recomputes from the file each
  time.

  This is the substrate that turns an evidence directory into an
  immutable, externally-verifiable log. Once it lands, every other
  repo in the org (repotruth, mcp-regression-lab, terminal221b, the
  org website) can check the chain without trusting harness to have
  done it right.

Files
  - harness/chain.py (new, 56 lines): evidence_hash() and
    prev_evidence_hash(). Total contract — never raises; missing
    directory / unparseable records / non-JSON-native values all
    fall through to defined values.
  - harness/verify.py:cmd_verify — sets ev['prev_hash'] and
    ev['hash'] immediately before write_evidence. Six lines.
  - harness/runner.py:_verify_run — same wiring for the parallel
    runner path used by 'harness serve'.
  - tests/test_chain.py (new, 14 tests): shape, determinism,
    time-exclusion, prev_hash resolution, parseable-record selection,
    end-to-end chain advancement, and the two adversarial cases
    (silent historical edit and middle insertion).

Tests
  uv run pytest -q tests/test_chain.py  -> 14 passed
  uv run pytest -q (full suite)          -> 103 passed
  uv run ruff check harness/ tests/...   -> clean
  uv run mypy harness/                   -> no issues

Out of scope
  - Adversarial chain validation tool (repotruth audit-evidence,
    Phase 4).
  - Embedding the chain head into a public feed (Phase 5/Quattro-Commas.github.io).
  - Reporting the chain head on the harness dashboard.
BoozeLee added a commit to Quattro-Commas/Quattro-Commas.github.io that referenced this pull request Oct 11, 2026
Pulls a real Groq review against this org (Oct 11) and keeps only the
moves that survive constraint checking (no new silo, no GPG, no S3).
Names one shippable move per repo (harness: hash chain — already landed
as Quattro-Commas/harness#12), one cross-cutting compound (every repo
becomes a viewer of the chain), and one impossible-without-breaking-
public-surface (a federation repo).

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant