Skip to content

feat(verify): stamp reproducibility metadata into evidence JSON - #10

Open
BoozeLee wants to merge 1 commit into
mainfrom
fix/embed-repro-metadata
Open

BoozeLee wants to merge 1 commit into
mainfrom
fix/embed-repro-metadata

Conversation

@BoozeLee

Copy link
Copy Markdown
Collaborator

What this PR adds

harness verify --embed-repro now writes a stable reproducibility block into the evidence JSON it produces, alongside the existing task, goal, accept, results, files, stat, verdict, time fields.

{
  "captured_at": "2026-10-11T10:23:32+00:00",
  "harness_sha": "cd10f245519756198fe9c5e37baf414cdeedb07f",
  "harness_version": "1.1.0",
  "python_version": "3.11.9",
  "python_implementation": "CPython",
  "platform": "Linux-7.2.9-zen1-1-zen-x86_64-with-glibc2.44",
  "host": "omarchy",
  "packages_sha256": null,
  "packages_count": null,
  "github_sha": null,
  "github_run_id": null,
  "github_run_attempt": null
}

Stable across consecutive invocations on the same machine: harness_sha, harness_version, python_version, platform, host, packages_sha256, packages_count. The only wall-clock field is captured_at. The GitHub-Actions fields (github_sha, github_run_id, github_run_attempt) come from os.environ and pick up automatically in CI.

Why it matters

Evidence is the single artifact that flows from harness verify to reviewers, dashboards, and PR-merge gates. Today, an evidence JSON says what passed; tomorrow's reviewer also needs which build of harness, on which Python, on which host passed. The repro block makes evidence self-describing — six months later, anyone holding the JSON can prove what runtime produced it. This is the smallest possible cross-cutting substrate for the larger cross-repo verification story in the org's Q4 roadmap (STRATEGY.md in Quattro-Commas.github.io).

Files

File Change Why
harness/repro.py (new) +108 lines embed_repro_metadata(root, *, harness_sha=None) -> dict. Total contract — never raises; missing data lands as None.
harness/verify.py +3 lines cmd_verify writes ev["repro"] = embed_repro_metadata(r) when a.embed_repro is set.
harness/cli.py +2 lines --embed-repro flag on the verify subparser (default off, no behaviour change for existing callers).
harness/evidence.py +24 lines render adds an optional Reproducibility section to the HTML evidence.
tests/test_repro.py (new) +123 lines 10 tests pin the shape, type, stability, override, GitHub-env propagation, and never-raise contract.

Tests

  • uv run pytest -q tests/test_repro.py → 10 passed
  • uv run pytest -q (full suite) → ~104 passed
  • uv run ruff check (4 touched files) → clean
  • uv run mypy harness/repro.py → no issues

Behaviour change

None for existing callers. --embed-repro is opt-in; default off. A re-run of any current harness verify invocation produces byte-identical evidence.

Follow-ups (out of scope here)

  • runner.py:_verify_run parity — the same flag wired through serve/the API. Today, only the CLI path emits repro; the dashboard inherits nothing until that lands.
  • Plumbing through the task contract so harness task start --embed-repro makes repro part of the contract (no longer passed per-invocation).

harness verify --embed-repro now attaches a stable 'repro' block to the
evidence it writes:

  {
    'captured_at': '...',
    'harness_sha': '<40-hex>',
    'harness_version': 'X.Y.Z',
    'python_version': '3.11.9',
    'python_implementation': 'CPython',
    'platform': '...',
    'host': '...',
    'packages_sha256': '<sha256 or null>',
    'packages_count': N,
    'github_sha': '<sha or null>',
    'github_run_id': '... or null',
    'github_run_attempt': '... or null'
  }

Stable on the same machine: harness_sha, harness_version, python_version,
platform, host, packages_sha256, packages_count. captured_at is the only
wall-clock field. github_* fields pick up GITHUB_* env vars when running
under Actions.

implementation
  - harness/repro.py (new): embed_repro_metadata(root, *, harness_sha=None).
    Total contract: never raises; missing data lands as None. harness_sha
    resolves by walking up from the installed harness module's origin
    looking for '.git' and 'git rev-parse HEAD'. packages_sha256 derives
    from 'python -m pip freeze'.
  - harness/verify.py cmd_verify: when --embed-repro is on, ev['repro']
    is set before write_evidence. Same code path as run-direct and
    runner.py:_verify_run is a follow-up (tracked separately).
  - harness/cli.py: --embed-repro flag on the verify subparser (default off).
  - harness/evidence.py render: optional 'Reproducibility' section in the
    HTML evidence so reviewers can scan it without opening JSON.

tests
  - tests/test_repro.py (new, 10 tests, all pass): shape, well-typedness,
    field stability across consecutive calls, override harness_sha,
    github env propagation, total/never-raise contract, root arg symmetry.

verified
  - uv run pytest -q tests/test_repro.py  → 10 passed
  - uv run pytest -q (full suite)         → ~104 passed
  - uv run ruff check (4 touched files)   → clean
  - uv run mypy harness/repro.py          → no issues

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant