fix(evals): enforce canonical promise ledger format by mdsmithaustin · Pull Request #199 · mdsmithaustin/pstack

mdsmithaustin · 2026-10-12T19:30:01Z

What Changed

Add canonical ledger formatting with sorted keys, atomic UTF-8 writes, a read-only --check mode, and duplicate-key rejection.
Reformat ledger.json, enforce canonical form in CI and local lint gates, and document the formatting workflow with regression coverage.
Correct the principle promise to include all 24 leaves, including principle-explain-the-number, and check its owners and count against the pinned rules.

Validation

Risk Assessment

✅ Low: The ledger.json change is a pure key reorder. A semantic diff against the base shows the only value changes are the intended principle-leaf-summaries-match fix: the claim goes from 23 to 24, and principle-explain-the-number is added to owners, matching the 24 PRINCIPLE_RULES keys. The format command meets each round-1 requirement: it encodes before opening a file, writes a mkstemp file in the same directory, applies copymode, then os.replace. Duplicate keys are rejected in the shared load/parse path. Red tests cover the surrogate, duplicate-key and CRLF cases. The check is wired into gate-check.sh lint and lint.yml.

Testing

Targeted tests and real CLI scenarios passed, including malformed-input preservation and interrupted-write recovery. Drift diagnostics produced the expected failures on altered disposable data, but were not validated live. Evidence was retained and temporary files removed; lint and CI phases were left to the outer executor.

Live validation: ✅ go - 7 of 8 scenarios driven live against the product

Scenario	Result	Live	Evidence
Check the committed ledger and audit its promise bindings successfully	✅ pass	live	Ledger CLI transcript: committed format --check and audit both exited 0.
Format nested unsorted JSON into one canonical UTF-8 file and repeat without changes	✅ pass	live	Ledger CLI transcript: independently specified canonical bytes, preserved permissions, changed inode, and unchanged bytes on repeated formatting.
Reject noncanonical input with --check while preserving the file	✅ pass	live	Ledger CLI transcript: noncanonical CRLF input produced exit 1 and the fix command without changing input bytes.
Attempt to format a lone-surrogate ledger without corrupting it	✅ pass	live	Ledger CLI transcript: UnicodeEncodeError, unchanged original SHA-256, and no temporary file.
Interrupt a real formatter during its atomic write and successfully retry	✅ pass	live	Ledger CLI transcript: SIGKILL after adjacent temporary-file creation preserved original SHA-256; subsequent format and check succeeded.
Reject duplicate keys at every tested dictionary depth without modifying the ledger	✅ pass	live	Ledger CLI transcript: root, nested, and array-contained duplicates rejected by format, format --check, audit, and owners with ValueError naming each key.
Query the corrected principle promise and receive its 24 owners	✅ pass	live	Ledger CLI transcript: owners returned 24 entries including principle-explain-the-number; persisted claim names 24 principle links.
Detect principle-owner and count drift with actionable test diagnostics	⏸️ untested	no	The prior payload established only unit-test execution against committed data and disposable ledger copies, not a live product result. This assertion message exists only in a unit test and has no sepa…

Evidence: Ledger CLI transcript and drift diagnostics

$ python3 evals/promises/ledger.py format --check

exit=0

$ python3 evals/promises/ledger.py audit

exit=0
bound only to deferred cases, so not yet run: automate-me-drafts-mode-skill-and-opens-pr, automate-me-mines-transcripts, automate-me-update-mode-mines-since-last-edit, create-verification-skill-interviews-repo, create-verification-skill-invocable, maintain-verification-skill-audits, opening-a-pr-gated-publish-never-bypasses, opening-a-pr-worktree-small-commits-link, recall-rebuilds-context-brief, reflect-three-reviewers-then-synthesizer, setup-checks-codex-multi-agent, setup-checks-personas-offers-registration, setup-checks-task-tool-switches, setup-detects-models-and-asks-budget, setup-offers-verification-skill-once, setup-reports-three-states-separately, setup-rerun-keeps-changed-roles, setup-writes-models-config, shipping-fresh-verifier-per-pr-never-author, shipping-verifies-each-pr-independently, swarm-splits-verification-by-feature-map, verification-skill-proven-once, verification-skill-written-to-agents-skills, worktree-request-honored
0 problem(s), 864 units, 234 promises, 24 deferred

$ python3 evals/promises/ledger.py format --check --ledger ~/.no-mistakes/worktrees/fbee15f974d9/01M4RX3A3X5KSGGMTVF6G7V0M2/.ledger-validation/ledger.json

exit=1
ledger.json is not in canonical form; run python3 evals/promises/ledger.py format

$ python3 evals/promises/ledger.py format --ledger ~/.no-mistakes/worktrees/fbee15f974d9/01M4RX3A3X5KSGGMTVF6G7V0M2/.ledger-validation/ledger.json

exit=0

Persisted canonical output, matching an independently written byte contract:
{
 "a": [
  {
   "a": "café",
   "z": 0
  },
  3
 ],
 "z": {
  "a": 1,
  "b": 2
 }
}


$ python3 evals/promises/ledger.py format --check --ledger ~/.no-mistakes/worktrees/fbee15f974d9/01M4RX3A3X5KSGGMTVF6G7V0M2/.ledger-validation/ledger.json

exit=0

$ python3 evals/promises/ledger.py format --ledger ~/.no-mistakes/worktrees/fbee15f974d9/01M4RX3A3X5KSGGMTVF6G7V0M2/.ledger-validation/ledger.json

exit=0

Read-only rejection, recursive sorting including dictionaries in arrays, UTF-8, LF, final newline, permission preservation, atomic inode replacement, and idempotence verified.

$ python3 evals/promises/ledger.py format --ledger <surrogate ledger>
Traceback (most recent call last):
  File "~/.no-mistakes/worktrees/fbee15f974d9/01M4RX3A3X5KSGGMTVF6G7V0M2/evals/promises/ledger.py", line 253, in <module>
    sys.exit(main())
  File "~/.no-mistakes/worktrees/fbee15f974d9/01M4RX3A3X5KSGGMTVF6G7V0M2/evals/promises/ledger.py", line 233, in main
    return format_ledger(args.ledger, args.check)
  File "~/.no-mistakes/worktrees/fbee15f974d9/01M4RX3A3X5KSGGMTVF6G7V0M2/evals/promises/ledger.py", line 70, in format_ledger
    write_atomically(path, wanted.encode("utf-8"))
UnicodeEncodeError: 'utf-8' codec can't encode character '\ud800' in position 18: surrogates not allowed
exit=1

Surrogate failure preserved original SHA-256 5cb3522bcf85f8fe5539cfc6e3714ab54d62211834482503b9922c1a087bb4db and created no temporary file.

root duplicate via format: exit=1; ValueError: duplicate key 'promises'; keep one entry and delete the other; original bytes unchanged

root duplicate via format --check: exit=1; ValueError: duplicate key 'promises'; keep one entry and delete the other; original bytes unchanged

root duplicate via audit: exit=1; ValueError: duplicate key 'promises'; keep one entry and delete the other; original bytes unchanged

root duplicate via owners --upstream ~/.no-mistakes/worktrees/fbee15f974d9/01M4RX3A3X5KSGGMTVF6G7V0M2: exit=1; ValueError: duplicate key 'promises'; keep one entry and delete the other; original bytes unchanged

nested duplicate via format: exit=1; ValueError: duplicate key '05-build:7'; keep one entry and delete the other; original bytes unchanged

nested duplicate via format --check: exit=1; ValueError: duplicate key '05-build:7'; keep one entry and delete the other; original bytes unchanged

nested duplicate via audit: exit=1; ValueError: duplicate key '05-build:7'; keep one entry and delete the other; original bytes unchanged

nested duplicate via owners --upstream ~/.no-mistakes/worktrees/fbee15f974d9/01M4RX3A3X5KSGGMTVF6G7V0M2: exit=1; ValueError: duplicate key '05-build:7'; keep one entry and delete the other; original bytes unchanged

array duplicate via format: exit=1; ValueError: duplicate key 'claim'; keep one entry and delete the other; original bytes unchanged

array duplicate via format --check: exit=1; ValueError: duplicate key 'claim'; keep one entry and delete the other; original bytes unchanged

array duplicate via audit: exit=1; ValueError: duplicate key 'claim'; keep one entry and delete the other; original bytes unchanged

array duplicate via owners --upstream ~/.no-mistakes/worktrees/fbee15f974d9/01M4RX3A3X5KSGGMTVF6G7V0M2: exit=1; ValueError: duplicate key 'claim'; keep one entry and delete the other; original bytes unchanged

Killed real formatter after adjacent temp opened: exit=-9, temp_bytes=67108886; original ledger SHA-256 unchanged: ee7500cbadb6db49bcb24150a3fb1a114783ffc121288c96797efce381e0d4d6

$ python3 evals/promises/ledger.py format --ledger ~/.no-mistakes/worktrees/fbee15f974d9/01M4RX3A3X5KSGGMTVF6G7V0M2/.ledger-validation/ledger.json

exit=0

$ python3 evals/promises/ledger.py format --check --ledger ~/.no-mistakes/worktrees/fbee15f974d9/01M4RX3A3X5KSGGMTVF6G7V0M2/.ledger-validation/ledger.json

exit=0

Formatting retry after interruption succeeded and preserved the ledger data.

$ python3 evals/promises/ledger.py owners --upstream ~/.no-mistakes/worktrees/fbee15f974d9/01M4RX3A3X5KSGGMTVF6G7V0M2 principle-leaf-summaries-match

exit=0
{
 "principle-leaf-summaries-match": {
  "owners": {
   "skills/principle-laziness-protocol/SKILL.md": "verbatim",
   "skills/principle-foundational-thinking/SKILL.md": "verbatim",
   "skills/principle-redesign-from-first-principles/SKILL.md": "verbatim",
   "skills/principle-attack-the-premise/SKILL.md": "verbatim",
   "skills/principle-subtract-before-you-add/SKILL.md": "verbatim",
   "skills/principle-minimize-reader-load/SKILL.md": "verbatim",
   "skills/principle-outcome-oriented-execution/SKILL.md": "verbatim",
   "skills/principle-experience-first/SKILL.md": "verbatim",
   "skills/principle-exhaust-the-design-space/SKILL.md": "verbatim",
   "skills/principle-build-the-lever/SKILL.md": "verbatim",
   "skills/principle-model-the-domain/SKILL.md": "verbatim",
   "skills/principle-boundary-discipline/SKILL.md": "verbatim",
   "skills/principle-type-system-discipline/SKILL.md": "verbatim",
   "skills/principle-make-operations-idempotent/SKILL.md": "verbatim",
   "skills/principle-migrate-callers-then-delete-legacy-apis/SKILL.md": "verbatim",
   "skills/principle-separate-before-serializing-shared-state/SKILL.md": "verbatim",
   "skills/principle-prove-it-works/SKILL.md": "verbatim",
   "skills/principle-fix-root-causes/SKILL.md": "verbatim",
   "skills/principle-sequence-verifiable-units/SKILL.md": "verbatim",
   "skills/principle-test-behavior-not-implementation/SKILL.md": "verbatim",
   "skills/principle-explain-the-number/SKILL.md": "verbatim",
   "skills/principle-guard-the-context-window/SKILL.md": "verbatim",
   "skills/principle-never-block-on-the-human/SKILL.md": "verbatim",
   "skills/principle-encode-lessons-in-structure/SKILL.md": "verbatim"
  },
  "possibly_upstream": true
 }
}

Published ledger contract has 24 owners, includes explain-the-number, and claims Each of the 24 principle links.

Disposable owners drift: $ python3 -m unittest -v test_static.TestStaticPromises.test_principle_promise_owners_are_the_pinned_leaves
test_principle_promise_owners_are_the_pinned_leaves (test_static.TestStaticPromises) ... FAIL

======================================================================
FAIL: test_principle_promise_owners_are_the_pinned_leaves (test_static.TestStaticPromises)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "~/.no-mistakes/worktrees/fbee15f974d9/01M4RX3A3X5KSGGMTVF6G7V0M2/.ledger-validation/drift/evals/promises/test_static.py", line 186, in test_principle_promise_owners_are_the_pinned_leaves
    self.assertEqual(
AssertionError: Lists differ: ['ski[308 chars]iple-fix-root-causes/SKILL.md', 'skills/princi[859 chars].md'] != ['ski[308 chars]iple-explain-the-number/SKILL.md', 'skills/pri[907 chars].md']

First differing element 6:
'skills/principle-fix-root-causes/SKILL.md'
'skills/principle-explain-the-number/SKILL.md'

Second list contains 1 additional elements.
First extra element 23:
'skills/principle-type-system-discipline/SKILL.md'

Diff is 1343 characters long. Set self.maxDiff to None to see it. : principle-leaf-summaries-match lists owners ['skills/principle-laziness-protocol/SKILL.md', 'skills/principle-foundational-thinking/SKILL.md', 'skills/principle-redesign-from-first-principles/SKILL.md', 'skills/principle-attack-the-premise/SKILL.md', 'skills/principle-subtract-before-you-add/SKILL.md', 'skills/principle-minimize-reader-load/SKILL.md', 'skills/principle-outcome-oriented-execution/SKILL.md', 'skills/principle-experience-first/SKILL.md', 'skills/principle-exhaust-the-design-space/SKILL.md', 'skills/principle-build-the-lever/SKILL.md', 'skills/principle-model-the-domain/SKILL.md', 'skills/principle-boundary-discipline/SKILL.md', 'skills/principle-type-system-discipline/SKILL.md', 'skills/principle-make-operations-idempotent/SKILL.md', 'skills/principle-migrate-callers-then-delete-legacy-apis/SKILL.md', 'skills/principle-separate-before-serializing-shared-state/SKILL.md', 'skills/principle-prove-it-works/SKILL.md', 'skills/principle-fix-root-causes/SKILL.md', 'skills/principle-sequence-verifiable-units/SKILL.md', 'skills/principle-test-behavior-not-implementation/SKILL.md', 'skills/principle-guard-the-context-window/SKILL.md', 'skills/principle-never-block-on-the-human/SKILL.md', 'skills/principle-encode-lessons-in-structure/SKILL.md']; upstream added or removed a principle, so edit promise principle-leaf-summaries-match in evals/promises/ledger.json: set its owners to one skills/<name>/SKILL.md per PRINCIPLE_RULES key and its claim to 'Each of the 24 principle links'

----------------------------------------------------------------------
Ran 1 test in 0.006s

FAILED (failures=1)


Disposable claim drift: $ python3 -m unittest -v test_static.TestStaticPromises.test_principle_promise_owners_are_the_pinned_leaves
test_principle_promise_owners_are_the_pinned_leaves (test_static.TestStaticPromises) ... FAIL

======================================================================
FAIL: test_principle_promise_owners_are_the_pinned_leaves (test_static.TestStaticPromises)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "~/.no-mistakes/worktrees/fbee15f974d9/01M4RX3A3X5KSGGMTVF6G7V0M2/.ledger-validation/drift/evals/promises/test_static.py", line 191, in test_principle_promise_owners_are_the_pinned_leaves
    self.assertIn(
AssertionError: 'Each of the 24 principle links' not found in 'Each of the 23 principle links on the principles page resolves to an existing leaf whose rule matches the one-line summary the guide gives it.' : principle-leaf-summaries-match lists owners ['skills/principle-laziness-protocol/SKILL.md', 'skills/principle-foundational-thinking/SKILL.md', 'skills/principle-redesign-from-first-principles/SKILL.md', 'skills/principle-attack-the-premise/SKILL.md', 'skills/principle-subtract-before-you-add/SKILL.md', 'skills/principle-minimize-reader-load/SKILL.md', 'skills/principle-outcome-oriented-execution/SKILL.md', 'skills/principle-experience-first/SKILL.md', 'skills/principle-exhaust-the-design-space/SKILL.md', 'skills/principle-build-the-lever/SKILL.md', 'skills/principle-model-the-domain/SKILL.md', 'skills/principle-boundary-discipline/SKILL.md', 'skills/principle-type-system-discipline/SKILL.md', 'skills/principle-make-operations-idempotent/SKILL.md', 'skills/principle-migrate-callers-then-delete-legacy-apis/SKILL.md', 'skills/principle-separate-before-serializing-shared-state/SKILL.md', 'skills/principle-prove-it-works/SKILL.md', 'skills/principle-fix-root-causes/SKILL.md', 'skills/principle-sequence-verifiable-units/SKILL.md', 'skills/principle-test-behavior-not-implementation/SKILL.md', 'skills/principle-explain-the-number/SKILL.md', 'skills/principle-guard-the-context-window/SKILL.md', 'skills/principle-never-block-on-the-human/SKILL.md', 'skills/principle-encode-lessons-in-structure/SKILL.md'] and claims 'Each of the 23 principle links on the principles page resolves to an existing leaf whose rule matches the one-line summary the guide gives it.'; upstream added or removed a principle, so edit promise principle-leaf-summaries-match in evals/promises/ledger.json: set its owners to one skills/<name>/SKILL.md per PRINCIPLE_RULES key and its claim to 'Each of the 24 principle links'

----------------------------------------------------------------------
Ran 1 test in 0.003s

FAILED (failures=1)


All product checks completed successfully.

Evidence: Disposable validation driver

import hashlib
import json
import os
from pathlib import Path
import shutil
import signal
import subprocess
import sys
import time

root = Path.cwd()
work = root / '.ledger-validation'
evidence = Path('~/.no-mistakes/evidence/01M4RX3A3X5KSGGMTVF6G7V0M2')
cli = [sys.executable, 'evals/promises/ledger.py']
lines = []
def log(text):
    lines.append(text)
    print(text, flush=True)
def run(args, expected=0):
    proc = subprocess.run(cli + args, capture_output=True, text=True)
    log('$ python3 evals/promises/ledger.py ' + ' '.join(args))
    log(f'exit={proc.returncode}\n{proc.stdout}{proc.stderr}'.rstrip())
    assert proc.returncode == expected, (args, proc.returncode)
    return proc

def digest(path):
    return hashlib.sha256(path.read_bytes()).hexdigest()

try:
    run(['format', '--check'])
    run(['audit'])
    p = work / 'ledger.json'
    before = b'{\r\n "z": {"b": 2, "a": 1}, "a": [{"z": 0, "a": "caf\xc3\xa9"}, 3]\r\n}\r\n'
    wanted = '{\n "a": [\n  {\n   "a": "caf\u00e9",\n   "z": 0\n  },\n  3\n ],\n "z": {\n  "a": 1,\n  "b": 2\n }\n}\n'.encode('utf-8')
    p.write_bytes(before)
    p.chmod(0o640)
    old_inode = p.stat().st_ino
    rejection = run(['format', '--check', '--ledger', str(p)], 1)
    assert 'python3 evals/promises/ledger.py format' in rejection.stdout
    assert p.read_bytes() == before
    run(['format', '--ledger', str(p)])
    assert p.read_bytes() == wanted
    assert p.stat().st_ino != old_inode
    assert p.stat().st_mode & 0o777 == 0o640
    log('Persisted canonical output, matching an independently written byte contract:\n' + p.read_text())
    inode = p.stat().st_ino
    run(['format', '--check', '--ledger', str(p)])
    run(['format', '--ledger', str(p)])
    assert p.stat().st_ino == inode and p.read_bytes() == wanted
    log('Read-only rejection, recursive sorting including dictionaries in arrays, UTF-8, LF, final newline, permission preservation, atomic inode replacement, and idempotence verified.')

    p.write_bytes(b'{"z": "\\ud800", "a": 1}\n')
    old_hash = digest(p)
    folder = sorted(x.name for x in work.iterdir())
    proc = subprocess.run(cli + ['format', '--ledger', str(p)], capture_output=True, text=True)
    log('$ python3 evals/promises/ledger.py format --ledger <surrogate ledger>\n' + proc.stderr + f'exit={proc.returncode}')
    assert proc.returncode != 0 and 'UnicodeEncodeError' in proc.stderr
    assert digest(p) == old_hash
    assert sorted(x.name for x in work.iterdir()) == folder
    log('Surrogate failure preserved original SHA-256 ' + old_hash + ' and created no temporary file.')

    for name, raw, key in [
        ('root', b'{"promises": {}, "promises": {}, "units": {}}', 'promises'),
        ('nested', b'{"promises": {}, "units": {"05-build:7": {}, "05-build:7": {}}}', '05-build:7'),
        ('array', b'{"promises": {}, "units": {}, "extra": [{"claim": 1, "claim": 2}]}', 'claim'),
    ]:
        p.write_bytes(raw)
        for args in [['format'], ['format', '--check'], ['audit'], ['owners', '--upstream', str(root)]]:
            proc = subprocess.run(cli + args + ['--ledger', str(p)], capture_output=True, text=True)
            expected = f"ValueError: duplicate key {key!r}"
            assert proc.returncode != 0 and expected in proc.stderr
            assert p.read_bytes() == raw
            log(f'{name} duplicate via {" ".join(args)}: exit={proc.returncode}; {proc.stderr.splitlines()[-1]}; original bytes unchanged')

    # A real process is stopped and killed after it opens its adjacent temporary file.
    p.write_bytes(b'{"z":"' + b'x' * (64 * 1024 * 1024) + b'","a":1}')
    original = digest(p)
    proc = subprocess.Popen(cli + ['format', '--ledger', str(p)], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    deadline = time.monotonic() + 30
    temp = None
    while time.monotonic() < deadline:
        temps = list(work.glob('.ledger.json.*'))
        if temps:
            os.kill(proc.pid, signal.SIGSTOP)
            temp = temps[0]
            break
        if proc.poll() is not None:
            raise AssertionError('Formatter finished before interruption could be driven')
        time.sleep(0.0005)
    assert temp is not None, 'No adjacent temporary file was observed'
    os.kill(proc.pid, signal.SIGKILL)
    stdout, stderr = proc.communicate(timeout=10)
    assert proc.returncode == -signal.SIGKILL
    assert digest(p) == original
    log(f'Killed real formatter after adjacent temp opened: exit={proc.returncode}, temp_bytes={temp.stat().st_size}; original ledger SHA-256 unchanged: {original}')
    temp.unlink()
    run(['format', '--ledger', str(p)])
    run(['format', '--check', '--ledger', str(p)])
    assert json.loads(p.read_bytes())['a'] == 1
    log('Formatting retry after interruption succeeded and preserved the ledger data.')
    p.unlink()

    owner = run(['owners', '--upstream', str(root), 'principle-leaf-summaries-match'])
    result = json.loads(owner.stdout)['principle-leaf-summaries-match']
    assert len(result['owners']) == 24
    assert 'skills/principle-explain-the-number/SKILL.md' in result['owners']
    book = json.loads((root / 'evals/promises/ledger.json').read_bytes())
    promise = book['promises']['principle-leaf-summaries-match']
    assert 'Each of the 24 principle links' in promise['claim']
    log('Published ledger contract has 24 owners, includes explain-the-number, and claims Each of the 24 principle links.')

    # Run the real drift-check executable against disposable persisted ledger states.
    copy = work / 'drift'
    (copy / 'evals/promises').mkdir(parents=True)
    (copy / 'skills/setup-pstack/scripts').mkdir(parents=True)
    shutil.copy(root / 'evals/promises/test_static.py', copy / 'evals/promises/test_static.py')
    shutil.copy(root / 'skills/setup-pstack/scripts/skill-listing.py', copy / 'skills/setup-pstack/scripts/skill-listing.py')
    env = dict(os.environ, PYTHONPATH=str(copy / 'evals/promises'), PYTHONDONTWRITEBYTECODE='1')
    for drift in ['owners', 'claim']:
        mutated = json.loads(json.dumps(book))
        entry = mutated['promises']['principle-leaf-summaries-match']
        if drift == 'owners':
            entry['owners'].remove('skills/principle-explain-the-number/SKILL.md')
        else:
            entry['claim'] = entry['claim'].replace('24 principle links', '23 principle links')
        (copy / 'evals/promises/ledger.json').write_text(json.dumps(mutated))
        cmd = [sys.executable, '-m', 'unittest', '-v', 'test_static.TestStaticPromises.test_principle_promise_owners_are_the_pinned_leaves']
        proc = subprocess.run(cmd, cwd=copy, env=env, capture_output=True, text=True)
        log(f'Disposable {drift} drift: $ python3 -m unittest -v test_static.TestStaticPromises.test_principle_promise_owners_are_the_pinned_leaves\n' + proc.stderr)
        assert proc.returncode == 1
        assert 'principle-leaf-summaries-match lists owners [' in proc.stderr
        assert 'Each of the 24 principle links' in proc.stderr
        assert 'edit promise principle-leaf-summaries-match' in proc.stderr
    log('All product checks completed successfully.')
finally:
    (evidence / 'ledger-cli-transcript.txt').write_text('\n\n'.join(lines) + '\n', encoding='utf-8')

Pipeline

Updates from git push no-mistakes

✅ **Intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

✅ **Review** - passed

✅ No issues found.

✅ **Test** - passed

✅ No issues found.

Live validation: ✅ go - 7 of 8 scenarios driven live against the product

Scenario	Result	Live	Evidence
Check the committed ledger and audit its promise bindings successfully	✅ pass	live	Ledger CLI transcript: committed format --check and audit both exited 0.
Format nested unsorted JSON into one canonical UTF-8 file and repeat without changes	✅ pass	live	Ledger CLI transcript: independently specified canonical bytes, preserved permissions, changed inode, and unchanged bytes on repeated formatting.
Reject noncanonical input with --check while preserving the file	✅ pass	live	Ledger CLI transcript: noncanonical CRLF input produced exit 1 and the fix command without changing input bytes.
Attempt to format a lone-surrogate ledger without corrupting it	✅ pass	live	Ledger CLI transcript: UnicodeEncodeError, unchanged original SHA-256, and no temporary file.
Interrupt a real formatter during its atomic write and successfully retry	✅ pass	live	Ledger CLI transcript: SIGKILL after adjacent temporary-file creation preserved original SHA-256; subsequent format and check succeeded.
Reject duplicate keys at every tested dictionary depth without modifying the ledger	✅ pass	live	Ledger CLI transcript: root, nested, and array-contained duplicates rejected by format, format --check, audit, and owners with ValueError naming each key.
Query the corrected principle promise and receive its 24 owners	✅ pass	live	Ledger CLI transcript: owners returned 24 entries including principle-explain-the-number; persisted claim names 24 principle links.
Detect principle-owner and count drift with actionable test diagnostics	⏸️ untested	no	The prior payload established only unit-test execution against committed data and disposable ledger copies, not a live product result. This assertion message exists only in a unit test and has no sepa…

TMPDIR="$PWD/.ledger-validation/tmp" PYTHONDONTWRITEBYTECODE=1 PYTHONPATH=evals/promises python3 -m unittest -v test_ledger.CanonicalFormTest test_static.TestStaticPromises.test_principle_promise_owners_are_the_pinned_leaves
PYTHONDONTWRITEBYTECODE=1 python3 .ledger-validation/drive.py: real CLI formatting, checks, audit, owners, duplicate-key rejection, surrogate preservation, SIGKILL interruption, and retry
Executed the targeted principle-owner test against disposable ledger copies with a missing owner and an incorrect principle count.
Saved the CLI transcript and reproducer, removed disposable files, and confirmed git status --short was empty.

✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

mdsmithaustin · 2026-10-12T19:50:55Z

Verdict: FAIL
Head: ce9408c
Patch-id: 2e145ccd2b05e29adadf739b2540a26b97d50036
Docs: pass

Round 1 swarm at this head: gates + live + merge simulation (sonnet@high), consumer-parity review and data-safety review (opus@xhigh each; work model sonnet@high).

Gates: PASS (lint 6/6 incl. the new canonical-form step; audit 0 problems; only the known host NativeGit errors). Live: format --check and format restore committed bytes from shuffled copies; semantic diff vs trunk is exactly the intended principle drift fix. Merge simulation: three edit patterns that conflict on trunk merge clean at head. Consumers: no reader depends on key order; CI runs the check on every PR.

Fix-forward (in scope, the new format command):

evals/promises/ledger.py:43: format is not atomic. path.write_text truncates first; a lone surrogate escape (passes load and audit) made format raise after emptying the 402 KB file, and an interrupted write left invalid JSON. Write to a temp file beside the ledger and os.replace it (encode before opening). Red test with the surrogate case.
ledger.py:37: duplicate keys are dropped silently, last wins. A duplicated unit (as a bad merge resolution produces) passes --check as merely non-canonical, and format then erases the first entry with exit 0. Load with an object_pairs_hook that raises on a duplicate key, naming it. Red test.
test_principle_promise_owners_are_the_pinned_leaves: when upstream adds a principle, the sync list names only PRINCIPLE_RULES, so this test fails without saying what to edit. Make its failure message name the promise id, its owners list, and the "Each of the N" count in ledger.json.

Notes (no change needed): read_text normalizes CRLF so --check is not byte-exact (no CRLF producer exists). Sorting mitigates rather than eliminates conflicts: two branches adding ids that sort into the same gap (or after the last key) still conflict; say so in the PR body.

mdsmithaustin · 2026-10-12T20:32:32Z

Verdict: PASS+NOTES
Head: 92523a4
Patch-id: abdaa000918da694ca3ae733074bb4130095b496
Docs: pass

Round 2 swarm at this head: gates + live on temp ledger copies + merge simulation (sonnet@high), fix review and integration review against main af9e849 (opus@xhigh each; work model sonnet@high).

Gates: lint 6/6, audit with --upstream-ref 0 problems, format --check 0; only known host noise. Live: a lone surrogate and an interrupted write leave the ledger byte-identical with no temp file; duplicate keys (top level and nested) make format, --check, audit and owners refuse with the key named; CRLF and shuffled copies format back to the committed bytes. The owners test names the promise, owners and the count fix. Merge onto main is clean and the merged ledger stays canonical; #198, #200, #201 and #202 each merge onto it without touching ledger.json.

Notes (follow-ups, not blocking):

A duplicate nested field names only the field ('claim'), not the object holding it.
No .gitattributes pins ledger.json to LF, so an autocrlf=true checkout fails --check (no such checkout or CI exists).
No test covers write_atomically's temp cleanup or mode copy (the probe proves both work).
test_static.py reads the ledger with plain json.loads; guarded checks run first in CI and the gate.

Adds ledger.py format and format --check, rewrites ledger.json with sorted keys, and runs the check in the lint tier. Two branches that add different entries now merge without a conflict, because entries land at their sorted place instead of the end of a section.

The promise said 23 principle links and omitted principle-explain-the-number as an owner. A new static test asserts the owners equal the keys of PRINCIPLE_RULES.

…uidance

…ers fix

… rejection

mdsmithaustin changed the title ~~chore(evals): enforce canonical promise ledger formatting~~ fix: enforce canonical promise ledger formatting Oct 12, 2026

mdsmithaustin added 6 commits October 12, 2026 14:33

evals: fix the principle-leaf-summaries-match count and owners

3711bac

The promise said 23 principle links and omitted principle-explain-the-number as an owner. A new static test asserts the owners equal the keys of PRINCIPLE_RULES.

no-mistakes(document): Clarify ledger formatting and merge conflict g…

aeb7666

…uidance

evals: make ledger format atomic, reject duplicate keys, name the own…

9dbd8c3

…ers fix

no-mistakes(test): Fix CRLF detection in ledger canonicalization

2aa2270

no-mistakes(document): Document ledger write safety and duplicate-key…

b037a0e

… rejection

mdsmithaustin force-pushed the fix/ledger-canonical-form branch from 92523a4 to b037a0e Compare October 12, 2026 20:43

mdsmithaustin changed the title ~~fix: enforce canonical promise ledger formatting~~ fix(evals): enforce canonical promise ledger format Oct 12, 2026

mdsmithaustin merged commit a8a5321 into main Oct 12, 2026
7 checks passed

mdsmithaustin deleted the fix/ledger-canonical-form branch October 12, 2026 20:51

Navigation Menu

Repository navigation

fix(evals): enforce canonical promise ledger format - #199

fix(evals): enforce canonical promise ledger format#199
mdsmithaustin merged 6 commits into
mainfrom
fix/ledger-canonical-form

mdsmithaustin commented Oct 12, 2026 •

edited

Loading

Uh oh!

mdsmithaustin commented Oct 12, 2026

Uh oh!

mdsmithaustin commented Oct 12, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

mdsmithaustin commented Oct 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Changed

Risk Assessment

Testing

Pipeline

Uh oh!

mdsmithaustin commented Oct 12, 2026

Uh oh!

mdsmithaustin commented Oct 12, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

mdsmithaustin commented Oct 12, 2026 •

edited

Loading