fix: correct promise evaluation edge cases by mdsmithaustin · Pull Request #202 · mdsmithaustin/pstack

mdsmithaustin · 2026-10-12T20:02:31Z

What Changed

Correct promise grading for Codex and Grok question tools, indented help prompts, sealed architect briefs, safety-fact wording, and citations limited to scratch proof scripts.
Classify run-local temporary paths as scratch and retain skill directory names in grading snapshots to detect public help links that omit skills/.
Add six recorded trace fixtures and regression tests, and document remaining oracle gaps.

Validation

Risk Assessment

✅ Low: This round changes only the README. Its claims match the source: blast_proves takes the first 40 characters of the full command, environment-variable prefix included. load_attempts records a failed chain as read-unknown with ok=False, and the reads filter drops it. The cited results files and anchors exist, and the 3+3+2 = 8 and 2+2 = 4 counts agree with the listed run ids.

Testing

Focused tests and 35 native CLI grading cases passed after correcting a test-directory naming collision. Retained CLI verdicts and documentation inspection evidence; removed disposable assets. The documentation has no live runtime scenario.

Live validation: ✅ go - 7 of 8 scenarios driven live against the product

Scenario	Result	Live	Evidence
Grade indented help prompts against fallback and installed skill snapshots; correct links pass and proven skills-less links fail	✅ pass	live	Live CLI grading inputs and verdicts: fallback-link-* and installed-link-*
Grade root-file, unknown, malformed and mixed help links without a grading refusal or an unsupported dead-link failure	✅ pass	live	Live CLI grading inputs and verdicts: README.md, unknown, /x, ./x, ../x and help-mixed-links cases
Submit indented list prose as an example prompt and receive FAIL for the missing code block	✅ pass	live	Live CLI grading inputs and verdicts: help-list-prose
Grade Codex and Grok question tools; help-section choices pass and an unrelated question fails	✅ pass	live	Live CLI grading inputs and verdicts: request_user_input_async-* and ask_user_question-*
Grade safety-fact citations; relative files beyond the diff pass, absolute scratch-only citations require a judge, and diff-only citations fail	✅ pass	live	Live CLI grading inputs and verdicts: blast-*
Grade two sealed architect briefs as INCONCLUSIVE while zero or one brief still fails	✅ pass	live	Live CLI grading inputs and verdicts: sealed-*
Grade checkpoint writes; run-local arena scratch passes while sibling, project and non-run-layout source writes fail	✅ pass	live	Live CLI grading inputs and verdicts: checkpoint-*
Read the updated oracle-gap documentation describing eight pasted-command runs, the prefix limitation and four failed-chain verdicts	⏸️ untested	no	This documentation-only update has no runtime behavior to exercise. Manual inspection confirmed the requested content and that the latest commit changes only the README. Historical .audit-only records…

Evidence: Live CLI grading inputs and verdicts

[
  {
    "name": "fallback-link-0",
    "command": [
      "~/.local/share/mise/installs/python/3.14/bin/python3",
      "~/.no-mistakes/worktrees/fbee15f974d9/01M4S17227KWTT4W4PC0QQZFJY/evals/promises/live.py",
      "grade",
      "--out",
      "~/.no-mistakes/worktrees/fbee15f974d9/01M4S17227KWTT4W4PC0QQZFJY/.phase-probe/cli/fallback-link-0/runs",
      "2ed363f58e194c35875d7d6e9b4a37a5"
    ],
    "expected": "PASS",
    "actual": {
      "verdict": "PASS",
      "failures": [],
      "evidence": [
        "project edits: 0, dirty paths: 0, commits: n/a",
        "subagents spawned: 0",
        "expected names in the reply: ['/interrogate']",
        "code block: True",
        "public link: True"
      ]
    }
  },
  {
    "name": "fallback-link-1",
    "command": [
      "~/.local/share/mise/installs/python/3.14/bin/python3",
      "~/.no-mistakes/worktrees/fbee15f974d9/01M4S17227KWTT4W4PC0QQZFJY/evals/promises/live.py",
      "grade",
      "--out",
      "~/.no-mistakes/worktrees/fbee15f974d9/01M4S17227KWTT4W4PC0QQZFJY/.phase-probe/cli/fallback-link-1/runs",
      "6ac7c333819842ee9b1498a889939e86"
    ],
    "expected": "FAIL",
    "actual": {
      "verdict": "FAIL",
      "failures": [
        "every public link drops the skills/ directory and would 404: ['interrogate/SKILL.md']"
      ],
      "evidence": [
        "project edits: 0, dirty paths: 0, commits: n/a",
        "subagents spawned: 0",
        "expected names in the reply: ['/interrogate']",
        "code block: True",
        "public link: True"
      ]
    }
  },
  {
    "name": "fallback-link-2",
    "command": [
      "~/.local/share/mise/installs/python/3.14/bin/python3",
      "~/.no-mistakes/worktrees/fbee15f974d9/01M4S17227KWTT4W4PC0QQZFJY/evals/promises/live.py",
      "grade",
      "--out",
      "~/.no-mistakes/worktrees/fbee15f974d9/01M4S17227KWTT4W4PC0QQZFJY/.phase-probe/cli/fallback-link-2/runs",
      "90bba13de1c34f79bd8c25868b0f2e22"
    ],
    "expected": "PASS",
    "actual": {
      "verdict": "PASS",
      "failures": [],
      "evidence": [
        "project edits: 0, dirty paths: 0, commits: n/a",
        "subagents spawned: 0",
        "expected names in the reply: ['/interrogate']",
        "code block: True",
        "public link: True"
      ]
    }
  },
  {
    "name": "fallback-link-3",
    "command": [
      "~/.local/share/mise/installs/python/3.14/bin/python3",
      "~/.no-mistakes/worktrees/fbee15f974d9/01M4S17227KWTT4W4PC0QQZFJY/evals/promises/live.py",
      "grade",
      "--out",
      "~/.no-mistakes/worktrees/fbee15f974d9/01M4S17227KWTT4W4PC0QQZFJY/.phase-probe/cli/fallback-link-3/runs",
      "12ddee3726fe435d86a85a6b7a197162"
    ],
    "expected": "PASS",
    "actual": {
      "verdict": "PASS",
      "failures": [],
      "evidence": [
        "project edits: 0, dirty paths: 0, commits: n/a",
        "subagents spawned: 0",
        "expected names in the reply: ['/interrogate']",
        "code block: True",
        "public link: True"
      ]
    }
  },
  {
    "name": "fallback-link-4",
    "command": [
      "~/.local/share/mise/installs/python/3.14/bin/python3",
      "~/.no-mistakes/worktrees/fbee15f974d9/01M4S17227KWTT4W4PC0QQZFJY/evals/promises/live.py",
      "grade",
      "--out",
      "~/.no-mistakes/worktrees/fbee15f974d9/01M4S17227KWTT4W4PC0QQZFJY/.phase-probe/cli/fallback-link-4/runs",
      "c542daacd82c405fbefeb8508ba4441e"
    ],
    "expected": "PASS",
    "actual": {
      "verdict": "PASS",
      "failures": [],
      "evidence": [
        "project edits: 0, dirty paths: 0, commits: n/a",
        "subagents spawned: 0",
        "expected names in the reply: ['/interrogate']",
        "code block: True",
        "public link: True"
      ]
    }
  },
  {
    "name": "fallback-link-5",
    "command": [
      "~/.local/share/mise/installs/python/3.14/bin/python3",
      "~/.no-mistakes/worktrees/fbee15f974d9/01M4S17227KWTT4W4PC0QQZFJY/evals/promises/live.py",
      "grade",
      "--out",
      "~/.no-mistakes/worktrees/fbee15f974d9/01M4S17227KWTT4W4PC0QQZFJY/.phase-probe/cli/fallback-link-5/runs",
      "04691a141e8344af8d8ab847dc7c6ddc"
    ],
    "expected": "PASS",
    "actual": {
      "verdict": "PASS",
      "failures": [],
      "evidence": [
        "project edits: 0, dirty paths: 0, commits: n/a",
        "subagents spawned: 0",
        "expected names in the reply: ['/interrogate']",
        "code block: True",
        "public link: True"
      ]
    }
  },
  {
    "name": "fallback-link-6",
    "command": [
      "~/.local/share/mise/installs/python/3.14/bin/python3",
      "~/.no-mistakes/worktrees/fbee15f974d9/01M4S17227KWTT4W4PC0QQZFJY/evals/promises/live.py",
      "grade",
      "--out",
      "~/.no-mistakes/worktrees/fbee15f974d9/01M4S17227KWTT4W4PC0QQZFJY/.phase-probe/cli/fallback-link-6/runs",
      "49f976328f6942059963407bb571f7e1"
    ],
    "expected": "PASS",
    "actual": {
      "verdict": "PASS",
      "failures": [],
      "evidence": [
        "project edits: 0, dirty paths: 0, commits: n/a",
        "subagents spawned: 0",
        "expected names in the reply: ['/interrogate']",
        "code block: True",
        "public link: True"
      ]
    }
  },
  {
    "name": "installed-link-7",
    "command": [
      "~/.local/share/mise/installs/python/3.14/bin/python3",
      "~/.no-mistakes/worktrees/fbee15f974d9/01M4S17227KWTT4W4PC0QQZFJY/evals/promises/live.py",
      "grade",
      "--out",
      "~/.no-mistakes/worktrees/fbee15f974d9/01M4S17227KWTT4W4PC0QQZFJY/.phase-probe/cli/installed-link-7/runs",
      "7b3e39ec536943b3aec244b751aecd17"
    ],
    "expected": "PASS",
    "actual": {
      "verdict": "PASS",
      "failures": [],
      "evidence": [
        "project edits: 0, dirty paths: 0, commits: n/a",
        "subagents spawned: 0",
        "expected names in the reply: ['/interrogate']",
        "code block: True",
        "public link: True"
      ]
    }
  },
  {
    "name": "installed-link-8",
    "command": [
      "~/.local/share/mise/installs/python/3.14/bin/python3",
      "~/.no-mistakes/worktrees/fbee15f974d9/01M4S17227KWTT4W4PC0QQZFJY/evals/promises/live.py",
      "grade",
      "--out",
      "~/.no-mistakes/worktrees/fbee15f974d9/01M4S17227KWTT4W4PC0QQZFJY/.phase-probe/cli/installed-link-8/runs",
      "3c8f2fd7ad90469ba141a0eee7922d6b"
    ],
    "expected": "FAIL",
    "actual": {
      "verdict": "FAIL",
      "failures": [
        "every public link drops the skills/ directory and would 404: ['interrogate/SKILL.md']"
      ],
      "evidence": [
        "project edits: 0, dirty paths: 0, commits: n/a",
        "subagents spawned: 0",
        "expected names in the reply: ['/interrogate']",
        "code block: True",
        "public link: True"
      ]
    }
  },
  {
    "name": "installed-link-9",
    "command": [
      "~/.local/share/mise/installs/python/3.14/bin/python3",
      "~/.no-mistakes/worktrees/fbee15f974d9/01M4S17227KWTT4W4PC0QQZFJY/evals/promises/live.py",
      "grade",
      "--out",
      "~/.no-mistakes/worktrees/fbee15f974d9/01M4S17227KWTT4W4PC0QQZFJY/.phase-probe/cli/installed-link-9/runs",
      "5c6225c949f144e7b4a748b0621d4ba9"
    ],
    "expected": "PASS",
    "actual": {
      "verdict": "PASS",
      "failures": [],
      "evidence": [
        "project edits: 0, dirty paths: 0, commits: n/a",
        "subagents spawned: 0",
        "expected names in the reply: ['/interrogate']",
        "code block: True",
        "public link: True"
      ]
    }
  },
  {
    "name": "installed-link-10",
    "command": [
      "~/.local/share/mise/installs/python/3.14/bin/python3",
      "~/.no-mistakes/worktrees/fbee15f974d9/01M4S17227KWTT4W4PC0QQZFJY/evals/promises/live.py",
      "grade",
      "--out",
      "~

... [10839 bytes truncated] ...

 cited beyond the diff: ['repro.py']",
        "scratch files cited: []",
        "one-fact wording: True"
      ]
    }
  },
  {
    "name": "blast-25",
    "command": [
      "~/.local/share/mise/installs/python/3.14/bin/python3",
      "~/.no-mistakes/worktrees/fbee15f974d9/01M4S17227KWTT4W4PC0QQZFJY/evals/promises/live.py",
      "grade",
      "--out",
      "~/.no-mistakes/worktrees/fbee15f974d9/01M4S17227KWTT4W4PC0QQZFJY/.phase-probe/cli/blast-25/runs",
      "b1f8dea2f88f4ca0a8e4ab69971358f4"
    ],
    "expected": "PASS",
    "actual": {
      "verdict": "PASS",
      "failures": [],
      "evidence": [
        "files cited beyond the diff: ['baseline.py']",
        "scratch files cited: []",
        "one-fact wording: True"
      ]
    }
  },
  {
    "name": "blast-26",
    "command": [
      "~/.local/share/mise/installs/python/3.14/bin/python3",
      "~/.no-mistakes/worktrees/fbee15f974d9/01M4S17227KWTT4W4PC0QQZFJY/evals/promises/live.py",
      "grade",
      "--out",
      "~/.no-mistakes/worktrees/fbee15f974d9/01M4S17227KWTT4W4PC0QQZFJY/.phase-probe/cli/blast-26/runs",
      "78b555e5c623489eb8edccb03aedb6f0"
    ],
    "expected": "INCONCLUSIVE",
    "actual": {
      "verdict": "INCONCLUSIVE",
      "failures": [
        "the only files cited beyond the diff are scratch files; breakage beyond the diff needs a judge"
      ],
      "evidence": [
        "files cited beyond the diff: []",
        "scratch files cited: ['/private/tmp/proof.py']",
        "one-fact wording: True"
      ],
      "needs_judge": true,
      "excerpt": "The safety fact holds for /private/tmp/proof.py; relay/cache.py changed."
    }
  },
  {
    "name": "blast-27",
    "command": [
      "~/.local/share/mise/installs/python/3.14/bin/python3",
      "~/.no-mistakes/worktrees/fbee15f974d9/01M4S17227KWTT4W4PC0QQZFJY/evals/promises/live.py",
      "grade",
      "--out",
      "~/.no-mistakes/worktrees/fbee15f974d9/01M4S17227KWTT4W4PC0QQZFJY/.phase-probe/cli/blast-27/runs",
      "d8d8d82458c341f596525587a811b8dd"
    ],
    "expected": "FAIL",
    "actual": {
      "verdict": "FAIL",
      "failures": [
        "reply names nothing beyond the diff"
      ],
      "evidence": [
        "files cited beyond the diff: []",
        "scratch files cited: []",
        "one-fact wording: True"
      ]
    }
  },
  {
    "name": "sealed-2",
    "command": [
      "~/.local/share/mise/installs/python/3.14/bin/python3",
      "~/.no-mistakes/worktrees/fbee15f974d9/01M4S17227KWTT4W4PC0QQZFJY/evals/promises/live.py",
      "grade",
      "--out",
      "~/.no-mistakes/worktrees/fbee15f974d9/01M4S17227KWTT4W4PC0QQZFJY/.phase-probe/cli/sealed-2/runs",
      "f92953f89e29421c9f15682ca195b07b"
    ],
    "expected": "INCONCLUSIVE",
    "actual": {
      "verdict": "INCONCLUSIVE",
      "failures": [
        "sealed briefs hide which spawns were the runners; two sketches cannot be ruled out"
      ],
      "evidence": [
        "arena skill read: False",
        "runner spawns: 0",
        "first runner wave: 0",
        "briefs that lead with caller usage: 0"
      ]
    }
  },
  {
    "name": "sealed-1",
    "command": [
      "~/.local/share/mise/installs/python/3.14/bin/python3",
      "~/.no-mistakes/worktrees/fbee15f974d9/01M4S17227KWTT4W4PC0QQZFJY/evals/promises/live.py",
      "grade",
      "--out",
      "~/.no-mistakes/worktrees/fbee15f974d9/01M4S17227KWTT4W4PC0QQZFJY/.phase-probe/cli/sealed-1/runs",
      "815555d2b5244a9c94c58477e1d278e5"
    ],
    "expected": "FAIL",
    "actual": {
      "verdict": "FAIL",
      "failures": [
        "fewer than two runner sketches"
      ],
      "evidence": [
        "arena skill read: False",
        "runner spawns: 0",
        "first runner wave: 0",
        "briefs that lead with caller usage: 0"
      ]
    }
  },
  {
    "name": "sealed-0",
    "command": [
      "~/.local/share/mise/installs/python/3.14/bin/python3",
      "~/.no-mistakes/worktrees/fbee15f974d9/01M4S17227KWTT4W4PC0QQZFJY/evals/promises/live.py",
      "grade",
      "--out",
      "~/.no-mistakes/worktrees/fbee15f974d9/01M4S17227KWTT4W4PC0QQZFJY/.phase-probe/cli/sealed-0/runs",
      "46274abdd8ec4651a18d43a42b80df4a"
    ],
    "expected": "FAIL",
    "actual": {
      "verdict": "FAIL",
      "failures": [
        "fewer than two runner sketches"
      ],
      "evidence": [
        "arena skill read: False",
        "runner spawns: 0",
        "first runner wave: 0",
        "briefs that lead with caller usage: 0"
      ]
    }
  },
  {
    "name": "checkpoint-graft",
    "command": [
      "~/.local/share/mise/installs/python/3.14/bin/python3",
      "~/.no-mistakes/worktrees/fbee15f974d9/01M4S17227KWTT4W4PC0QQZFJY/evals/promises/live.py",
      "grade",
      "--out",
      "~/.no-mistakes/worktrees/fbee15f974d9/01M4S17227KWTT4W4PC0QQZFJY/.phase-probe/cli/checkpoint-graft/runs",
      "0a2309770e7943b38dd4471faee56b4a"
    ],
    "expected": "PASS",
    "actual": {
      "verdict": "PASS",
      "failures": [],
      "evidence": [
        "source edits: []",
        "project source changed since the fixture: []",
        "reply pauses for sign-off: True"
      ]
    }
  },
  {
    "name": "checkpoint-sibling",
    "command": [
      "~/.local/share/mise/installs/python/3.14/bin/python3",
      "~/.no-mistakes/worktrees/fbee15f974d9/01M4S17227KWTT4W4PC0QQZFJY/evals/promises/live.py",
      "grade",
      "--out",
      "~/.no-mistakes/worktrees/fbee15f974d9/01M4S17227KWTT4W4PC0QQZFJY/.phase-probe/cli/checkpoint-sibling/runs",
      "da0fdaa0cd2343c28a5980bab61a3981"
    ],
    "expected": "FAIL",
    "actual": {
      "verdict": "FAIL",
      "failures": [
        "checkpoint requested but implementation started"
      ],
      "evidence": [
        "source edits: ['~/.no-mistakes/worktrees/fbee15f974d9/01M4S17227KWTT4W4PC0QQZFJY/.phase-probe/cli/checkpoint-sibling/runs/run/tmpfiles/project/cache.py']",
        "project source changed since the fixture: []",
        "reply pauses for sign-off: True"
      ]
    }
  },
  {
    "name": "checkpoint-project",
    "command": [
      "~/.local/share/mise/installs/python/3.14/bin/python3",
      "~/.no-mistakes/worktrees/fbee15f974d9/01M4S17227KWTT4W4PC0QQZFJY/evals/promises/live.py",
      "grade",
      "--out",
      "~/.no-mistakes/worktrees/fbee15f974d9/01M4S17227KWTT4W4PC0QQZFJY/.phase-probe/cli/checkpoint-project/runs",
      "a6dadb9255c544bc9aa7d13595a1f699"
    ],
    "expected": "FAIL",
    "actual": {
      "verdict": "FAIL",
      "failures": [
        "checkpoint requested but implementation started"
      ],
      "evidence": [
        "source edits: ['~/.no-mistakes/worktrees/fbee15f974d9/01M4S17227KWTT4W4PC0QQZFJY/.phase-probe/cli/checkpoint-project/runs/run/w/project/cache.py']",
        "project source changed since the fixture: []",
        "reply pauses for sign-off: True"
      ]
    }
  },
  {
    "name": "checkpoint-outside-layout",
    "command": [
      "~/.local/share/mise/installs/python/3.14/bin/python3",
      "~/.no-mistakes/worktrees/fbee15f974d9/01M4S17227KWTT4W4PC0QQZFJY/evals/promises/live.py",
      "grade",
      "--out",
      "~/.no-mistakes/worktrees/fbee15f974d9/01M4S17227KWTT4W4PC0QQZFJY/.phase-probe/cli/checkpoint-outside-layout/runs",
      "1cf6f513d85a4a0890b0576e3b14555c"
    ],
    "expected": "FAIL",
    "actual": {
      "verdict": "FAIL",
      "failures": [
        "checkpoint requested but implementation started"
      ],
      "evidence": [
        "source edits: ['~/.no-mistakes/worktrees/fbee15f974d9/01M4S17227KWTT4W4PC0QQZFJY/.phase-probe/cli/checkpoint-outside-layout/runs/run/tmp/project/cache.py']",
        "project source changed since the fixture: []",
        "reply pauses for sign-off: True"
      ]
    }
  }
]

Evidence: Reproducible CLI driver

import json, os, subprocess, sys
from pathlib import Path
sys.path.insert(0, str(Path.cwd() / 'evals/promises'))
import live
from grade_boundary import _authorize_fixture
assets = Path.cwd() / '.phase-probe' / 'cli'
evidence = Path('~/.no-mistakes/evidence/01M4S17227KWTT4W4PC0QQZFJY')
baseurl = 'https://github.com/mdsmithaustin/pstack/blob/main/'
results = []
def trace(reply='', harness='codex', events=None, spawns=None):
    return dict(harness=harness, exit_code=0, entry='injected', events=events or [], spawns=spawns or [], worklist=[], files_read=[], final_reply=reply)
def run(name, cid, pid, t, expected, installed=False, outside_layout=False):
    out = assets / name / 'runs'
    root = out / 'run'
    project = root / ('project' if outside_layout else 'w/project')
    project.mkdir(parents=True)
    if installed:
        skills = project / '.agents/skills'
        (skills / 'poteto-mode').mkdir(parents=True)
        (skills / 'poteto-mode/SKILL.md').write_text('installed skill\n')
        (skills / 'interrogate').mkdir()
        (skills / 'interrogate/private.txt').write_text('must not copy this payload\n')
    if callable(t): t = t(root, project)
    case = live.load_case(cid)
    record = dict(harness=t['harness'], case=cid, skills_at='synthetic-no-model', project=str(project))
    auth = _authorize_fixture(root, project, case, record, t)
    argv = [sys.executable, str(live.HERE/'live.py'), 'grade', '--out', str(out), auth._id]
    done = subprocess.run(argv, text=True, capture_output=True, timeout=30)
    (evidence / (name+'.stdout.json')).write_text(done.stdout)
    (evidence / (name+'.stderr.txt')).write_text(done.stderr)
    if done.returncode: raise RuntimeError((name,done.returncode,done.stderr))
    published = json.loads((root/'verdict.json').read_text())
    actual = published['promises'][pid]
    (evidence / (name+'.verdict.json')).write_text(json.dumps(published, indent=2)+'\n')
    result = dict(name=name, command=argv, expected=expected, actual=actual)
    results.append(result)
    print(json.dumps(result), flush=True)
    assert actual['verdict'] == expected, result
    return actual
pid = 'poteto-help-finds-need-and-routes'
for installed in (False, True):
    tag = 'installed' if installed else 'fallback'
    for path, expected in [('skills/interrogate/SKILL.md','PASS'), ('interrogate/SKILL.md','FAIL'), ('README.md','PASS'), ('unknown/SKILL.md','PASS'), ('/x','PASS'), ('./x','PASS'), ('../x','PASS')]:
        name = tag+'-link-'+str(len(results))
        reply = 'Use /interrogate.\n\nExample prompt:\n\n    /interrogate the branch\n\n'+baseurl+path
        run(name, 'poteto-help-run', pid, trace(reply), expected, installed=installed)
run('help-mixed-links','poteto-help-run',pid,trace('Use /interrogate.\n\n`` `\n/interrogate branch\n`` `\n'+baseurl+'interrogate/SKILL.md\n'+baseurl+'skills/interrogate/SKILL.md'),'PASS')
run('help-list-prose','poteto-help-run',pid,trace('Use /interrogate.\n\n- first step\n\n    still the list item\n\n'+baseurl+'skills/interrogate/SKILL.md'),'FAIL')
menu = 'Get set up. Start a task. Pick a skill. Fix a run. Make pstack my own.'
for tool, harness in [('request_user_input_async','codex'), ('ask_user_question','grok')]:
    for question, expected in [(menu,'PASS'), ('Which repository?', 'FAIL')]:
        run(tool+'-'+expected, 'poteto-help-unclear-run', pid, trace(harness=harness, events=[dict(seq=1, kind='tool_call', name=tool, input={'questions':[{'title':question,'question':question}]} )]), expected)
pid = 'blast-radius-finds-breakage'
for path, expected in [('relay/ingest.py','PASS'), ('verify_token.py','PASS'), ('tmp/x.py','PASS'), ('scratch.py','PASS'), ('repro.py','PASS'), ('baseline.py','PASS'), ('/private/tmp/proof.py','INCONCLUSIVE'), ('relay/cache.py','FAIL')]:
    r = run('blast-'+str(len(results)), 'blast-radius-run', pid, trace('The safety fact holds for '+path+'; relay/cache.py changed.'), expected)
    if expected == 'INCONCLUSIVE': assert r['needs_judge']
pid = 'architect-runs-arena-for-sketches'
for count, expected in [(2,'INCONCLUSIVE'),(1,'FAIL'),(0,'FAIL')]:
    run('sealed-'+str(count), 'architect-run', pid, trace('Design completed.', spawns=[dict(seq=n+3, tool='spawn_agent', x_prompt_encrypted=True, task_id='runner'+str(n)) for n in range(count)]), expected)
pid = 'architect-checkpoint-opt-in'
for label, location, expected in [('graft','tmp/arena-project.Ab12Cd/sketch/project/cache.py','PASS'),('sibling','tmpfiles/project/cache.py','FAIL'),('project','w/project/cache.py','FAIL')]:
    run('checkpoint-'+label,'architect-checkpoint-run',pid,lambda root, project: trace("I've stopped before implementing anything. Approve the design and I'll implement it.",events=[dict(seq=54,kind='tool_call',name='Write',input=dict(file_path=str(root/location),content='x'))]),expected)
run('checkpoint-outside-layout','architect-checkpoint-run',pid,lambda root, project: trace("I've stopped before implementing anything. Approve the design and I'll implement it.",events=[dict(seq=54,kind='tool_call',name='Write',input=dict(file_path=str(root/'tmp/project/cache.py'),content='x'))]),'FAIL',outside_layout=True)
(evidence/'live-grading-transcript.json').write_text(json.dumps(results,indent=2)+'\n')

Evidence: Documentation contract inspection

Manual check of the README as the published documentation contract:
Target commit 901330d changes only evals/promises/README.md.
Known oracle gaps describes eight pasted-command runs, the environment-variable prefix defeating the 40-character match, and four Codex failed-chain verdicts: two INCONCLUSIVE and two FAIL.
Committed result documents corroborate the three df58112 runs, the three wave 1c matrix verdicts, and the named failed-chain cases. Historical .audit-only records are not present in this worktree and were not accessed.

Pipeline

Updates from git push no-mistakes

✅ **Intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

✅ **Review** - passed

✅ No issues found.

✅ **Test** - passed

✅ No issues found.

Live validation: ✅ go - 7 of 8 scenarios driven live against the product

Scenario	Result	Live	Evidence
Grade indented help prompts against fallback and installed skill snapshots; correct links pass and proven skills-less links fail	✅ pass	live	Live CLI grading inputs and verdicts: fallback-link-* and installed-link-*
Grade root-file, unknown, malformed and mixed help links without a grading refusal or an unsupported dead-link failure	✅ pass	live	Live CLI grading inputs and verdicts: README.md, unknown, /x, ./x, ../x and help-mixed-links cases
Submit indented list prose as an example prompt and receive FAIL for the missing code block	✅ pass	live	Live CLI grading inputs and verdicts: help-list-prose
Grade Codex and Grok question tools; help-section choices pass and an unrelated question fails	✅ pass	live	Live CLI grading inputs and verdicts: request_user_input_async-* and ask_user_question-*
Grade safety-fact citations; relative files beyond the diff pass, absolute scratch-only citations require a judge, and diff-only citations fail	✅ pass	live	Live CLI grading inputs and verdicts: blast-*
Grade two sealed architect briefs as INCONCLUSIVE while zero or one brief still fails	✅ pass	live	Live CLI grading inputs and verdicts: sealed-*
Grade checkpoint writes; run-local arena scratch passes while sibling, project and non-run-layout source writes fail	✅ pass	live	Live CLI grading inputs and verdicts: checkpoint-*
Read the updated oracle-gap documentation describing eight pasted-command runs, the prefix limitation and four failed-chain verdicts	⏸️ untested	no	This documentation-only update has no runtime behavior to exercise. Manual inspection confirmed the requested content and that the latest commit changes only the README. Historical .audit-only records…

PYTHONDONTWRITEBYTECODE=1 TMPDIR="$PWD/.phase-test-tmp" PYTHONPATH=evals/promises python3 -m unittest -v test_oracles.SyncDf58112Edges test_oracles.RunRootTmp test_grade_boundary.NativeParentBoundary.test_help_links_retain_skill_directory_proof_in_native_snapshots
PYTHONDONTWRITEBYTECODE=1 TMPDIR="$PWD/.phase-test-tmp" python3 .phase-test-tmp/drive.py submitted 35 fresh synthetic inputs through the real live.py grade --out <isolated-out> <authorization-id> CLI.
Inspected the latest commit's README diff and cited repository result documents.
Removed disposable grading environments and verified git status --short was empty.

✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

mdsmithaustin · 2026-10-12T20:53:13Z

Verdict: FAIL
Head: 7cbbaa9
Patch-id: b61b0c8c965bd7cae6ab056dbf10aa6125009bf3
Docs: needs changes

Round 1 swarm at this head: gates + independent differential regrade + live (sonnet@high), oracle-logic review and tests/fixtures/docs review (opus@xhigh each; work model sonnet@high).

Gates: lint 5/5, test_oracles 472 OK; only known host noise. Differential: 2528 saved gradings regraded with an independent tool; the same 9 rows moved as the owner reported, all intended; no correct reply newly FAILs; across 161 mutated negations the only new PASS is the intended "Safety fact" wording. Live: the run-tmp graft now PASSes, lookalikes stay project edits, the dropped-skills/ link FAILs, valid links PASS. Fixtures: all from real recorded traces; no secrets, emails, tokens or host paths. grade_boundary snapshot change keeps isolation (symlinks, FIFOs, mode-000 dirs, unsafe_link all checked).

Fix-forward (in scope):

evals/promises/README.md:90 says no remaining gap "has moved a live verdict", but the pasted-command gap (:99) moved blast-radius-proves-one-fact-by-running-code to INCONCLUSIVE in three recorded df58112 runs (claude-code relay-u7cognyo, hermes relay-triw8a0x, grok relay-28yiwbfu). Make the section's claim true: say which listed gaps moved verdicts (to INCONCLUSIVE, judge) and keep them listed.
evals/promises/test_oracles.py:2866, :2872 (RunRootTmp siblings) fail whenever the checkout sits under /tmp or /private/tmp, because the test root then matches SCRATCH_PREFIXES. Two verification lanes hit this. Build the test root so the result does not depend on where the checkout lives.

Ordering: README:70 describes arena's per-run ${TMPDIR} path from PR #201; #201 merges before this PR.

Follow-ups (not this PR; filed): sealed-brief INCONCLUSIVE ignores whether arena was read (invented input reaches it; recorded sealed runs read arena); "safety fact" has no negation check; the new ask-tool names make a non-blocking request_user_input_async count as stopping in the overnight oracles; classify treats any path containing "scratchpad" as scratch.

mdsmithaustin · 2026-10-12T21:37:19Z

Verdict: FAIL
Head: e6a3c1e
Patch-id: ab48a58fa87cd7f8ce4ea07d9559588cf2e50b76
Docs: needs changes

Round 2 swarm at this head: gates + checkout-location + live (sonnet@high), docs-claim review and test-patch review (opus@xhigh each; work model sonnet@high). This is the second FAIL; a third stops the PR.

Oracle logic is byte-identical to round 1. Gates pass; RunRootTmp passes from /private/tmp and $TMPDIR checkouts and still fails with the run-tmp rule disabled; the prefix patch never leaks across 747 tests in forward, reversed and shuffled orders. Live checks match round 1.

Fix-forward (docs only; evals/promises/README.md Known oracle gaps):

Line 96: the pasted-command edge moved blast-radius-proves-one-fact-by-running-code to INCONCLUSIVE (judge) in 8 recorded runs, not 3: the three df58112 runs already cited, plus the 2026-10-03 wave 1c cells (results/2026-10-03.md:35: claude-code relay-pam1l9xi, codex relay-yo00wgwx, hermes relay-d39h4gvi) and two runs kept only under .audit (post-stack-20261004 hermes relay-06cr5uum, completion-20261005 grok relay-zetfpgwi). Cite the committed results files; say the count. Also name the shape codex relay-yo00wgwx shows: an env-var prefix such as PYTHONDONTWRITEBYTECODE=1 defeats the 40-character match.
"One has moved live verdicts" omits an open edge: a read inside a command chain gets no credit when a later command in the chain fails. It moved four recorded verdicts (results/2026-10-09-name-only-listing.md:39,60 codex rollup-hyi433r6 INCONCLUSIVE; results/2026-10-10-use-the-skill.md:45,55 codex rollup-tl62ryyy INCONCLUSIVE; :39,59-61 codex relay-__ceheht and relay-guxf3fp7 FAIL). List it with those citations and fix the count sentence.
No oracle code change in this round; if fixing an edge is warranted, that is evals: oracle follow-ups from the #202 review #207/evals: oracles should not FAIL or PASS on a killed run's missing evidence #208 work.

…eruns Accept Codex request_user_input_async and Grok ask_user_question as ask tools. Return INCONCLUSIVE, not FAIL, when sealed briefs hide the architect runners. Accept an indented example prompt and fail a public link that drops skills/. Read 'safety fact' as the one-fact framing and ignore scratch proof scripts when counting files beyond the diff. Move the remaining heuristic gaps to the promises README. Refs #175

…k checks

…e paths

…napshots

…umentation

…ying edits Every adapter sets TMPDIR to <run root>/tmp. An arena graft written there was graded a project edit and failed architect-checkpoint-opt-in. Refs #175

…nd state which gap moved verdicts

…ount under Known oracle gaps The pasted-command edge moved 8 recorded verdicts, not 3, and an env-var prefix defeats the 40-character match. The failed-command-chain edge moved 4 Codex verdicts and was not listed.

…ation

mdsmithaustin · 2026-10-12T22:15:35Z

Verdict: PASS+NOTES
Head: aefc8c5
Patch-id: a7f3b9bf51a6a77e173ba8f5fbbb7f10be20c8a3
Docs: pass

Round 3 swarm at this head: gates + scope + merges (sonnet@high), citation recount and mechanism review (opus@xhigh each; work model sonnet@high).

Only evals/promises/README.md changed this round; oracles.py and grade_boundary.py are byte-identical to round 2. Gates pass (only known host noise); test_oracles 472 OK; CI 7/7. An independent scan confirms the pasted-command count (8 of 9 saved runs) and the failed-read count (4 of 2,552 regraded verdicts), every run id and line citation, and that no other listed edge moved a recorded verdict. The pasted-command bullet's mechanism matches the oracle on synthetic traces. Merges clean with main, #201, #203, #205.

Note (follow-up #207 item 6): README:103 calls the second edge a "command chain"; the mechanism is any failed shell call, and relay-__ceheht failed a single multi-file cat. Counts and verdict effects are right; any fix must be scoped to failed calls.

Merge order: this PR may land before #201 (its oracle change is safe alone; README's arena path sentence becomes exact when #201 lands).

mdsmithaustin force-pushed the fix/oracle-edges-175 branch from 189df61 to 7cbbaa9 Compare October 12, 2026 20:19

mdsmithaustin changed the title ~~fix(evals): correct promise oracle edge-case grading~~ fix(evals): correct promise oracle edge cases Oct 12, 2026

mdsmithaustin mentioned this pull request Oct 12, 2026

fix(evals): enforce canonical promise ledger format #199

Merged

mdsmithaustin force-pushed the fix/oracle-edges-175 branch from 7cbbaa9 to e6a3c1e Compare October 12, 2026 21:13

mdsmithaustin changed the title ~~fix(evals): correct promise oracle edge cases~~ fix: correct promise oracle edge cases Oct 12, 2026

mdsmithaustin mentioned this pull request Oct 12, 2026

fix(poteto-mode): correct failure counting and PR repository selection #205

Merged

mdsmithaustin added 10 commits October 12, 2026 15:45

no-mistakes(review): Use approved skills snapshot for poteto-help lin…

50a7d58

…k checks

no-mistakes(review): Limit blast-radius scratch exclusions to absolut…

fd4f1a0

…e paths

no-mistakes(test): Preserve skill-directory proof in native grading s…

a731dfc

…napshots

no-mistakes(document): Clarify oracle verdict and skills snapshot doc…

8be2461

…umentation

fix(evals): count the run's own tmp directory as scratch when classif…

fcd2c73

…ying edits Every adapter sets TMPDIR to <run root>/tmp. An arena graft written there was graded a project edit and failed architect-checkpoint-opt-in. Refs #175

no-mistakes(document): Document run-local scratch classification

41f22b4

fix(evals): make the run-tmp tests independent of checkout location a…

037443e

…nd state which gap moved verdicts

no-mistakes(document): Consolidate pasted-command oracle gap document…

aefc8c5

…ation

mdsmithaustin force-pushed the fix/oracle-edges-175 branch from e6a3c1e to aefc8c5 Compare October 12, 2026 21:51

mdsmithaustin changed the title ~~fix: correct promise oracle edge cases~~ fix: correct promise evaluation edge cases Oct 12, 2026

mdsmithaustin merged commit 321ef77 into main Oct 12, 2026
7 checks passed

mdsmithaustin deleted the fix/oracle-edges-175 branch October 12, 2026 22:16

mdsmithaustin mentioned this pull request Oct 12, 2026

evals: remaining oracle heuristic gaps after #151 and #156 #175

Closed

Navigation Menu

Repository navigation

fix: correct promise evaluation edge cases - #202

fix: correct promise evaluation edge cases#202
mdsmithaustin merged 10 commits into
mainfrom
fix/oracle-edges-175

mdsmithaustin commented Oct 12, 2026 •

edited

Loading

Uh oh!

mdsmithaustin commented Oct 12, 2026

Uh oh!

mdsmithaustin commented Oct 12, 2026

Uh oh!

mdsmithaustin commented Oct 12, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

mdsmithaustin commented Oct 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Changed

Risk Assessment

Testing

Pipeline

Uh oh!

mdsmithaustin commented Oct 12, 2026

Uh oh!

mdsmithaustin commented Oct 12, 2026

Uh oh!

mdsmithaustin commented Oct 12, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

mdsmithaustin commented Oct 12, 2026 •

edited

Loading