diff --git a/evals/promises/README.md b/evals/promises/README.md index b5a833b3..b95ab9be 100644 --- a/evals/promises/README.md +++ b/evals/promises/README.md @@ -77,6 +77,8 @@ Before the first case, `live.py run` checks the selected Claude Code, Codex, or For Claude Code, Codex, and Grok, a turn counts as an auth death when it exited nonzero without timing out and the tail of its stderr or the last nonempty line of its stream carries that CLI's login-failure text. A `LoginRequired` failure during preparation stops the run before any turn. `live.py` writes a `verdict.json` that marks every promise that applies to the harness `ERROR` and records the CLI and its login command under `error`. It skips harvest and grading, so that run has no `trace.json`. When a turn dies on auth, `live.py` still harvests and grades the run. A `PASS` or `FAIL` the grader decides stands, and every other applicable promise becomes `ERROR`, keeping any `needs_judge` flag and `excerpt`. After writing either verdict, the wave stops with exit 3. A harvest refusal after an auth death propagates unchanged and stops the batch before the run gets `trace.json` or `verdict.json`. Running `live.py grade` on a sealed auth-death run later re-grades it from its trace, keeps the `error` record, and marks `ERROR` again on every promise the new grade leaves undecided. A run whose login died before any turn has no sealed trace, so `live.py grade` refuses it with exit 2 and leaves its verdict unchanged. In each of these verdicts a promise that does not apply to the harness is marked `not_applicable`, so it stays out of the report. `live.py report` shows `ERROR` in the table and leaves those verdicts out of each harness's `pass rate: of ` line. This rate counts applicable promise verdicts, not runs. `live.py` and `live.py report` decide applicability by asking each oracle about an empty trace from the harness, because a grade that returns early carries no `not_applicable` mark. A grade returns early when the run never started, its transcript could not be read, or host skills touched it. `INCONCLUSIVE` stays in the denominator. When every applicable verdict is `ERROR`, the rate reads `none graded`. A turn that exited 0 still grades as before, even if it contains login-failure text. +For a fixture at `/w/`, `View.classify` treats absolute paths that resolve under `/tmp` as scratch. Every adapter sets `TMPDIR` to that directory. Arena grafts under `/tmp/arena-.XXXXXX` therefore do not count as project edits. This run-specific rule does not apply to fixtures outside the `w/` layout or to sibling directories such as `/tmpfiles`. + `--model` and `--effort` set the model and reasoning effort for every turn of every case in one `run` invocation, on Claude Code and Codex. Each turn's recorded argv carries both values, and the sealed run metadata keeps its fields. Other adapters refuse the flags. A case's `skill_overlays` lists `{file, old, new}` replacements, with `file` relative to the installed skills root. `live.py` applies them after it installs the skill tree and before trusted setup captures the run, and refuses the run when an `old` does not occur exactly once. Arm cases use overlays to run one skill sentence several ways while the branch keeps its own text. Every trace also gets `x_implicit_off`, the installed pstack skills whose `agents/openai.yaml` sets `policy.allow_implicit_invocation: false`, computed by `managed_skills` from the evaluator's own listing script even when the installed tree predates that script. @@ -93,6 +95,34 @@ A turn whose text is `/compact` compacts the session. Claude Code sends it throu Live runs spend model budget. Codex and Hermes draw on the same ChatGPT plan when Hermes uses the `openai-codex` provider. A Feature case takes about 5 minutes on Claude Code and over 20 minutes on Codex. +## Known oracle gaps + +These heuristic edges are open. Two have moved recorded verdicts. + +- **Pasted command.** The oracle requires the first 40 characters of a run command to appear in the reply after whitespace normalization. A reply that pastes only output goes to the judge. The edge sent `blast-radius-proves-one-fact-by-running-code` to `INCONCLUSIVE` and the judge in 8 recorded runs. Three are the [df58112 runs](results/2026-10-08-sync-df58112.md) on Claude Code (`relay-u7cognyo`), Hermes (`relay-triw8a0x`), and Grok (`relay-28yiwbfu`). Three are wave 1c runs on Claude Code (`relay-pam1l9xi`), Codex (`relay-yo00wgwx`), and Hermes (`relay-d39h4gvi`). Their verdicts are in the `blast-radius-proves-one-fact-by-running-code` row of the [2026-10-03 matrix](results/2026-10-03.md#matrix), which does not name the run ids. The ids come from the saved verdicts under `.audit`. Two more runs appear in no committed results file and exist only under `.audit`. They are Hermes `relay-06cr5uum` (`post-stack-20261004`) and Grok `relay-zetfpgwi` (`completion-20261005`). An environment-variable prefix defeats the 40-character match. Codex `relay-yo00wgwx` ran `PYTHONDONTWRITEBYTECODE=1 python3 /private/tmp/relay-blast-radius-proof.py --require-offset-isolation` and pasted the same command without `PYTHONDONTWRITEBYTECODE=1 `. +- **Failed command chain.** A read inside a command chain gets no credit when a later command in the chain fails. `load_attempts` grades the whole shell call, so a failed chain records the read as `read-unknown`, which the Codex load rule does not accept. The edge moved four recorded verdicts on Codex. Two are `INCONCLUSIVE`. They are `rollup-hyi433r6` in the [2026-10-09 results](results/2026-10-09-name-only-listing.md#classification) and `rollup-tl62ryyy` in the [2026-10-10 results](results/2026-10-10-use-the-skill.md#classification). The other two are `FAIL`. They are `relay-__ceheht`, which the 2026-10-10 results name, and `relay-guxf3fp7`, which they do not. The id `relay-guxf3fp7` comes from the saved verdicts under `.audit`. + +No other listed edge is recorded as moving a verdict. Fix one only after a recorded run's verdict moves because of it. Work it red-first, and run a differential over every saved corpus against `main`. Both counts must be 0: correct replies that fail at head but pass on `main`, and negated replies that pass at head but fail on `main`. A new FAIL needs proof of the defect. See [Run live cases](#run-live-cases) for unresolved evidence and judge referrals. + +**Promise oracles** + +- Nothing in `report.py` or `live.py` reads `needs_judge`. A person reads those runs. In the PR #156 corpus, 122 of 308 correct review-verdict replies went to the judge. +- `reply_label` reads these label shapes differently from the pre-#156 oracle: underscore-wrapped and backtick-wrapped labels, labels ended by a newline, `Role:` with its value on the next line, a bold prefix before the role, topic-first headings such as `## Parser - Explorer 2`, separator-free `## Explorer 2 Findings`, and findings-first headings. +- The arena base check fails `Based on the judge's scores I kept candidate 2` and markup-wrapped bases (`_candidate 2_`, `[candidate 2](...)`), which the pre-#156 oracle passed through a `base` substring match. +- The sources check passes commit-only shapes: comma lists of hashes, consecutive commit-only lines, and `Sources searched:` labels. +- Timing: the `SOURCES_SECTION` whitespace run is quadratic on pathological tails. Absolute-path `classify` is slow on paths with 100k components. `edits()` is slower than before on 200 KB python-bound commands. +- `blast-radius-finds-breakage` treats a cited file as scratch only when it is under an absolute scratch path such as /tmp or /private/tmp. A project-relative proof script still counts as a file beyond the diff. +- `architect-sketches-before-implementation` reports "no sketch phase visible" for a run whose briefs are sealed. The verdict is right and the reason is not. +- `poteto-help-finds-need-and-routes` accepts a triple-backtick fence or an indented block that follows a paragraph. It does not accept `~~~` fences or a block nested in a list item. Its link check fails only when every extracted public link omits `skills/` and starts with a skill directory found in the approved skills snapshot. A refused or failed directory lookup does not prove a dead link. The check does not prove that any other link resolves. + +**Canon `steering-review-asks` oracle** + +- `.mock.calls` on a non-clipboard mock counts as a clipboard read. +- A `readText(el)` rendered helper is excluded as clipboard. +- A fake clipboard bound to an unrecognized constructor counts as rendered, as on trunk. +- Disjoint fragments of one tail fail the present ask. +- The `clipboard_reads` fixed point is quadratic on thousands of chained links. + ## Parent grading authority Trusted setup issues a parent-grade authorization before candidate turns. It captures the selected project identity, baseline order, skill pin, case, reference bytes, oracle code, and owned workspace allocation. It stores this authority in a private controller directory adjacent to `--out`, outside each run root and Hermes's run-root mount. The controller directory contains an issuer key and signed run authorizations. Candidate records, copied manifests, trace paths, and Git discovery output cannot add permission grants. Keep the controller directory at its original path. @@ -188,7 +218,7 @@ The interpreter fingerprint is also pinned in `grade_boundary.py`. `/usr/bin/git The policy denies file reads and writes by default. It grants the approved project and worktree files, private scratch, protected reference and skill copies, and the explicit native runtime directories and files. It permits reads of exact ancestor directories so descriptor traversal can open them. This exposes directory names at those ancestors, including ancestors outside the project. It grants no subtree access through those ancestors. The home directory appears as an exact ancestor grant. File-content grants under that home cover only the reviewed runtime and explicitly authorized run assets. Harness configuration, authentication, Keychains, and Docker sockets have no file grants. `HOME` and `TMPDIR` point to private scratch. -Controller and oracle reads use no-follow directory descriptors. Unexpected symlinks, hard links, special files, root replacement, or changed raw inputs refuse. Publication uses a controller-owned directory descriptor and atomic rename. A pre-existing verdict symlink or hard link refuses without truncating its target. Installed skills retain the existing selection order and the candidate bytes present at grade admission. Protected skill inputs contain `poteto-mode/SKILL.md` and every direct Markdown playbook, which are the skill files the current oracles inspect. Generated package directories in the trusted source are not grading inputs. The worker reads the protected admission copies and does not reinstall the recorded provenance pin. Case references, pre-turn history bytes, fallback skill inputs, metadata, and output are outside executable-check write grants. +Controller and oracle reads use no-follow directory descriptors. Unexpected symlinks, hard links, special files, root replacement, or changed raw inputs refuse. Publication uses a controller-owned directory descriptor and atomic rename. A pre-existing verdict symlink or hard link refuses without truncating its target. Installed skills retain the existing selection order and the candidate bytes present at grade admission. Protected skill inputs contain `poteto-mode/SKILL.md` and every direct Markdown playbook, which are the skill files the current oracles inspect. The snapshot also retains top-level directory names as empty directories for the public-link check, without copying their contents beyond those selected files. The worker reads the protected admission copies and does not reinstall the recorded provenance pin. Case references, pre-turn history bytes, fallback skill inputs, metadata, and output are outside executable-check write grants. Guarded stat and directory traversal errors produce `unsafe_link` refusals. Missing optional inputs keep their existing absent-file behavior. Publication syscall errors produce `output_unsafe` refusals. Failures before the atomic rename preserve prior verdict bytes. An error after rename can leave the published bytes in place. diff --git a/evals/promises/grade_boundary.py b/evals/promises/grade_boundary.py index 63995060..e745dfb0 100644 --- a/evals/promises/grade_boundary.py +++ b/evals/promises/grade_boundary.py @@ -301,6 +301,13 @@ def _copy_skill_inputs(source, destination): finally: playbooks.close() destination.mkdir(parents=True, mode=0o700) + for name in os.listdir(root.fd): + try: + directory = stat.S_ISDIR(root.info(name).st_mode) + except (GradeRefused, FileNotFoundError): + continue + if directory: + (destination / name).mkdir(exist_ok=True, mode=0o700) for rel in paths: target = destination / rel target.parent.mkdir(parents=True, exist_ok=True, mode=0o700) diff --git a/evals/promises/oracles.py b/evals/promises/oracles.py index e22da52b..d31a4ee0 100644 --- a/evals/promises/oracles.py +++ b/evals/promises/oracles.py @@ -62,7 +62,7 @@ def _files(path, pattern, recursive=False): SPAWN_TOOLS = {"Agent", "Task", "spawn_agent", "spawn_subagent", "delegate_task", "SubAgentActivity"} WAIT_TOOLS = {"get_command_or_subagent_output", "wait_agent", "TaskOutput"} FOLLOWUP_TOOLS = {"followup_task", "send_message"} -ASK_TOOLS = {"AskUserQuestion", "ask_user", "request_user_input", "AskQuestion", "ask_question"} +ASK_TOOLS = {"AskUserQuestion", "ask_user", "request_user_input", "request_user_input_async", "ask_user_question", "AskQuestion", "ask_question"} PRIVATE_PREFIXES = (".claude/", ".agents/", ".codex/", ".hermes/", ".grok/", ".git/") CACHE_PATH = re.compile(r"(?:^|/)(?:__pycache__|\.pytest_cache)(?:/|$)|\.pyc$|(?:^|/)node_modules/\.cache(?:/|$)") HEREDOC = re.compile(r"(?\]\"'`#?]+)") +LIST_ITEM = re.compile(r"\s*(?:[-*+]|\d+[.)])\s") +INDENTED_CODE = re.compile(r"(?: {4}|\t)\S") + + +def has_code_block(reply): + if "```" in reply: + return True + lines = reply.split("\n") + for n, line in enumerate(lines): + if INDENTED_CODE.match(line) and (n == 0 or not lines[n - 1].strip()): + before = lines[n - 2] if n >= 2 else "" + if not before.strip() or not (before[0].isspace() or LIST_ITEM.match(before)): + return True + return False + + +def link_drops_skills_dir(link_path, skills_root): + first = link_path.split("/", 1)[0] + if first in ("", ".", "..", "skills"): + return False + try: + return _is_dir(Path(skills_root) / first) + except Exception: + return False @oracle("poteto-help-finds-need-and-routes") @@ -2483,13 +2513,16 @@ def poteto_help(view): failures.append("the unclear need got no multiple-choice question over the help sections") else: named = [n for n in expect.get("names_any", []) if n.lower() in reply.lower()] - evidence += [f"expected names in the reply: {named}", f"code block: {'```' in reply}", f"public link: {PUBLIC_COPY in reply}"] + links = PUBLIC_LINK.findall(reply) + evidence += [f"expected names in the reply: {named}", f"code block: {has_code_block(reply)}", f"public link: {PUBLIC_COPY in reply}"] if not named: failures.append(f"the reply names none of {expect.get('names_any')}") - if "```" not in reply: + if not has_code_block(reply): failures.append("no example prompt in a code block") if PUBLIC_COPY not in reply: failures.append("no link to the public copy of the file the answer came from") + elif links and all(link_drops_skills_dir(path, view.skills_root) for path in links): + failures.append(f"every public link drops the skills/ directory and would 404: {links[:3]}") if failures: if not reply and view.killed: return inconclusive("run killed before a reply", *evidence) @@ -2503,10 +2536,15 @@ def blast_finds(view): if gate: return gate diff_files = set((view.case.get("expect") or {}).get("diff_files") or []) - cited = set(re.findall(r"\b[\w/.-]+\.(?:py|ts|sh|md|json)\b", view.final_reply)) - beyond = sorted(f for f in cited if f not in diff_files and not any(f.endswith("/" + d) or d.endswith("/" + f) for d in diff_files)) - fact = re.search(r"one fact|safe because", view.final_reply, re.I) - evidence = [f"files cited beyond the diff: {beyond[:5]}", f"one-fact wording: {bool(fact)}"] + cited = set(re.findall(r"(?= 2: + return inconclusive("sealed briefs hide which spawns were the runners; two sketches cannot be ruled out", *evidence) return inconclusive("run killed before the sketch fan-out", *evidence) if view.killed else failed("fewer than two runner sketches", *evidence) if len(wave) < 2: return failed("runners spawned sequentially", *evidence) diff --git a/evals/promises/test_grade_boundary.py b/evals/promises/test_grade_boundary.py index bf6a0bf3..448f2717 100644 --- a/evals/promises/test_grade_boundary.py +++ b/evals/promises/test_grade_boundary.py @@ -334,6 +334,30 @@ def test_unknown_cli_run_id_refuses_without_loading_record_paths(self): @unittest.skipUnless(NATIVE, "native parent grading requires the reviewed Darwin 25.6.0 arm64 runtime") class NativeParentBoundary(BoundaryFixtures): + def test_help_links_retain_skill_directory_proof_in_native_snapshots(self): + case = live.load_case("poteto-help-run") + pid = "poteto-help-finds-need-and-routes" + recorded = json.loads((live.HERE / "testdata/traces/oracle-edges/hermes-poteto-help-indented-prompt.json").read_text()) + for installed in (False, True): + if installed: + skills = self.project / ".agents/skills" + (skills / "poteto-mode").mkdir(parents=True) + (skills / "poteto-mode/SKILL.md").write_text("installed skill\n") + (skills / "interrogate").mkdir() + for path, verdict in ((None, "FAIL"), ("interrogate/SKILL.md", "FAIL"), + ("poteto-mode/SKILL.md", "FAIL"), ("README.md", "PASS"), + ("skills/interrogate/SKILL.md", "PASS"), ("unknown/SKILL.md", "PASS")): + with self.subTest(installed=installed, path=path): + reply = recorded["final_reply"] if path is None else ( + "Use /interrogate.\n\n```text\n/interrogate the branch\n```\n\n" + "https://github.com/mdsmithaustin/pstack/blob/main/" + path) + trace = {**recorded, "final_reply": reply} + record = {**self.record(case), "harness": trace["harness"]} + result = live.grade(self.authority(case, trace, record=record))["promises"][pid] + self.assertEqual(result["verdict"], verdict, result) + if verdict == "FAIL": + self.assertTrue(any("skills/" in failure for failure in result["failures"]), result) + def test_project_traversal_errors_return_cli_refusal_receipts(self): case = self.deslop() base = self.repository() diff --git a/evals/promises/test_oracles.py b/evals/promises/test_oracles.py index 5f4f8d77..e463672f 100644 --- a/evals/promises/test_oracles.py +++ b/evals/promises/test_oracles.py @@ -6,6 +6,7 @@ import time import unittest from pathlib import Path +from unittest import mock import live import oracles @@ -2717,6 +2718,163 @@ def test_one_readable_explainer_still_fails(self): self.assertEqual(result["verdict"], FAIL, result) +class SyncDf58112Edges(unittest.TestCase): + help_menu = "Get set up. Start a task with /poteto-mode. Pick a skill for a situation. Fix a run that went wrong. Make pstack my own." + + def grade_help(self, trace, case_id="poteto-help-run"): + return grade("poteto-help-finds-need-and-routes", trace, load_case(case_id)) + + def test_an_indented_example_prompt_is_a_code_block(self): + trace = load_trace("oracle-edges/hermes-poteto-help-indented-prompt.json") + trace["final_reply"] = trace["final_reply"].replace("blob/main/", "blob/main/skills/") + result = self.grade_help(trace) + self.assertEqual(result["verdict"], PASS, result) + + def test_a_public_link_without_the_skills_directory_fails(self): + result = self.grade_help(load_trace("oracle-edges/hermes-poteto-help-indented-prompt.json")) + self.assertEqual(result["verdict"], FAIL, result) + self.assertTrue(any("skills/" in f for f in result["failures"]), result) + self.assertFalse(any("code block" in f for f in result["failures"]), result) + + def test_indented_prose_after_a_list_item_is_not_a_code_block(self): + reply = ("Use /interrogate.\n\n- first step\n\n still the list item\n\n" + "https://github.com/mdsmithaustin/pstack/blob/main/skills/interrogate/SKILL.md") + result = self.grade_help(minimal(final_reply=reply)) + self.assertEqual(result["verdict"], FAIL, result) + self.assertIn("no example prompt in a code block", result["failures"]) + + def test_one_live_link_among_dead_ones_passes_the_link_check(self): + reply = ("Use /interrogate.\n\n```text\n/interrogate the branch\n```\n\n" + "https://github.com/mdsmithaustin/pstack/blob/main/interrogate/SKILL.md\n" + "https://github.com/mdsmithaustin/pstack/blob/main/skills/interrogate/SKILL.md") + self.assertEqual(self.grade_help(minimal(final_reply=reply))["verdict"], PASS) + + def test_a_link_to_a_repo_root_file_is_not_a_dead_link(self): + reply = ("Use /interrogate.\n\n```text\n/interrogate the branch\n```\n\n" + "https://github.com/mdsmithaustin/pstack/blob/main/README.md") + self.assertEqual(self.grade_help(minimal(final_reply=reply))["verdict"], PASS) + + def test_public_links_use_the_approved_skills_snapshot(self): + from grade_boundary import GradeRefused + + with tempfile.TemporaryDirectory(dir=HERE) as folder: + skills = Path(folder) / "skills" + for name in ("poteto-mode", "interrogate"): + (skills / name).mkdir(parents=True) + + def exists(path, kind=None): + if not Path(path).is_relative_to(skills): + raise GradeRefused("unapproved_path", path) + return Path(path).is_dir() if kind == "dir" else Path(path).exists() + + reader = mock.Mock(skills=skills, exists=exists) + with mock.patch.object(oracles, "_IO", reader): + trace = load_trace("oracle-edges/hermes-poteto-help-indented-prompt.json") + result = self.grade_help(trace) + self.assertEqual(result["verdict"], FAIL, result) + self.assertTrue(any("skills/" in f for f in result["failures"]), result) + for path in ("README.md", "skills/interrogate/SKILL.md", "/x", "./x", "../x"): + with self.subTest(path=path): + trace["final_reply"] = ("Use /interrogate.\n\n```text\n/interrogate the branch\n```\n\n" + "https://github.com/mdsmithaustin/pstack/blob/main/" + path) + self.assertEqual(self.grade_help(trace)["verdict"], PASS) + + def test_a_failed_skill_lookup_does_not_prove_a_dead_link(self): + from grade_boundary import GradeRefused + + trace = load_trace("oracle-edges/hermes-poteto-help-indented-prompt.json") + for error in (GradeRefused("unapproved_path", HERE), OSError("lookup failed")): + with self.subTest(error=error): + reader = mock.Mock(skills=HERE / "skills-snapshot", exists=mock.Mock(side_effect=error)) + with mock.patch.object(oracles, "_IO", reader): + self.assertEqual(self.grade_help(trace)["verdict"], PASS) + + def test_the_async_and_grok_ask_tools_count_as_a_question(self): + for name in ("codex-request-user-input-async.json", "grok-ask-user-question.json"): + trace = load_trace("oracle-edges/" + name) + call = trace["events"][0] + call["input"] = {"questions": [{"question": self.help_menu, "title": self.help_menu}]} + result = self.grade_help(trace, "poteto-help-unclear-run") + self.assertEqual(result["verdict"], PASS, (name, result)) + + def test_sealed_codex_briefs_do_not_prove_fewer_than_two_runners(self): + trace = load_trace("oracle-edges/codex-architect-sealed-briefs.json") + result = grade("architect-runs-arena-for-sketches", trace, load_case("architect-run")) + self.assertEqual(result["verdict"], INCONCLUSIVE, result) + + def test_one_sealed_spawn_still_fails_the_runner_count(self): + trace = load_trace("oracle-edges/codex-architect-sealed-briefs.json") + trace["spawns"] = trace["spawns"][:1] + result = grade("architect-runs-arena-for-sketches", trace, load_case("architect-run")) + self.assertEqual(result["verdict"], FAIL, result) + + def test_the_safety_fact_wording_names_the_one_fact(self): + result = grade("blast-radius-finds-breakage", load_trace("oracle-edges/hermes-blast-radius-safety-fact.json"), load_case("blast-radius-run")) + self.assertEqual(result["verdict"], PASS, result) + + def test_a_relative_verify_file_keeps_the_safety_fact_reply_passing(self): + trace = load_trace("oracle-edges/hermes-blast-radius-safety-fact.json") + trace["final_reply"] += "\nThe safety fact also holds for relay/ingest.py and verify_token.py." + result = grade("blast-radius-finds-breakage", trace, load_case("blast-radius-run")) + self.assertEqual(result["verdict"], PASS, result) + self.assertIn("verify_token.py", result["evidence"][0]) + + def test_relative_scratch_names_count_as_files_beyond_the_diff(self): + for path in ("verify_token.py", "tmp/x.py", "scratch.py", "repro.py", "baseline.py"): + with self.subTest(path=path): + trace = minimal(final_reply=f"The safety fact holds for {path}; relay/cache.py changed.") + result = grade("blast-radius-finds-breakage", trace, load_case("blast-radius-run")) + self.assertEqual(result["verdict"], PASS, result) + + def test_a_reply_citing_only_its_own_proof_script_needs_a_judge(self): + result = grade("blast-radius-finds-breakage", load_trace("oracle-edges/codex-blast-radius-own-proof-script.json"), load_case("blast-radius-run")) + self.assertEqual(result["verdict"], INCONCLUSIVE, result) + self.assertTrue(result.get("needs_judge"), result) + + def test_a_safety_fact_over_only_the_proof_script_is_not_a_pass(self): + trace = load_trace("oracle-edges/codex-blast-radius-own-proof-script.json") + trace["final_reply"] += "\nThe one fact it is safe because of: every caller passes offset 0." + result = grade("blast-radius-finds-breakage", trace, load_case("blast-radius-run")) + self.assertEqual(result["verdict"], INCONCLUSIVE, result) + + def test_a_reply_that_cites_nothing_but_the_diff_still_fails(self): + result = grade("blast-radius-finds-breakage", minimal(final_reply="The change to relay/cache.py looks safe to me."), load_case("blast-radius-run")) + self.assertEqual(result["verdict"], FAIL, result) + + +class RunRootTmp(unittest.TestCase): + def run_root(self): + patch = mock.patch.object(oracles, "SCRATCH_PREFIXES", ()) + patch.start() + self.addCleanup(patch.stop) + folder = tempfile.TemporaryDirectory(dir=HERE) + self.addCleanup(folder.cleanup) + root = Path(folder.name).resolve() + project = root / "w" / "relay" + project.mkdir(parents=True) + return root, project + + def checkpoint(self, root, project, file_path): + events = [{"seq": 54, "kind": "tool_call", "name": "Write", "input": {"file_path": str(file_path), "content": "x"}}] + reply = "I've stopped before implementing anything. Approve the design and I'll implement it." + return grade("architect-checkpoint-opt-in", minimal(events=events, final_reply=reply), load_case("architect-checkpoint-run"), project) + + def test_an_arena_graft_under_the_run_tmp_is_not_a_project_edit(self): + root, project = self.run_root() + graft = root / "tmp" / "arena-relay.Ab12Cd" / "synthesis" / "sketch" / "relay" / "cache.py" + self.assertEqual(self.checkpoint(root, project, graft)["verdict"], PASS) + + def test_a_write_beside_the_run_tmp_still_counts(self): + root, project = self.run_root() + self.assertEqual(self.checkpoint(root, project, root / "tmpfiles" / "relay" / "cache.py")["verdict"], FAIL) + self.assertEqual(self.checkpoint(root, project, project / "relay" / "cache.py")["verdict"], FAIL) + + def test_a_tmp_beside_a_project_outside_the_run_layout_is_not_the_run_tmp(self): + root, _ = self.run_root() + view = oracles.View(minimal(), load_case("architect-checkpoint-run"), root / "relay") + self.assertEqual(view.classify(str(root / "tmp" / "relay" / "cache.py")), "source") + + class RationaleReadPaths(unittest.TestCase): def arena_reads(self, commands, brief="Write `design-notes.md` with your reasoning."): calls = [{"seq": s, "kind": "tool_call", "name": "Agent", "input": {"description": f"cand {c}", "prompt": brief}} for s, c in zip((7, 9, 11, 13, 15), "ABCDE")] diff --git a/evals/promises/testdata/traces/oracle-edges/codex-architect-sealed-briefs.json b/evals/promises/testdata/traces/oracle-edges/codex-architect-sealed-briefs.json new file mode 100644 index 00000000..8d0ef23d --- /dev/null +++ b/evals/promises/testdata/traces/oracle-edges/codex-architect-sealed-briefs.json @@ -0,0 +1,517 @@ +{ + "harness": "codex", + "cli_version": "0.160.1", + "model": "gpt-6.1-sol", + "effort": "high", + "exit_code": 0, + "entry": "injected", + "worklist": [ + { + "seq": 6, + "carrier": "update_plan (inside exec; items from exec --json todo_list)", + "items": [ + { + "text": "Ground the import pipeline and its callers", + "state": "pending" + }, + { + "text": "Sketch and compare caller-first designs", + "state": "pending" + }, + { + "text": "Present the synthesized design for agreement", + "state": "pending" + }, + { + "text": "Implement only after the design checkpoint", + "state": "pending" + }, + { + "text": "Revisit the architecture if implementation exposes a pattern of problems", + "state": "pending" + } + ], + "turn": 0 + }, + { + "seq": 45, + "carrier": "update_plan (inside exec; items from exec --json todo_list)", + "items": [ + { + "text": "Ground the import pipeline and its callers", + "state": "pending" + }, + { + "text": "Sketch and compare caller-first designs", + "state": "pending" + }, + { + "text": "Present the synthesized design for agreement", + "state": "pending" + }, + { + "text": "Implement only after the design checkpoint", + "state": "pending" + }, + { + "text": "Revisit the architecture if implementation exposes a pattern of problems", + "state": "pending" + } + ], + "turn": 0 + }, + { + "seq": 75, + "carrier": "update_plan (inside exec; items from exec --json todo_list)", + "items": [ + { + "text": "Ground the import pipeline and its callers", + "state": "completed" + }, + { + "text": "Sketch and compare caller-first designs", + "state": "pending" + }, + { + "text": "Present the synthesized design for agreement", + "state": "pending" + }, + { + "text": "Implement only after the design checkpoint", + "state": "pending" + }, + { + "text": "Revisit the architecture if implementation exposes a pattern of problems", + "state": "pending" + } + ], + "turn": 0 + }, + { + "seq": 158, + "carrier": "update_plan (inside exec; items from exec --json todo_list)", + "items": [ + { + "text": "Ground the import pipeline and its callers", + "state": "completed" + }, + { + "text": "Sketch and compare caller-first designs", + "state": "pending" + }, + { + "text": "Present the synthesized design for agreement", + "state": "pending" + }, + { + "text": "Implement only after the design checkpoint", + "state": "pending" + }, + { + "text": "Revisit the architecture if implementation exposes a pattern of problems", + "state": "pending" + } + ], + "turn": 0 + }, + { + "seq": 162, + "carrier": "update_plan (inside exec; items from exec --json todo_list)", + "items": [ + { + "text": "Ground the import pipeline and its callers", + "state": "completed" + }, + { + "text": "Sketch and compare caller-first designs", + "state": "completed" + }, + { + "text": "Present the synthesized design for agreement", + "state": "pending" + }, + { + "text": "Implement only after the design checkpoint", + "state": "pending" + }, + { + "text": "Revisit the architecture if implementation exposes a pattern of problems", + "state": "pending" + } + ], + "turn": 0 + }, + { + "seq": 195, + "carrier": "update_plan (inside exec; items from exec --json todo_list)", + "items": [ + { + "text": "Ground the import pipeline and its callers", + "state": "completed" + }, + { + "text": "Sketch and compare caller-first designs", + "state": "completed" + }, + { + "text": "Present the synthesized design for agreement", + "state": "completed" + }, + { + "text": "Implement after agreement on the caller contract (deferred)", + "state": "pending" + }, + { + "text": "Revisit architecture if later implementation reveals repeated problems (conditional)", + "state": "pending" + } + ], + "turn": 0 + }, + { + "seq": 195, + "carrier": "update_plan (inside exec; items from exec --json todo_list)", + "items": [ + { + "text": "Ground the import pipeline and its callers", + "state": "completed" + }, + { + "text": "Sketch and compare caller-first designs", + "state": "completed" + }, + { + "text": "Present the synthesized design for agreement", + "state": "completed" + }, + { + "text": "Implement after agreement on the caller contract (deferred)", + "state": "pending" + }, + { + "text": "Revisit architecture if later implementation reveals repeated problems (conditional)", + "state": "pending" + } + ], + "turn": 0 + } + ], + "spawns": [ + { + "seq": 29, + "turn": 0, + "tool": "spawn_agent", + "persona": null, + "model": "gpt-6.1-sol", + "effort": "max", + "fork_turns": "none", + "prompt_head": null, + "x_prompt_encrypted": true, + "x_thread_id": "01a1199f-09a9-7430-9afa-18d9d8ab2a6a", + "x_child_first_reply": "I\u2019m using the how skill to trace the import pipeline and its callers. I\u2019ll keep this read-only and identify the contracts a redesign would need to preserve.\n", + "x_persona_line": null, + "x_agent_role": "default" + }, + { + "seq": 42, + "turn": 0, + "tool": "spawn_agent", + "persona": null, + "model": "gpt-6.1-sol", + "effort": "high", + "fork_turns": "none", + "prompt_head": null, + "x_prompt_encrypted": true, + "x_thread_id": "01a1199f-4d57-7170-96a1-99fbeba9c7e5", + "x_child_first_reply": "I\u2019m using the why skill to collect the import path\u2019s history and stated constraints. I\u2019ll distinguish documented requirements from behavior that has no recorded", + "x_persona_line": null, + "x_agent_role": "default" + }, + { + "seq": 63, + "turn": 0, + "tool": "spawn_agent", + "persona": null, + "model": "gpt-6.1-sol", + "effort": "max", + "fork_turns": "none", + "prompt_head": null, + "x_prompt_encrypted": true, + "x_thread_id": "01a119a0-8955-72a2-bb59-99b0ab69d45e", + "x_child_first_reply": "I'm using the why framework and unslop pass to separate documented constraints from inferred rationale.\n", + "x_persona_line": null, + "x_agent_role": "default" + }, + { + "seq": 80, + "turn": 0, + "tool": "spawn_agent", + "persona": null, + "model": "gpt-6.1-sol", + "effort": "max", + "fork_turns": "none", + "prompt_head": null, + "x_prompt_encrypted": true, + "x_thread_id": "01a119a1-9799-7842-8907-10e44d17c945", + "x_child_first_reply": "I\u2019m applying the unslop skill to the design prose. I\u2019ll ground the direct-function candidate in the shared task and compare it with the pipeline and prepare/app", + "x_persona_line": null, + "x_agent_role": "default" + }, + { + "seq": 83, + "turn": 0, + "tool": "spawn_agent", + "persona": null, + "model": "gpt-6.1-sol", + "effort": "xhigh", + "fork_turns": "none", + "prompt_head": null, + "x_prompt_encrypted": true, + "x_thread_id": "01a119a1-af93-7373-9043-750871f38288", + "x_child_first_reply": "I\u2019m using the unslop skill for the design write-up. I\u2019ll read the shared brief, then develop the Importer candidate and test its caller cost against the alterna", + "x_persona_line": null, + "x_agent_role": "default" + }, + { + "seq": 86, + "turn": 0, + "tool": "spawn_agent", + "persona": null, + "model": "gpt-6-luna", + "effort": "xhigh", + "fork_turns": "none", + "prompt_head": null, + "x_prompt_encrypted": true, + "x_child_first_reply": null, + "x_persona_line": null, + "x_agent_role": "default" + }, + { + "seq": 147, + "turn": 0, + "tool": "spawn_agent", + "persona": null, + "model": "gpt-6-luna", + "effort": "xhigh", + "fork_turns": "none", + "prompt_head": null, + "x_prompt_encrypted": true, + "x_thread_id": "01a119a7-1c6e-7d30-a473-808a1ae6fe73", + "x_child_first_reply": "I\u2019ll compare the candidates against the weighted rubric, checking each design\u2019s public usage and the specified failure, retry, ID, and format cases.", + "x_persona_line": null, + "x_agent_role": "default" + } + ], + "files_read": [ + "/private/tmp/sync-rerun/codex/relay-nau0vbp9/w/relay/.agents/skills/unslop/SKILL.md", + "/private/tmp/sync-rerun/codex/relay-nau0vbp9/w/relay/.agents/skills/architect/SKILL.md", + "/private/tmp/sync-rerun/codex/relay-nau0vbp9/w/relay/.agents/skills/how/SKILL.md", + "/private/tmp/sync-rerun/codex/relay-nau0vbp9/w/relay/.agents/skills/pstack-harness/SKILL.md", + "/private/tmp/sync-rerun/codex/relay-nau0vbp9/w/relay/.agents/skills/arena/SKILL.md", + "/private/tmp/sync-rerun/codex/relay-nau0vbp9/w/relay/.agents/skills/architect/references/runner-prompt.md", + "/private/tmp/sync-rerun/codex/relay-nau0vbp9/w/relay/.agents/skills/architect/references/rationale-template.md", + "/private/tmp/sync-rerun/codex/relay-nau0vbp9/w/relay/.agents/skills/architect/references/design-red-flags.md", + "/private/tmp/sync-rerun/codex/relay-nau0vbp9/w/relay/.agents/skills/how/references/explainer-prompt.md", + "/private/tmp/sync-rerun/codex/relay-nau0vbp9/w/relay/.agents/skills/pstack-harness/references/portable-paths.md", + "/private/tmp/sync-rerun/codex/relay-nau0vbp9/w/relay/README.md", + "/private/tmp/sync-rerun/codex/relay-nau0vbp9/w/relay/relay/__main__.py", + "/private/tmp/sync-rerun/codex/relay-nau0vbp9/w/relay/relay/ingest.py", + "/private/tmp/sync-rerun/codex/relay-nau0vbp9/w/relay/relay/store.py", + "/private/tmp/sync-rerun/codex/relay-nau0vbp9/w/relay/relay/cache.py", + "/private/tmp/sync-rerun/codex/relay-nau0vbp9/w/relay/relay/parse.py", + "/private/tmp/sync-rerun/codex/relay-nau0vbp9/w/relay/relay/oldparse.py", + "/private/tmp/sync-rerun/codex/relay-nau0vbp9/w/relay/relay/sources/*.py", + "/private/tmp/sync-rerun/codex/relay-nau0vbp9/w/relay/.agents/skills/why/SKILL.md", + "/private/tmp/sync-rerun/codex/relay-nau0vbp9/w/relay/.agents/skills/spec-probes/SKILL.md", + "/private/tmp/sync-rerun/codex/relay-nau0vbp9/w/relay/.agents/skills/verify-commands/SKILL.md", + "/private/tmp/sync-rerun/codex/relay-nau0vbp9/w/relay/.agents/skills/documentation-impact/SKILL.md", + "/private/tmp/sync-rerun/codex/relay-nau0vbp9/w/relay/.agents/skills/why/references/epistemics.md", + "/private/tmp/sync-rerun/codex/relay-nau0vbp9/w/relay/.agents/skills/why/references/investigator-prompt.md", + "/private/tmp/sync-rerun/codex/relay-nau0vbp9/w/relay/.agents/skills/why/references/sources/source-control.md", + "/private/tmp/sync-rerun/codex/relay-nau0vbp9/w/relay/.agents/skills/why/references/synthesizer-prompt.md", + "/private/tmp/sync-rerun/codex/relay-nau0vbp9/w/relay/.agents/skills/technical-writing/SKILL.md", + "/private/tmp/sync-rerun/codex/relay-nau0vbp9/w/relay/relay/summarize.py", + "/private/tmp/sync-rerun/codex/relay-nau0vbp9/w/relay/relay/verify.py", + "/private/tmp/sync-rerun/codex/relay-nau0vbp9/w/relay/relay/retry.py", + "/private/tmp/sync-rerun/codex/relay-nau0vbp9/w/relay/relay/export.py", + "/private/tmp/sync-rerun/codex/relay-nau0vbp9/w/relay/relay/dedupe.py", + "/private/tmp/sync-rerun/codex/relay-nau0vbp9/w/relay/relay/__init__.py", + "/private/tmp/sync-rerun/codex/relay-nau0vbp9/w/relay/tests/test_parse.py", + "/private/tmp/sync-rerun/codex/relay-nau0vbp9/w/relay/tools/old_callers.sh", + "/private/tmp/sync-rerun/codex/relay-nau0vbp9/w/relay/feeds/sample.json", + "/private/tmp/sync-rerun/codex/relay-nau0vbp9/w/relay/feeds/sample.csv", + "/private/tmp/relay-import-design.xPLeac/importer.md", + "/private/tmp/relay-import-design.xPLeac/function.md" + ], + "final_reply": "The [design](/private/tmp/sync-rerun/codex/relay-nau0vbp9/w/relay/docs/import-pipeline-design.md) keeps the existing caller API:\n\n```python\nfrom relay.ingest import import_feed\n\ncount = import_feed(\"feeds/sample.json\", \"out/\")\n```\n\nOne call handles fresh loading, validation, and writing. The proposed contract is:\n\n- Count source rows, including duplicates; the last duplicate ID wins.\n- `FeedError` guarantees import changed no destination state.\n- `StoreWriteError` reports completed rows; earlier writes remain.\n- Each file replacement is atomic; the batch can be partial.\n\nThe document includes signatures, module ownership, caller migration, and the comparison with importer-object and preview/apply designs. Empty-feed policy and portable ID rules remain explicit decisions.\n\nIndependent design review passed. No implementation code changed.", + "events": [ + { + "kind": "tool_call", + "name": "spawn_agent", + "input": { + "task_name": "ground_import", + "agent_type": "default", + "model": "gpt-6.1-sol", + "reasoning_effort": "max", + "fork_turns": "none", + "message": "gAAAAABqxxIv9J_zujnGtFQmHzEsc-NgDhNP934OgweKyiOpxkpU16GwAyYU..." + }, + "id": "call_2b18f573027a41cdbe73aee67125e9d4", + "seq": 29, + "turn": 0 + }, + { + "kind": "tool_result", + "name": "spawn_agent", + "ok": true, + "output_head": "{\"task_name\":\"/root/ground_import\"}", + "id": "call_2b18f573027a41cdbe73aee67125e9d4", + "seq": 31, + "turn": 0 + }, + { + "kind": "tool_call", + "name": "spawn_agent", + "input": { + "task_name": "import_rationale", + "agent_type": "default", + "model": "gpt-6.1-sol", + "reasoning_effort": "high", + "fork_turns": "none", + "message": "gAAAAABqxxJA-cIEprhNVzmWu9p-GbmDiMljfz7zdFkCA0ddTBrYZMiBI8Uo..." + }, + "id": "call_9b0832a037b3413e84bd3824b124f052", + "seq": 42, + "turn": 0 + }, + { + "kind": "tool_result", + "name": "spawn_agent", + "ok": true, + "output_head": "{\"task_name\":\"/root/import_rationale\"}", + "id": "call_9b0832a037b3413e84bd3824b124f052", + "seq": 44, + "turn": 0 + }, + { + "kind": "tool_call", + "name": "spawn_agent", + "input": { + "task_name": "rationale_constraints", + "agent_type": "default", + "model": "gpt-6.1-sol", + "reasoning_effort": "max", + "fork_turns": "none", + "message": "gAAAAABqxxKRlclwzNTKdmcWqlC1-Y1HJM_SJ1jSddE4RP5Rr6CWYWd0b9Ef..." + }, + "id": "call_16390692e88e42e5b7c6b9199d68881d", + "seq": 63, + "turn": 0 + }, + { + "kind": "tool_result", + "name": "spawn_agent", + "ok": true, + "output_head": "{\"task_name\":\"/root/rationale_constraints\"}", + "id": "call_16390692e88e42e5b7c6b9199d68881d", + "seq": 65, + "turn": 0 + }, + { + "kind": "tool_call", + "name": "spawn_agent", + "input": { + "task_name": "design_function", + "agent_type": "default", + "model": "gpt-6.1-sol", + "reasoning_effort": "max", + "fork_turns": "none", + "message": "gAAAAABqxxLWMlURsqRaBdA4DCZ6oY0zoX27VovzUKUW6ZRM1E_e4tuNfVpp..." + }, + "id": "call_d3f418141b164d24a522288b31b37afe", + "seq": 80, + "turn": 0 + }, + { + "kind": "tool_result", + "name": "spawn_agent", + "ok": true, + "output_head": "{\"task_name\":\"/root/design_function\"}", + "id": "call_d3f418141b164d24a522288b31b37afe", + "seq": 82, + "turn": 0 + }, + { + "kind": "tool_call", + "name": "spawn_agent", + "input": { + "task_name": "design_importer", + "agent_type": "default", + "model": "gpt-6.1-sol", + "reasoning_effort": "xhigh", + "fork_turns": "none", + "message": "gAAAAABqxxLdeNbcvQaIfc3zFBFp2VyUENe9Q9UuW2OszsIwwbzw5CGqHzdl..." + }, + "id": "call_a723ea058e1e466e9d6e3308ad07083f", + "seq": 83, + "turn": 0 + }, + { + "kind": "tool_result", + "name": "spawn_agent", + "ok": true, + "output_head": "{\"task_name\":\"/root/design_importer\"}", + "id": "call_a723ea058e1e466e9d6e3308ad07083f", + "seq": 85, + "turn": 0 + }, + { + "kind": "tool_call", + "name": "spawn_agent", + "input": { + "task_name": "design_plan", + "agent_type": "default", + "model": "gpt-6-luna", + "reasoning_effort": "xhigh", + "fork_turns": "none", + "message": "gAAAAABqxxLitOIlb5mAViaB8EyeupBGG__8n8mamTr7w_5Ms53t-NIeFTqJ..." + }, + "id": "call_7a3b61432ed849428034a22ba46c46aa", + "seq": 86, + "turn": 0 + }, + { + "kind": "tool_result", + "name": "spawn_agent", + "ok": true, + "output_head": "collab spawn failed: agent thread limit reached", + "id": "call_7a3b61432ed849428034a22ba46c46aa", + "seq": 87, + "turn": 0 + }, + { + "kind": "tool_call", + "name": "spawn_agent", + "input": { + "task_name": "design_judge", + "agent_type": "default", + "model": "gpt-6-luna", + "reasoning_effort": "xhigh", + "fork_turns": "none", + "message": "gAAAAABqxxRARIOEcn6-RwlrmmukTZn1Ty5lryWtJcpbsMXTtOIcE6CjpNyO..." + }, + "id": "call_54e00b1134ad4e32bfa52e3a83704d95", + "seq": 147, + "turn": 0 + }, + { + "kind": "tool_result", + "name": "spawn_agent", + "ok": true, + "output_head": "{\"task_name\":\"/root/design_judge\"}", + "id": "call_54e00b1134ad4e32bfa52e3a83704d95", + "seq": 149, + "turn": 0 + } + ] +} \ No newline at end of file diff --git a/evals/promises/testdata/traces/oracle-edges/codex-blast-radius-own-proof-script.json b/evals/promises/testdata/traces/oracle-edges/codex-blast-radius-own-proof-script.json new file mode 100644 index 00000000..dc878f44 --- /dev/null +++ b/evals/promises/testdata/traces/oracle-edges/codex-blast-radius-own-proof-script.json @@ -0,0 +1,110 @@ +{ + "harness": "codex", + "cli_version": "0.160.1", + "model": "gpt-6.1-sol", + "effort": "high", + "exit_code": 0, + "entry": "injected", + "worklist": [], + "spawns": [], + "files_read": [ + "/private/tmp/sync-rerun/codex/relay-_iee4f2j/w/relay/.agents/skills/unslop/SKILL.md", + "/private/tmp/sync-rerun/codex/relay-_iee4f2j/w/relay/.agents/skills/blast-radius/SKILL.md", + "/private/tmp/sync-rerun/codex/relay-_iee4f2j/w/relay/.agents/skills/why/SKILL.md", + "/private/tmp/sync-rerun/codex/relay-_iee4f2j/w/relay/.agents/skills/verify-commands/SKILL.md", + "/private/tmp/sync-rerun/codex/relay-_iee4f2j/w/relay/.agents/skills/pstack-harness/SKILL.md", + "/private/tmp/sync-rerun/codex/relay-_iee4f2j/w/relay/README.md", + "/private/tmp/sync-rerun/codex/relay-_iee4f2j/w/relay/relay/cache.py", + "/private/tmp/sync-rerun/codex/relay-_iee4f2j/w/relay/relay/ingest.py", + "/private/tmp/sync-rerun/codex/relay-_iee4f2j/w/relay/relay/parse.py", + "/private/tmp/sync-rerun/codex/relay-_iee4f2j/w/relay/relay/store.py", + "/private/tmp/sync-rerun/codex/relay-_iee4f2j/w/relay/relay/__main__.py", + "/private/tmp/sync-rerun/codex/relay-_iee4f2j/w/relay/relay/sources/*.py", + "/private/tmp/sync-rerun/codex/relay-_iee4f2j/w/relay/relay/export.py", + "/private/tmp/sync-rerun/codex/relay-_iee4f2j/w/relay/relay/dedupe.py", + "/private/tmp/sync-rerun/codex/relay-_iee4f2j/w/relay/relay/retry.py", + "/private/tmp/sync-rerun/codex/relay-_iee4f2j/w/relay/relay/summarize.py", + "/private/tmp/sync-rerun/codex/relay-_iee4f2j/w/relay/relay/verify.py", + "/private/tmp/sync-rerun/codex/relay-_iee4f2j/w/relay/tests/test_parse.py", + "/private/tmp/sync-rerun/codex/relay-_iee4f2j/w/relay/feeds/sample.json", + "/private/tmp/sync-rerun/codex/relay-_iee4f2j/w/relay/feeds/sample.csv", + "/private/tmp/sync-rerun/codex/relay-_iee4f2j/w/relay/relay/sources/__init__.py", + "/private/tmp/sync-rerun/codex/relay-_iee4f2j/w/relay/relay/sources/json_source.py", + "/private/tmp/sync-rerun/codex/relay-_iee4f2j/w/relay/relay/sources/csv_source.py", + "/private/tmp/sync-rerun/codex/relay-_iee4f2j/w/relay/.agents/skills/why/references/epistemics.md" + ], + "final_reply": "Reviewed `fdb6450` against its parent. **Safe for current callers, with one confirmed Python API compatibility change.**\n\nThe key now returns `feed` directly. Besides dropping the offset, it stops converting the feed to a string. See [cache.py](/private/tmp/sync-rerun/codex/relay-_iee4f2j/w/relay/relay/cache.py:4).\n\nThe safety fact is **proven at level 4, by running real code**: every current consumer passes offset `0`, and the JSON and CSV loaders return the whole feed. Import, summary, and verification produced identical results and stored files under both key implementations, across all 12 execution orderings.\n\nOne compatibility risk is confirmed. A string path and an equivalent `pathlib.Path` previously shared a cache entry. They now create separate entries. After importing with a string, changing the file, and verifying with a `Path`, the proof produced:\n\n```text\nBefore: no verification problems; 1 cache entry\nAfter: x: missing name; 2 cache entries\n```\n\nThis follows from [cache.py\u2019s dictionary lookup](/private/tmp/sync-rerun/codex/relay-_iee4f2j/w/relay/relay/cache.py:9). Likelihood is low for this repo: the CLI supplies strings, and no mixed-type caller exists here. Python callers can incur extra reads and observe different feed contents within one process.\n\nCleared by execution:\n\n- Different feeds retain separate entries.\n- Failed loads leave no cached value, and retry recovers.\n- Different offsets return identical whole-feed content with the shipped loaders.\n\nThe [proof script](/private/tmp/relay-blast-radius.INDrKG/proof.py) passed **5 tests**; the existing suite passed **1 test**. Repository files remain unchanged.\n\nBefore merging, decide whether string and `Path` inputs must share cached contents. If so, retain string conversion. Rerun the proof from the repository root with:\n\n```sh\npython3 /private/tmp/relay-blast-radius.INDrKG/proof.py \"$PWD\"\n```\n\nFailure signal: nonzero exit or a unittest `FAILED` summary.", + "events": [ + { + "kind": "text", + "text": "I\u2019m using the blast-radius skill to trace the diff\u2019s downstream effects and run a proof of the key safety assumption. I\u2019ll use unslop for the writeup.\n", + "seq": 7, + "turn": 0 + }, + { + "kind": "text", + "text": "The working tree is clean, so I\u2019m reviewing the latest commit, `fdb6450`, which removes the offset from the cache key. Every current caller passes offset `0`, and the loader reads the whole feed. I\u2019m checking that import, summary, verification, and retry still behave the same with the old and new keys.\n", + "seq": 28, + "turn": 0 + }, + { + "kind": "tool_call", + "name": "exec", + "input": { + "code": "text(await tools.apply_patch(\"*** Begin Patch\\n*** Add File:..." + }, + "id": "call_2990c4a4dd924f0eae89832e5a51d1ab", + "seq": 39, + "turn": 0 + }, + { + "kind": "tool_call", + "name": "FileChange", + "input": { + "changes": { + "/private/tmp/relay-blast-radius.INDrKG/proof.py": { + "type": "add", + "content": "import itertools\nimport json\nfrom pathlib import Path\nimport..." + } + }, + "status": "completed", + "stderr": "" + }, + "seq": 40, + "turn": 0 + }, + { + "kind": "tool_call", + "name": "exec", + "input": { + "code": "text(await tools.exec_command({cmd:\"python3 /private/tmp/rel..." + }, + "id": "call_6eb0b2a3ed3641e09595bfffeb23a583", + "seq": 42, + "turn": 0 + }, + { + "kind": "tool_call", + "name": "exec_command", + "input": { + "cmd": "python3 /private/tmp/relay-blast-radius.INDrKG/proof.py \"$PWD\"", + "cwd": "/private/tmp/sync-rerun/codex/relay-_iee4f2j/w/relay" + }, + "id": "exec-3cb85634-e6f9-4da1-9044-c211f16124b0", + "seq": 43, + "turn": 0 + }, + { + "kind": "text", + "text": "The proof passed all 12 import, summary, and verification orderings against the parent commit. I also confirmed a compatibility change: a string path and an equivalent `Path` used to share a cache entry; now they load separately and can see different contents if the file changes. The CLI uses strings, and I found no mixed-type caller in the repo.\n", + "seq": 48, + "turn": 0 + }, + { + "kind": "text", + "text": "Reviewed `fdb6450` against its parent. **Safe for current callers, with one confirmed Python API compatibility change.**\n\nThe key now returns `feed` directly. Besides dropping the offset, it stops converting the feed to a string. See [cache.py](/private/tmp/sync-rerun/codex/relay-_iee4f2j/w/relay/relay/cache.py:4).\n\nThe safety fact is **proven at level 4, by running real code**: every current consumer passes offset `0`, and the JSON and CSV loaders return the whole feed. Import, summary, and verification produced identical results and stored files under both key implementations, across all 12 execution orderings.\n\nOne compatibility risk is confirmed. A string path and an equivalent `pathlib.Path` previously shared a cache entry. They now create separate entries. After importing with a string, changing the file, and verifying with a `Path`, the proof produced:\n\n```text\nBefore: no verification problems; 1 cache entry\nAfter: x: missing name; 2 cache entries\n```\n\nThis follows from [cache.py\u2019s dictionary lookup](/private/tmp/sync-rerun/codex/relay-_iee4f2j/w/relay/relay/cache.py:9). Likelihood is low for this repo: the CLI supplies strings, and no mixed-type caller exists here. Python callers can incur extra reads and observe different feed contents within one process.\n\nCleared by execution:\n\n- Different feeds retain separate entries.\n- Failed loads leave no cached value, and retry recovers.\n- Different offsets return identical whole-feed content with the shipped loaders.\n\nThe [proof script](/private/tmp/relay-blast-radius.INDrKG/proof.py) passed **5 tests**; the existing suite passed **1 test**. Repository files remain unchanged.\n\nBefore merging, decide whether string and `Path` inputs must share cached contents. If so, retain string conversion. Rerun the proof from the repository root with:\n\n```sh\npython3 /private/tmp/relay-blast-radius.INDrKG/proof.py \"$PWD\"\n```\n\nFailure signal: nonzero exit or a unittest `FAILED` summary.", + "seq": 49, + "turn": 0 + } + ] +} \ No newline at end of file diff --git a/evals/promises/testdata/traces/oracle-edges/codex-request-user-input-async.json b/evals/promises/testdata/traces/oracle-edges/codex-request-user-input-async.json new file mode 100644 index 00000000..22988500 --- /dev/null +++ b/evals/promises/testdata/traces/oracle-edges/codex-request-user-input-async.json @@ -0,0 +1,37 @@ +{ + "harness": "codex", + "cli_version": "0.160.0", + "model": "gpt-6.1-sol", + "effort": "high", + "exit_code": 0, + "entry": "injected", + "worklist": [], + "spawns": [], + "files_read": [], + "final_reply": "", + "events": [ + { + "kind": "tool_call", + "name": "request_user_input_async", + "input": { + "questions": [ + { + "title": "Which queue should I run, and which GitHub repository should receive the PRs? Send the queue items or a queue link and the repository URL. This checkout has no queue file or configured Git remote." + } + ] + }, + "id": "call_XA7tVqaDHmn7AYh8UqPSa4ma", + "seq": 24, + "turn": 0 + }, + { + "kind": "tool_result", + "name": "request_user_input_async", + "ok": true, + "output_head": "{\"accepted\":true}", + "id": "call_XA7tVqaDHmn7AYh8UqPSa4ma", + "seq": 25, + "turn": 0 + } + ] +} \ No newline at end of file diff --git a/evals/promises/testdata/traces/oracle-edges/grok-ask-user-question.json b/evals/promises/testdata/traces/oracle-edges/grok-ask-user-question.json new file mode 100644 index 00000000..45f81b29 --- /dev/null +++ b/evals/promises/testdata/traces/oracle-edges/grok-ask-user-question.json @@ -0,0 +1,47 @@ +{ + "harness": "grok", + "cli_version": "1.0.46", + "model": "grok-4.7-build", + "effort": "high", + "exit_code": 0, + "entry": "injected", + "worklist": [], + "spawns": [], + "files_read": [], + "final_reply": "", + "events": [ + { + "kind": "tool_call", + "name": "ask_user_question", + "input": { + "questions": [ + { + "question": "Want to pick a model for each role and a reasoning budget now?", + "options": [ + { + "label": "Now (Recommended)", + "description": "Type /setup-pstack before the first task. It writes ~/.agents/pstack-models.md. The getting-started steps above still stand." + }, + { + "label": "Later", + "description": "Every role keeps its default model until you run /setup-pstack." + } + ] + } + ] + }, + "id": "call-36f074ed-34c1-4085-8e1f-0d8920fa4b91-7", + "seq": 17, + "turn": 0 + }, + { + "kind": "tool_result", + "name": "ask_user_question", + "ok": true, + "output_head": "No user is available to answer questions in this non-interactive session. Continue with your best judgment; do not wait for clarification.", + "id": "call-36f074ed-34c1-4085-8e1f-0d8920fa4b91-7", + "seq": 18, + "turn": 0 + } + ] +} \ No newline at end of file diff --git a/evals/promises/testdata/traces/oracle-edges/hermes-blast-radius-safety-fact.json b/evals/promises/testdata/traces/oracle-edges/hermes-blast-radius-safety-fact.json new file mode 100644 index 00000000..a84f5854 --- /dev/null +++ b/evals/promises/testdata/traces/oracle-edges/hermes-blast-radius-safety-fact.json @@ -0,0 +1,119 @@ +{ + "harness": "hermes", + "cli_version": "Hermes Agent v0.21.4 (2026.9.21) \u00b7 upstream d337b736", + "model": "gpt-5.6-sol", + "effort": "high", + "exit_code": 0, + "entry": "injected", + "worklist": [ + { + "seq": 2, + "turn": 0, + "carrier": "todo_list", + "items": [ + { + "text": "Inspect the active diff, repository guidance, and affected symbols", + "state": "in progress" + }, + { + "text": "Trace callers, dependencies, and non-obvious blast radius", + "state": "pending" + }, + { + "text": "Run a focused proof and relevant test suite", + "state": "pending" + }, + { + "text": "Report confirmed risks, cleared concerns, and merge recommendation", + "state": "pending" + } + ] + }, + { + "seq": 38, + "turn": 0, + "carrier": "todo_list", + "items": [ + { + "text": "Inspect the active diff, repository guidance, and affected symbols", + "state": "completed" + }, + { + "text": "Trace callers, dependencies, and non-obvious blast radius", + "state": "in progress" + }, + { + "text": "Run a focused proof and relevant test suite", + "state": "pending" + }, + { + "text": "Report confirmed risks, cleared concerns, and merge recommendation", + "state": "pending" + } + ] + }, + { + "seq": 58, + "turn": 0, + "carrier": "todo_list", + "items": [ + { + "text": "Inspect the active diff, repository guidance, and affected symbols", + "state": "completed" + }, + { + "text": "Trace callers, dependencies, and non-obvious blast radius", + "state": "completed" + }, + { + "text": "Run a focused proof and relevant test suite", + "state": "completed" + }, + { + "text": "Report confirmed risks, cleared concerns, and merge recommendation", + "state": "in progress" + } + ] + }, + { + "seq": 60, + "turn": 0, + "carrier": "todo_list", + "items": [ + { + "text": "Inspect the active diff, repository guidance, and affected symbols", + "state": "completed" + }, + { + "text": "Trace callers, dependencies, and non-obvious blast radius", + "state": "completed" + }, + { + "text": "Run a focused proof and relevant test suite", + "state": "completed" + }, + { + "text": "Report confirmed risks, cleared concerns, and merge recommendation", + "state": "completed" + } + ] + } + ], + "spawns": [], + "files_read": [ + "/private/tmp/sync-rerun/hermes/relay-triw8a0x/w/relay/README.md", + "/private/tmp/sync-rerun/hermes/relay-triw8a0x/w/relay/relay/ingest.py", + "/private/tmp/sync-rerun/hermes/relay-triw8a0x/w/relay/relay/__main__.py", + "/private/tmp/sync-rerun/hermes/relay-triw8a0x/w/relay/relay/sources/__init__.py", + "/private/tmp/sync-rerun/hermes/relay-triw8a0x/w/relay/relay/sources/json_source.py", + "/private/tmp/sync-rerun/hermes/relay-triw8a0x/w/relay/relay/sources/csv_source.py", + "/private/tmp/sync-rerun/hermes/relay-triw8a0x/w/relay/relay/__init__.py", + "/private/tmp/sync-rerun/hermes/relay-triw8a0x/w/relay/tests/test_parse.py", + "/private/tmp/sync-rerun/hermes/relay-triw8a0x/w/relay/.agents/skills/unslop/SKILL.md", + "/private/tmp/sync-rerun/hermes/relay-triw8a0x/w/relay/relay/cache.py", + "/private/tmp/sync-rerun/hermes/relay-triw8a0x/w/relay/relay/summarize.py", + "/private/tmp/sync-rerun/hermes/relay-triw8a0x/w/relay/relay/verify.py" + ], + "final_reply": "Finding\n\n[P1] Keep `offset` in the memoization key. `relay/cache.py:4-12`\n\n`fetch()` still accepts an offset, but `cache_key()` now ignores it. Calls for the same feed at different offsets return the first cached value, and later loaders never run.\n\nProof using the real implementation before and after this commit:\n\n before: ('page-0', 'page-100', [('partners', 0), ('partners', 100)])\n after: ('page-0', 'page-0', [('partners', 0)])\n AssertionError: different offsets now collide\n\nThis can return the wrong page or stale data to any downstream caller that varies `offset`. Likelihood is high for such callers, and the result is silent data corruption rather than an exception.\n\nWhat it does\n\nThe change shortens the cache key, but also changes cache identity from `(feed, offset)` to `feed`. Those are separate concerns. Log formatting should not determine memoization semantics.\n\nThe one safety fact\n\nAll checked-in application paths currently pass offset `0`:\n\n- `relay/ingest.py:9`\n- `relay/summarize.py:13`\n- `relay/verify.py:8`\n\nI instrumented and ran all three paths. They produced:\n\n application fetch calls: [\n ('feeds/sample.json', 0),\n ('feeds/sample.json', 0),\n ('feeds/sample.json', 0)\n ]\n\nThis proves the current in-repository application is unaffected. It does not prove safety for callers of `relay.cache.fetch()` outside the repository.\n\nCleared\n\n- The existing suite passes: 1 test, 0 failures.\n- The working tree remains clean.\n- No checked-in code logs the cache key, so this commit does not improve any logging path present in this repository.\n- Existing tests cover parsing only and would not detect this regression.\n\nBefore merge\n\nDo not merge as written. Preserve `offset` in the memoization key and shorten only the value used for logging. Add a cache test that fetches one feed at two offsets with different loader results and asserts that both loaders run and both results are returned.", + "events": [] +} \ No newline at end of file diff --git a/evals/promises/testdata/traces/oracle-edges/hermes-poteto-help-indented-prompt.json b/evals/promises/testdata/traces/oracle-edges/hermes-poteto-help-indented-prompt.json new file mode 100644 index 00000000..e8aae46c --- /dev/null +++ b/evals/promises/testdata/traces/oracle-edges/hermes-poteto-help-indented-prompt.json @@ -0,0 +1,16 @@ +{ + "harness": "hermes", + "cli_version": "Hermes Agent v0.21.4 (2026.9.21) \u00b7 upstream d337b736", + "model": "gpt-5.6-sol", + "effort": "high", + "exit_code": 0, + "entry": "injected", + "worklist": [], + "spawns": [], + "files_read": [ + "/private/tmp/sync-rerun/hermes/tally-fizkc2k0/w/tally/.agents/skills/interrogate/SKILL.md", + "/private/tmp/sync-rerun/hermes/tally-fizkc2k0/w/tally/.agents/skills/poteto-mode/SKILL.md" + ], + "final_reply": "Use /poteto-mode. It selects the Code review playbook for a branch and can call /interrogate when a multi-model adversarial review is warranted.\n\n /poteto-mode Review this branch against main. Report blocking issues with file and line references, and verify each finding. Do not modify code.\n\nUse /interrogate directly only when you specifically want several models to challenge the diff without applying fixes.\n\nSources:\nhttps://github.com/mdsmithaustin/pstack/blob/main/poteto-mode/SKILL.md\nhttps://github.com/mdsmithaustin/pstack/blob/main/interrogate/SKILL.md", + "events": [] +} \ No newline at end of file