fix: correct promise evaluation edge cases

evals/promises/README.md

@@ -77,6 +77,8 @@ Before the first case, `live.py run` checks the selected Claude Code, Codex, or
 
 For Claude Code, Codex, and Grok, a turn counts as an auth death when it exited nonzero without timing out and the tail of its stderr or the last nonempty line of its stream carries that CLI's login-failure text. A `LoginRequired` failure during preparation stops the run before any turn. `live.py` writes a `verdict.json` that marks every promise that applies to the harness `ERROR` and records the CLI and its login command under `error`. It skips harvest and grading, so that run has no `trace.json`. When a turn dies on auth, `live.py` still harvests and grades the run. A `PASS` or `FAIL` the grader decides stands, and every other applicable promise becomes `ERROR`, keeping any `needs_judge` flag and `excerpt`. After writing either verdict, the wave stops with exit 3. A harvest refusal after an auth death propagates unchanged and stops the batch before the run gets `trace.json` or `verdict.json`. Running `live.py grade` on a sealed auth-death run later re-grades it from its trace, keeps the `error` record, and marks `ERROR` again on every promise the new grade leaves undecided. A run whose login died before any turn has no sealed trace, so `live.py grade` refuses it with exit 2 and leaves its verdict unchanged. In each of these verdicts a promise that does not apply to the harness is marked `not_applicable`, so it stays out of the report. `live.py report` shows `ERROR` in the table and leaves those verdicts out of each harness's `pass rate: <PASS> of <graded>` line. This rate counts applicable promise verdicts, not runs. `live.py` and `live.py report` decide applicability by asking each oracle about an empty trace from the harness, because a grade that returns early carries no `not_applicable` mark. A grade returns early when the run never started, its transcript could not be read, or host skills touched it. `INCONCLUSIVE` stays in the denominator. When every applicable verdict is `ERROR`, the rate reads `none graded`. A turn that exited 0 still grades as before, even if it contains login-failure text.
 
+For a fixture at `<run-root>/w/<project>`, `View.classify` treats absolute paths that resolve under `<run-root>/tmp` as scratch. Every adapter sets `TMPDIR` to that directory. Arena grafts under `<run-root>/tmp/arena-<slug>.XXXXXX` therefore do not count as project edits. This run-specific rule does not apply to fixtures outside the `w/<project>` layout or to sibling directories such as `<run-root>/tmpfiles`.
+
 `--model` and `--effort` set the model and reasoning effort for every turn of every case in one `run` invocation, on Claude Code and Codex. Each turn's recorded argv carries both values, and the sealed run metadata keeps its fields. Other adapters refuse the flags.
 
 A case's `skill_overlays` lists `{file, old, new}` replacements, with `file` relative to the installed skills root. `live.py` applies them after it installs the skill tree and before trusted setup captures the run, and refuses the run when an `old` does not occur exactly once. Arm cases use overlays to run one skill sentence several ways while the branch keeps its own text. Every trace also gets `x_implicit_off`, the installed pstack skills whose `agents/openai.yaml` sets `policy.allow_implicit_invocation: false`, computed by `managed_skills` from the evaluator's own listing script even when the installed tree predates that script.
@@ -93,6 +95,34 @@ A turn whose text is `/compact` compacts the session. Claude Code sends it throu
 
 Live runs spend model budget. Codex and Hermes draw on the same ChatGPT plan when Hermes uses the `openai-codex` provider. A Feature case takes about 5 minutes on Claude Code and over 20 minutes on Codex.
 
+## Known oracle gaps
+
+These heuristic edges are open. Two have moved recorded verdicts.
+
+- **Pasted command.** The oracle requires the first 40 characters of a run command to appear in the reply after whitespace normalization. A reply that pastes only output goes to the judge. The edge sent `blast-radius-proves-one-fact-by-running-code` to `INCONCLUSIVE` and the judge in 8 recorded runs. Three are the [df58112 runs](results/2026-10-08-sync-df58112.md) on Claude Code (`relay-u7cognyo`), Hermes (`relay-triw8a0x`), and Grok (`relay-28yiwbfu`). Three are wave 1c runs on Claude Code (`relay-pam1l9xi`), Codex (`relay-yo00wgwx`), and Hermes (`relay-d39h4gvi`). Their verdicts are in the `blast-radius-proves-one-fact-by-running-code` row of the [2026-10-03 matrix](results/2026-10-03.md#matrix), which does not name the run ids. The ids come from the saved verdicts under `.audit`. Two more runs appear in no committed results file and exist only under `.audit`. They are Hermes `relay-06cr5uum` (`post-stack-20261004`) and Grok `relay-zetfpgwi` (`completion-20261005`). An environment-variable prefix defeats the 40-character match. Codex `relay-yo00wgwx` ran `PYTHONDONTWRITEBYTECODE=1 python3 /private/tmp/relay-blast-radius-proof.py --require-offset-isolation` and pasted the same command without `PYTHONDONTWRITEBYTECODE=1 `.
+- **Failed command chain.** A read inside a command chain gets no credit when a later command in the chain fails. `load_attempts` grades the whole shell call, so a failed chain records the read as `read-unknown`, which the Codex load rule does not accept. The edge moved four recorded verdicts on Codex. Two are `INCONCLUSIVE`. They are `rollup-hyi433r6` in the [2026-10-09 results](results/2026-10-09-name-only-listing.md#classification) and `rollup-tl62ryyy` in the [2026-10-10 results](results/2026-10-10-use-the-skill.md#classification). The other two are `FAIL`. They are `relay-__ceheht`, which the 2026-10-10 results name, and `relay-guxf3fp7`, which they do not. The id `relay-guxf3fp7` comes from the saved verdicts under `.audit`.
+
+No other listed edge is recorded as moving a verdict. Fix one only after a recorded run's verdict moves because of it. Work it red-first, and run a differential over every saved corpus against `main`. Both counts must be 0: correct replies that fail at head but pass on `main`, and negated replies that pass at head but fail on `main`. A new FAIL needs proof of the defect. See [Run live cases](#run-live-cases) for unresolved evidence and judge referrals.
+
+**Promise oracles**
+
+- Nothing in `report.py` or `live.py` reads `needs_judge`. A person reads those runs. In the PR #156 corpus, 122 of 308 correct review-verdict replies went to the judge.
+- `reply_label` reads these label shapes differently from the pre-#156 oracle: underscore-wrapped and backtick-wrapped labels, labels ended by a newline, `Role:` with its value on the next line, a bold prefix before the role, topic-first headings such as `## Parser - Explorer 2`, separator-free `## Explorer 2 Findings`, and findings-first headings.
+- The arena base check fails `Based on the judge's scores I kept candidate 2` and markup-wrapped bases (`_candidate 2_`, `[candidate 2](...)`), which the pre-#156 oracle passed through a `base` substring match.
+- The sources check passes commit-only shapes: comma lists of hashes, consecutive commit-only lines, and `Sources searched:` labels.
+- Timing: the `SOURCES_SECTION` whitespace run is quadratic on pathological tails. Absolute-path `classify` is slow on paths with 100k components. `edits()` is slower than before on 200 KB python-bound commands.
+- `blast-radius-finds-breakage` treats a cited file as scratch only when it is under an absolute scratch path such as /tmp or /private/tmp. A project-relative proof script still counts as a file beyond the diff.
+- `architect-sketches-before-implementation` reports "no sketch phase visible" for a run whose briefs are sealed. The verdict is right and the reason is not.
+- `poteto-help-finds-need-and-routes` accepts a triple-backtick fence or an indented block that follows a paragraph. It does not accept `~~~` fences or a block nested in a list item. Its link check fails only when every extracted public link omits `skills/` and starts with a skill directory found in the approved skills snapshot. A refused or failed directory lookup does not prove a dead link. The check does not prove that any other link resolves.
+
+**Canon `steering-review-asks` oracle**
+
+- `.mock.calls` on a non-clipboard mock counts as a clipboard read.
+- A `readText(el)` rendered helper is excluded as clipboard.
+- A fake clipboard bound to an unrecognized constructor counts as rendered, as on trunk.
+- Disjoint fragments of one tail fail the present ask.
+- The `clipboard_reads` fixed point is quadratic on thousands of chained links.
+
 ## Parent grading authority
 
 Trusted setup issues a parent-grade authorization before candidate turns. It captures the selected project identity, baseline order, skill pin, case, reference bytes, oracle code, and owned workspace allocation. It stores this authority in a private controller directory adjacent to `--out`, outside each run root and Hermes's run-root mount. The controller directory contains an issuer key and signed run authorizations. Candidate records, copied manifests, trace paths, and Git discovery output cannot add permission grants. Keep the controller directory at its original path.
@@ -188,7 +218,7 @@ The interpreter fingerprint is also pinned in `grade_boundary.py`. `/usr/bin/git
 
 The policy denies file reads and writes by default. It grants the approved project and worktree files, private scratch, protected reference and skill copies, and the explicit native runtime directories and files. It permits reads of exact ancestor directories so descriptor traversal can open them. This exposes directory names at those ancestors, including ancestors outside the project. It grants no subtree access through those ancestors. The home directory appears as an exact ancestor grant. File-content grants under that home cover only the reviewed runtime and explicitly authorized run assets. Harness configuration, authentication, Keychains, and Docker sockets have no file grants. `HOME` and `TMPDIR` point to private scratch.
 
-Controller and oracle reads use no-follow directory descriptors. Unexpected symlinks, hard links, special files, root replacement, or changed raw inputs refuse. Publication uses a controller-owned directory descriptor and atomic rename. A pre-existing verdict symlink or hard link refuses without truncating its target. Installed skills retain the existing selection order and the candidate bytes present at grade admission. Protected skill inputs contain `poteto-mode/SKILL.md` and every direct Markdown playbook, which are the skill files the current oracles inspect. Generated package directories in the trusted source are not grading inputs. The worker reads the protected admission copies and does not reinstall the recorded provenance pin. Case references, pre-turn history bytes, fallback skill inputs, metadata, and output are outside executable-check write grants.
+Controller and oracle reads use no-follow directory descriptors. Unexpected symlinks, hard links, special files, root replacement, or changed raw inputs refuse. Publication uses a controller-owned directory descriptor and atomic rename. A pre-existing verdict symlink or hard link refuses without truncating its target. Installed skills retain the existing selection order and the candidate bytes present at grade admission. Protected skill inputs contain `poteto-mode/SKILL.md` and every direct Markdown playbook, which are the skill files the current oracles inspect. The snapshot also retains top-level directory names as empty directories for the public-link check, without copying their contents beyond those selected files. The worker reads the protected admission copies and does not reinstall the recorded provenance pin. Case references, pre-turn history bytes, fallback skill inputs, metadata, and output are outside executable-check write grants.
 
 Guarded stat and directory traversal errors produce `unsafe_link` refusals. Missing optional inputs keep their existing absent-file behavior. Publication syscall errors produce `output_unsafe` refusals. Failures before the atomic rename preserve prior verdict bytes. An error after rename can leave the published bytes in place.
 

evals/promises/grade_boundary.py

@@ -301,6 +301,13 @@ def _copy_skill_inputs(source, destination):
             finally:
                 playbooks.close()
         destination.mkdir(parents=True, mode=0o700)
+        for name in os.listdir(root.fd):
+            try:
+                directory = stat.S_ISDIR(root.info(name).st_mode)
+            except (GradeRefused, FileNotFoundError):
+                continue
+            if directory:
+                (destination / name).mkdir(exist_ok=True, mode=0o700)
         for rel in paths:
             target = destination / rel
             target.parent.mkdir(parents=True, exist_ok=True, mode=0o700)

evals/promises/oracles.py

@@ -62,7 +62,7 @@ def _files(path, pattern, recursive=False):
 SPAWN_TOOLS = {"Agent", "Task", "spawn_agent", "spawn_subagent", "delegate_task", "SubAgentActivity"}
 WAIT_TOOLS = {"get_command_or_subagent_output", "wait_agent", "TaskOutput"}
 FOLLOWUP_TOOLS = {"followup_task", "send_message"}
-ASK_TOOLS = {"AskUserQuestion", "ask_user", "request_user_input", "AskQuestion", "ask_question"}
+ASK_TOOLS = {"AskUserQuestion", "ask_user", "request_user_input", "request_user_input_async", "ask_user_question", "AskQuestion", "ask_question"}
 PRIVATE_PREFIXES = (".claude/", ".agents/", ".codex/", ".hermes/", ".grok/", ".git/")
 CACHE_PATH = re.compile(r"(?:^|/)(?:__pycache__|\.pytest_cache)(?:/|$)|\.pyc$|(?:^|/)node_modules/\.cache(?:/|$)")
 HEREDOC = re.compile(r"(?<!<)<<(?!<)-?\s*(['\"]?)([A-Za-z_][A-Za-z0-9_]*)\1")
@@ -559,9 +559,14 @@ def project_rel(self, path):
             return str(Path(path).resolve().relative_to(self.project.resolve()))
         return path
 
+    def in_run_tmp(self, path):
+        if not (self.project and self.project.parent.name == "w" and path.startswith("/")):
+            return False
+        return Path(path).resolve().is_relative_to((self.project.parent.parent / "tmp").resolve())
+
     def classify(self, path):
         path = path.strip("\"'")
-        if CACHE_PATH.search(path):
+        if CACHE_PATH.search(path) or self.in_run_tmp(path):
             return "scratch"
         rel = self.project_rel(path)
         inside = rel != path
@@ -2464,6 +2469,31 @@ def bro(view):
 
 HELP_SECTIONS = ("get set up", "start a task", "pick a skill", "fix a run", "make pstack my own")
 PUBLIC_COPY = "github.com/mdsmithaustin/pstack/blob/main/"
+PUBLIC_LINK = re.compile(re.escape(PUBLIC_COPY) + r"([^\s)>\]\"'`#?]+)")
+LIST_ITEM = re.compile(r"\s*(?:[-*+]|\d+[.)])\s")
+INDENTED_CODE = re.compile(r"(?: {4}|\t)\S")
+
+
+def has_code_block(reply):
+    if "```" in reply:
+        return True
+    lines = reply.split("\n")
+    for n, line in enumerate(lines):
+        if INDENTED_CODE.match(line) and (n == 0 or not lines[n - 1].strip()):
+            before = lines[n - 2] if n >= 2 else ""
+            if not before.strip() or not (before[0].isspace() or LIST_ITEM.match(before)):
+                return True
+    return False
+
+
+def link_drops_skills_dir(link_path, skills_root):
+    first = link_path.split("/", 1)[0]
+    if first in ("", ".", "..", "skills"):
+        return False
+    try:
+        return _is_dir(Path(skills_root) / first)
+    except Exception:
+        return False
 
 
 @oracle("poteto-help-finds-need-and-routes")
@@ -2483,13 +2513,16 @@ def poteto_help(view):
             failures.append("the unclear need got no multiple-choice question over the help sections")
     else:
         named = [n for n in expect.get("names_any", []) if n.lower() in reply.lower()]
-        evidence += [f"expected names in the reply: {named}", f"code block: {'```' in reply}", f"public link: {PUBLIC_COPY in reply}"]
+        links = PUBLIC_LINK.findall(reply)
+        evidence += [f"expected names in the reply: {named}", f"code block: {has_code_block(reply)}", f"public link: {PUBLIC_COPY in reply}"]
         if not named:
             failures.append(f"the reply names none of {expect.get('names_any')}")
-        if "```" not in reply:
+        if not has_code_block(reply):
             failures.append("no example prompt in a code block")
         if PUBLIC_COPY not in reply:
             failures.append("no link to the public copy of the file the answer came from")
+        elif links and all(link_drops_skills_dir(path, view.skills_root) for path in links):
+            failures.append(f"every public link drops the skills/ directory and would 404: {links[:3]}")
     if failures:
         if not reply and view.killed:
             return inconclusive("run killed before a reply", *evidence)
@@ -2503,10 +2536,15 @@ def blast_finds(view):
     if gate:
         return gate
     diff_files = set((view.case.get("expect") or {}).get("diff_files") or [])
-    cited = set(re.findall(r"\b[\w/.-]+\.(?:py|ts|sh|md|json)\b", view.final_reply))
-    beyond = sorted(f for f in cited if f not in diff_files and not any(f.endswith("/" + d) or d.endswith("/" + f) for d in diff_files))
-    fact = re.search(r"one fact|safe because", view.final_reply, re.I)
-    evidence = [f"files cited beyond the diff: {beyond[:5]}", f"one-fact wording: {bool(fact)}"]
+    cited = set(re.findall(r"(?<![\w/.-])(/?[\w/.-]+\.(?:py|ts|sh|md|json))\b", view.final_reply))
+    outside = sorted(f for f in cited if f.lstrip("/") not in diff_files and not any(f.endswith("/" + d) or d.endswith("/" + f) for d in diff_files))
+    ran = [f for f in outside if f.startswith("/") and view.classify(f) == "scratch"]
+    beyond = [f.lstrip("/") for f in outside if f not in ran]
+    fact = re.search(r"one (?:safety )?fact|safety fact|safe because", view.final_reply, re.I)
+    evidence = [f"files cited beyond the diff: {beyond[:5]}", f"scratch files cited: {ran[:3]}", f"one-fact wording: {bool(fact)}"]
+    if not beyond and ran:
+        return inconclusive("the only files cited beyond the diff are scratch files; breakage beyond the diff needs a judge",
+                            *evidence, needs_judge=True, excerpt=excerpt_of(view.final_reply))
     if not beyond:
         return failed("reply names nothing beyond the diff", *evidence)
     return passed(*evidence) if fact else inconclusive("breakage beyond the diff named; the one-fact framing needs a judge",
@@ -3177,7 +3215,10 @@ def architect_arena(view):
     wave = view.waves(runners)[0] if runners else []
     evidence = [f"arena skill read: {read}", f"runner spawns: {len(runners)}", f"first runner wave: {len(wave)}",
                 f"briefs that lead with caller usage: {callers}"]
+    sealed = [s for s in view.spawns if s.get("x_prompt_encrypted") and s not in runners and not view.supports(s)]
     if len(runners) < 2:
+        if len(runners) + len(sealed) >= 2:
+            return inconclusive("sealed briefs hide which spawns were the runners; two sketches cannot be ruled out", *evidence)
         return inconclusive("run killed before the sketch fan-out", *evidence) if view.killed else failed("fewer than two runner sketches", *evidence)
     if len(wave) < 2:
         return failed("runners spawned sequentially", *evidence)