From 461b24db301e49428b6a5909c93ddec504d34865 Mon Sep 17 00:00:00 2001 From: mdsmithaustin <15272014+mdsmithaustin@users.noreply.github.com> Date: Mon, 12 Oct 2026 13:19:13 -0600 Subject: [PATCH 01/12] evals: preflight CLI logins before a live wave and grade auth deaths ERROR Each adapter gets a free login check that stops 'live.py run' before any case spends a token and names the CLI and its login command. A run whose turn dies on a login failure gets an ERROR verdict for every promise, skips grading, stops the wave, and drops out of the pass-rate denominator in 'live.py report'. Closes #168 --- evals/promises/README.md | 6 +- evals/promises/harnesses/claude_code.py | 19 +++ evals/promises/harnesses/codex.py | 21 ++- evals/promises/harnesses/grok.py | 25 +++- evals/promises/harnesses/login.py | 69 +++++++++ evals/promises/live.py | 67 ++++++--- evals/promises/report.py | 18 +++ evals/promises/test_live.py | 187 +++++++++++++++++++++++- evals/promises/test_report.py | 25 ++++ 9 files changed, 408 insertions(+), 29 deletions(-) create mode 100644 evals/promises/harnesses/login.py diff --git a/evals/promises/README.md b/evals/promises/README.md index 83622542..8de7edbb 100644 --- a/evals/promises/README.md +++ b/evals/promises/README.md @@ -71,7 +71,11 @@ python3 evals/promises/live.py run --harness --c python3 evals/promises/live.py report --upstream-ref upstream/main ``` -Each run makes a fresh git repository from `fixtures//` and `histories//`. Each history step is committed in order, except a step with `"commit": false`, which stays as uncommitted work in the tree. A step with `"branch"` switches to that branch first, creating it at the current commit when it does not exist, so later steps land there and `main` keeps the fixture. It installs `skills/` from `--skills-at` (default `HEAD`) where the harness discovers project skills, and runs the case's turns headless. Each run directory gets `run.json`, `trace.json`, and `verdict.json`. Every adapter writes the same trace schema, with events, files read, worklist snapshots, spawns, and the final reply. A tool event carries its harness call id, so each result pairs with its own call. `oracles.py` grades each promise as `PASS`, `FAIL`, or `INCONCLUSIVE`. A verdict is `INCONCLUSIVE` when the trace cannot show the behavior, or when the harness reports `x_host_skill_hits`, which means the user's own skills touched the run. Some oracles also return `INCONCLUSIVE` when only a judge can decide, such as whether a reply's wording reports a review `pass`. That verdict carries `needs_judge: true` and an `excerpt` of the graded reply in `verdict.json`. In a multi-turn case the graded reply can come from an earlier turn than `final_reply`, and the verdict's evidence names that turn. When the excerpt is not enough, read that turn's text events in `trace.json` before you tally the run. +Each run makes a fresh git repository from `fixtures//` and `histories//`. Each history step is committed in order, except a step with `"commit": false`, which stays as uncommitted work in the tree. A step with `"branch"` switches to that branch first, creating it at the current commit when it does not exist, so later steps land there and `main` keeps the fixture. It installs `skills/` from `--skills-at` (default `HEAD`) where the harness discovers project skills, and runs the case's turns headless. Each run directory gets `run.json`, `trace.json`, and `verdict.json`. Every adapter writes the same trace schema, with events, files read, worklist snapshots, spawns, and the final reply. A tool event carries its harness call id, so each result pairs with its own call. `oracles.py` grades each promise as `PASS`, `FAIL`, or `INCONCLUSIVE`, and `live.py` alone adds `ERROR` for a dead login. A verdict is `INCONCLUSIVE` when the trace cannot show the behavior, or when the harness reports `x_host_skill_hits`, which means the user's own skills touched the run. Some oracles also return `INCONCLUSIVE` when only a judge can decide, such as whether a reply's wording reports a review `pass`. That verdict carries `needs_judge: true` and an `excerpt` of the graded reply in `verdict.json`. In a multi-turn case the graded reply can come from an earlier turn than `final_reply`, and the verdict's evidence names that turn. When the excerpt is not enough, read that turn's text events in `trace.json` before you tally the run. + +Before the first case, `live.py run` asks each CLI whether its login is usable, at no model cost. A failed check stops the wave with exit 3 before any case runs, and the message names the CLI and its login command. Claude Code runs `claude auth status`. Codex needs the `auth.json` file and a passing `codex login status`. Grok refuses a host token that expires within five minutes, because the run's read-only provider never refreshes it. Claude Code and Codex refresh their own access tokens when they run, so their checks do not read token expiry. Hermes has no free status command, and its preparation step still refuses a missing `auth.json`. + +A run that dies on auth after the wave starts is graded `ERROR`, not `FAIL` or `INCONCLUSIVE`. A turn counts as an auth death when it exited nonzero without timing out and the tail of its stderr or the last line of its stream carries that CLI's login-failure text. `live.py` writes a `verdict.json` that marks every promise of the case `ERROR` and records the CLI and its login command under `error`. It skips harvest and grading, since a dead login says nothing about the skill, and it stops the wave after that run. `live.py report` shows `ERROR` in the table and leaves those verdicts out of each harness's `pass rate: of ` line. `INCONCLUSIVE` stays in the denominator. A run that died on auth but exited 0 still grades as before. `--model` and `--effort` set the model and reasoning effort for every turn of every case in one `run` invocation, on Claude Code and Codex. Each turn's recorded argv carries both values, and the sealed run metadata keeps its fields. Other adapters refuse the flags. diff --git a/evals/promises/harnesses/claude_code.py b/evals/promises/harnesses/claude_code.py index f29702db..b8d9b730 100644 --- a/evals/promises/harnesses/claude_code.py +++ b/evals/promises/harnesses/claude_code.py @@ -15,11 +15,15 @@ from pathlib import Path import live +from harnesses.login import Login, LoginRequired, check_status SKILLS_DIR = ".claude/skills" PRIVATE_DIRS = (".claude/",) SHARES_HOST_TMP = False +LOGIN = Login("claude", "claude auth login", re.compile( + r"Please run /login|Not logged in|Login expired|OAuth token (?:revoked|has expired)|Invalid API key|" + r"Failed to authenticate|Authentication required", re.I)) DEFAULT_EFFORT = "high" SETTING_SOURCES = "user,project" LISTING_SCRIPT = Path("setup-pstack/scripts/skill-listing.py") @@ -196,6 +200,21 @@ def store(run): return config_dir(run) / "projects" +def preflight(): + """Ask the CLI for its login status. The CLI refreshes an expired access token itself when it runs, so this fails only + when it reports no login.""" + binary = CLAUDE_BINARY if CLAUDE_BINARY.is_file() else shutil.which("claude") + if not binary: + raise LoginRequired(LOGIN, "no claude binary to ask") + out = check_status(LOGIN, [binary, "auth", "status"]) + try: + logged_in = json.loads(out).get("loggedIn") + except ValueError: + logged_in = None + if logged_in is not True: + raise LoginRequired(LOGIN, f"`claude auth status` reports loggedIn {logged_in!r}") + + def prepare(run): state = _bind_paths(run, _host_runtime()) script = state.paths.project / SKILLS_DIR / "pstack-harness/scripts/subagents.py" diff --git a/evals/promises/harnesses/codex.py b/evals/promises/harnesses/codex.py index a00811cb..b68bc7ab 100644 --- a/evals/promises/harnesses/codex.py +++ b/evals/promises/harnesses/codex.py @@ -9,6 +9,7 @@ import live from grade_boundary import GradeRefused, read_file, write_file +from harnesses.login import Login, LoginRequired, check_status SKILLS_DIR = ".agents/skills" PRIVATE_DIRS = [".agents/", ".codex/"] @@ -19,6 +20,9 @@ EFFORT = "high" COMPACT = "/compact" COMPACT_TOKEN_LIMIT = 2000 +LOGIN = Login("codex", "codex login", re.compile( + r"access token could not be refreshed|refresh token was (?:already used|rejected)|log out and sign in again|" + r"authentication session could not be refreshed|Not logged in", re.I)) ENV_KEPT = ("PATH", "LANG", "LC_ALL", "LC_CTYPE", "LC_MESSAGES", "TERM", "USER", "LOGNAME", "SHELL", "TZ", "SSL_CERT_FILE", "SSL_CERT_DIR", "HTTP_PROXY", "HTTPS_PROXY", "NO_PROXY", "http_proxy", "https_proxy", "no_proxy") @@ -57,13 +61,26 @@ def resolve_binary(env): raise RuntimeError(f"no working codex binary; rejected: {json.dumps(rejected)}") +def host_auth(): + return Path(os.environ.get("CODEX_HOME") or Path.home() / ".codex") / "auth.json" + + +def preflight(): + """Ask the CLI whether it holds a login. The CLI refreshes an expired access token itself when it runs, so the token's + own expiry is not checked here.""" + auth = host_auth() + if not auth.is_file(): + raise LoginRequired(LOGIN, f"no login at {auth}") + check_status(LOGIN, [resolve_binary(dict(os.environ))["path"], "login", "status"]) + + def prepare(run): root = run.root for name in ("home", "codex-home", "tmp", "transcripts"): (root / name).mkdir(exist_ok=True) - auth = Path(os.environ.get("CODEX_HOME") or Path.home() / ".codex") / "auth.json" + auth = host_auth() if not auth.is_file(): - raise RuntimeError(f"no codex login at {auth}") + raise LoginRequired(LOGIN, f"no login at {auth}") link = root / "codex-home" / "auth.json" if not link.is_symlink(): link.symlink_to(auth) diff --git a/evals/promises/harnesses/grok.py b/evals/promises/harnesses/grok.py index 448af118..00db3698 100644 --- a/evals/promises/harnesses/grok.py +++ b/evals/promises/harnesses/grok.py @@ -11,6 +11,7 @@ import live from grade_boundary import GradeRefused, copy_file +from harnesses.login import Login, LoginRequired SKILLS_DIR = ".agents/skills" PRIVATE_DIRS = [".agents/"] @@ -132,6 +133,21 @@ def host_token_expiry(auth): return datetime.datetime.fromisoformat(entry["expires_at"].replace("Z", "+00:00")) if entry else None +LOGIN = Login("grok", "grok login", re.compile( + r"Run `grok login`|auth token is invalid or expired|Not authenticated|Authentication required|no xAI credential", re.I)) +TOKEN_MARGIN = datetime.timedelta(minutes=5) + + +def preflight(): + """Refuse a missing or expired host token. The run's read-only provider never refreshes it, so only `grok login` on the host does.""" + auth = host_grok_home() / "auth.json" + if not auth.is_file(): + raise LoginRequired(LOGIN, f"no login at {auth}") + expiry = host_token_expiry(auth) + if expiry is not None and expiry <= datetime.datetime.now(datetime.timezone.utc) + TOKEN_MARGIN: + raise LoginRequired(LOGIN, f"the host token expired at {expiry:%Y-%m-%dT%H:%MZ}, and the run's read-only provider never refreshes it") + + def prepare(run): root = run.root for name in ("home", "grok-home", "tmp", "transcripts"): @@ -139,13 +155,8 @@ def prepare(run): for name in HOME_DIRS: (home(run) / name).mkdir(exist_ok=True) (home(run) / ".gitconfig").write_text(GITCONFIG) + preflight() auth = host_grok_home() / "auth.json" - if not auth.is_file(): - raise RuntimeError(f"no grok login at {auth}") - expiry = host_token_expiry(auth) - if expiry is not None and expiry <= datetime.datetime.now(datetime.timezone.utc) + datetime.timedelta(minutes=5): - raise RuntimeError(f"the host Grok token expired at {expiry:%Y-%m-%dT%H:%MZ}; run `grok login` on the host, " - "since the run's read-only provider never refreshes it") provider = root / "auth-provider.py" provider.write_text(AUTH_PROVIDER.format(python=shutil.which("python3") or "/usr/bin/python3", auth=str(auth))) provider.chmod(0o755) @@ -159,7 +170,7 @@ def prepare(run): text=True, timeout=120) (root / "login.log").write_text(login.stdout + login.stderr) if login.returncode != 0: - raise RuntimeError(f"grok login failed under the run's own home: {(login.stdout + login.stderr)[-300:]}") + raise LoginRequired(LOGIN, f"`grok login` failed under the run's own home: {(login.stdout + login.stderr)[-300:]}") inspected = subprocess.run([launch["path"], "inspect", "--json"], cwd=run.project, env=env, capture_output=True, text=True, timeout=120, check=True) (root / "inspect.json").write_text(inspected.stdout) diff --git a/evals/promises/harnesses/login.py b/evals/promises/harnesses/login.py new file mode 100644 index 00000000..14ca356c --- /dev/null +++ b/evals/promises/harnesses/login.py @@ -0,0 +1,69 @@ +"""What a live wave needs from each CLI's login, shared by the adapters and live.py.""" + +import re +import subprocess +from dataclasses import dataclass +from pathlib import Path + +from grade_boundary import read_file + +ERROR = "ERROR" +STATUS_TIMEOUT_S = 60 +DEATH_WINDOW = 4000 + + +@dataclass(frozen=True) +class Login: + cli: str + relogin: str + death: re.Pattern + + +class LoginRequired(RuntimeError): + def __init__(self, login, detail): + super().__init__(f"{login.cli} login is not usable: {detail}. Run `{login.relogin}`, then start the wave again") + self.login = login + self.detail = detail + + +def check_status(login, argv, env=None): + """Run the CLI's own login status command. It spends no model tokens.""" + try: + done = subprocess.run([str(a) for a in argv], env=env, capture_output=True, text=True, timeout=STATUS_TIMEOUT_S, + stdin=subprocess.DEVNULL) + except (OSError, subprocess.TimeoutExpired) as error: + raise LoginRequired(login, f"`{' '.join(map(str, argv))}` did not finish: {error!r}") from error + if done.returncode: + raise LoginRequired(login, f"`{' '.join(map(str, argv))}` exited {done.returncode}: {(done.stdout + done.stderr).strip()[-300:]}") + return done.stdout + + +def auth_death(login, record): + """The auth-failure text that ended a turn, or None. Only a turn that exited nonzero counts, and only its stderr tail + and last stream line, so a reply that quotes a login message in a run that finished is not read as a dead login.""" + if not record.get("exit_code") or record.get("timed_out"): + return None + tail = [] + if record.get("stderr"): + tail.append(_read(record["stderr"])[-DEATH_WINDOW:]) + if record.get("stream"): + lines = [line for line in _read(record["stream"]).splitlines() if line.strip()] + tail.append(lines[-1][-DEATH_WINDOW:] if lines else "") + found = login.death.search("\n".join(tail)) + return found.group(0) if found else None + + +def _read(path): + try: + return read_file(Path(path)).decode(errors="replace") + except FileNotFoundError: + return "" + + +def error_verdict(case, harness, skills_at, login, detail): + """The verdict for a run a dead login ended. Every promise of the case is ERROR, so no skill result is claimed and + report.py leaves the run out of every pass rate.""" + reason = f"{login.cli} login died: {detail}" + return {"case": case["id"], "harness": harness, "skills_at": skills_at, + "error": {"cli": login.cli, "relogin": login.relogin, "detail": detail}, + "promises": {pid: {"verdict": ERROR, "failures": [reason], "evidence": []} for pid in case["promises"]}} diff --git a/evals/promises/live.py b/evals/promises/live.py index 23295d1c..d8dd17f1 100644 --- a/evals/promises/live.py +++ b/evals/promises/live.py @@ -17,7 +17,8 @@ from dataclasses import dataclass, field from pathlib import Path -from grade_boundary import create_file, grade, GradeRefused, read_file, _before_turns, _lookup, _seal, _write_record +from grade_boundary import create_file, grade, GradeRefused, read_file, _before_turns, _json, _lookup, _seal, _write_record +from harnesses.login import LoginRequired, auth_death, error_verdict HERE = Path(__file__).resolve().parent ROOT = HERE.parents[1] @@ -64,6 +65,13 @@ def adapter(name): return importlib.import_module(HARNESSES[name]) +def preflight(harness): + """Refuse a wave whose login is gone before any case spends a token. An adapter with no `preflight` has no free check.""" + check = getattr(adapter(harness), "preflight", None) + if check: + check() + + def load_case(case_id): case = json.loads((CASES / case_id / "case.json").read_text(encoding="utf-8")) case["id"] = case_id @@ -241,6 +249,28 @@ def fixture_lock(fixture): return handle +def drive(module, run, authorization): + """Prepare the run and play its turns. Returns (login, detail) when a login died before or during a turn, else None.""" + try: + module.prepare(run) + except LoginRequired as expired: + return expired.login, expired.detail + login = getattr(module, "LOGIN", None) + lock = fixture_lock(run.case["fixture"]) if module.SHARES_HOST_TMP and run.case.get("tmp_lock", True) else None + try: + for i, text in enumerate(run.case["turns"]): + record = module.turn(run, text, i) + run.turns.append(record) + _write_record(authorization, meta(run)) + dead = login and auth_death(login, record) + if dead: + return login, f"turn {i} exited {record['exit_code']} with {dead!r}" + finally: + if lock: + lock.close() + return None + + def run_case(harness, case_id, skills_at, out, index, hermes_retain_out=None, model=None, effort=None): if hermes_retain_out is not None and harness != "hermes": raise ValueError("--hermes-retain-out requires --harness hermes") @@ -268,19 +298,15 @@ def run_case(harness, case_id, skills_at, out, index, hermes_retain_out=None, mo authorization = _before_turns(run) pair = None try: - module.prepare(run) - lock = fixture_lock(case["fixture"]) if module.SHARES_HOST_TMP and case.get("tmp_lock", True) else None - try: - for i, text in enumerate(case["turns"]): - record = module.turn(run, text, i) - run.turns.append(record) - _write_record(authorization, meta(run)) - finally: - if lock: - lock.close() - trace = {**module.harvest(run), "x_implicit_off": run.implicit_off} - _seal(authorization, meta(run), trace) - verdict = grade(authorization) + died = drive(module, run, authorization) + if died: + verdict = error_verdict(case, harness, commit, *died) + with create_file(root / "verdict.json") as stream: + stream.write(_json(verdict)) + else: + trace = {**module.harvest(run), "x_implicit_off": run.implicit_off} + _seal(authorization, meta(run), trace) + verdict = grade(authorization) except BaseException as original: owner = getattr(run, "_hermes_evidence", None) if owner is not None and retention is not None: @@ -305,6 +331,8 @@ def run_case(harness, case_id, skills_at, out, index, hermes_retain_out=None, mo print(json.dumps({"run": str(root), "authorization_id": authorization._id, "case": case_id, "harness": harness, "n": index, **({"hermes_pair": str(pair.root)} if pair is not None else {}), "verdicts": {p: v["verdict"] for p, v in verdict["promises"].items()}})) + if died: + raise LoginRequired(*died) return root @@ -336,9 +364,14 @@ def main(argv=None): if args.cmd == "run": out = Path(args.out) out.mkdir(parents=True, exist_ok=True) - for case_id in args.case: - for n in range(args.runs): - run_case(args.harness, case_id, args.skills_at, out, n, args.hermes_retain_out, args.model, args.effort) + try: + preflight(args.harness) + for case_id in args.case: + for n in range(args.runs): + run_case(args.harness, case_id, args.skills_at, out, n, args.hermes_retain_out, args.model, args.effort) + except LoginRequired as expired: + print(f"wave stopped: {expired}", file=sys.stderr) + return 3 return 0 if args.cmd == "grade": for run_id in args.run_ids: diff --git a/evals/promises/report.py b/evals/promises/report.py index 69ceb348..380484a1 100644 --- a/evals/promises/report.py +++ b/evals/promises/report.py @@ -3,6 +3,7 @@ from pathlib import Path import ledger +from harnesses.login import ERROR HARNESSES = ("claude-code", "codex", "hermes", "grok") @@ -41,6 +42,22 @@ def cell(results): return ", ".join(f"{v} {n}" for v, n in sorted(counts.items())) +def pass_rates(cells): + """One line per harness. An ERROR verdict is a run a dead login ended, so it leaves the denominator.""" + counts = collections.defaultdict(collections.Counter) + for (_, harness), results in cells.items(): + counts[harness].update(v for v, _ in results) + lines = [] + for harness in HARNESSES: + tally = counts.get(harness) + if not tally: + continue + graded = sum(n for v, n in tally.items() if v != ERROR) + rate = f"{tally['PASS']} of {graded}" if graded else "none graded" + lines.append(f"{harness} pass rate: {rate}" + (f" (ERROR {tally[ERROR]} excluded)" if tally[ERROR] else "")) + return lines + + def render(out, upstream=None): book = ledger.load() cells = collect(out) @@ -60,5 +77,6 @@ def render(out, upstream=None): lines.append("") lines.append(f"{len(pids)} promises, {sum(totals.values())} verdicts: " + ", ".join(f"{v} {n}" for v, n in sorted(totals.items()))) + lines += pass_rates(cells) lines += listing_states(out) return "\n".join(lines) diff --git a/evals/promises/test_live.py b/evals/promises/test_live.py index 39032146..2e8f5f3b 100644 --- a/evals/promises/test_live.py +++ b/evals/promises/test_live.py @@ -10,11 +10,12 @@ import unittest from pathlib import Path from unittest import mock -from contextlib import redirect_stdout +from contextlib import redirect_stderr, redirect_stdout from contextlib import nullcontext import live from grade_boundary import GradeRefused, _authorize_fixture +from harnesses.login import LoginRequired, auth_death CODEX_CHAT = """I'm using `poteto-mode`. I'll identify the command first. @@ -333,7 +334,8 @@ def test_model_and_effort_refuse_harnesses_without_an_override(self): live.run_case(harness, "principle-steer-run", "HEAD", Path("/unused"), 0, model="m") def test_the_cli_passes_model_and_effort_to_every_case(self): - with tempfile.TemporaryDirectory(prefix="pstack-overrides-") as tmp, mock.patch.object(live, "run_case") as run_case: + with tempfile.TemporaryDirectory(prefix="pstack-overrides-") as tmp, mock.patch.object(live, "run_case") as run_case, \ + mock.patch.object(live, "preflight"): live.main(["run", "--harness", "codex", "--case", "a", "--case", "b", "--model", "gpt-6-luna", "--effort", "xhigh", "--out", tmp]) self.assertEqual([c.args[1] for c in run_case.call_args_list], ["a", "b"]) self.assertEqual({c.args[-2:] for c in run_case.call_args_list}, {("gpt-6-luna", "xhigh")}) @@ -357,6 +359,187 @@ def test_codex_and_claude_turns_carry_the_override_in_argv(self): self.assertEqual(argv[-5:], ("--effort", "high", "--model", "claude-sonnet-5-5", "go")) +def fake_cli(directory, name, script): + path = Path(directory) / name + path.write_text("#!/bin/sh\n" + script) + path.chmod(0o755) + return path + + +class LoginPreflight(unittest.TestCase): + """A wave stops on a login the CLI reports gone, before any case spends a token.""" + + def setUp(self): + tmp = tempfile.TemporaryDirectory(prefix="pstack-preflight-") + self.addCleanup(tmp.cleanup) + self.tmp = Path(tmp.name).resolve() + + def wave(self, harness, *patches): + stderr = io.StringIO() + with mock.patch.object(live, "run_case") as run_case, redirect_stderr(stderr): + for patch in patches: + self.enterContext(patch) + code = live.main(["run", "--harness", harness, "--case", "a", "--case", "b", "--out", str(self.tmp / "out")]) + return code, run_case, stderr.getvalue() + + def claude(self, script): + from harnesses import claude_code + return mock.patch.object(claude_code, "CLAUDE_BINARY", fake_cli(self.tmp, "claude", script)) + + def codex(self, script, with_auth=True): + home = self.tmp / "codex-home" + home.mkdir(exist_ok=True) + if with_auth: + (home / "auth.json").write_text("{}") + binary = fake_cli(self.tmp, "codex", script) + return mock.patch.dict(os.environ, {"CODEX_HOME": str(home), "CODEX_BIN": str(binary)}) + + def grok(self, expires_at): + from harnesses import grok + home = self.tmp / "grok-home" + home.mkdir(exist_ok=True) + (home / "auth.json").write_text(json.dumps({"xai": {"key": "k", "expires_at": expires_at}})) + return mock.patch.object(grok, "host_grok_home", return_value=home) + + def assertStopped(self, outcome, cli, command): + code, run_case, stderr = outcome + self.assertEqual(code, 3) + run_case.assert_not_called() + self.assertEqual(list((self.tmp / "out").iterdir()), []) + self.assertIn(f"{cli} login is not usable", stderr) + self.assertIn(f"Run `{command}`", stderr) + + def test_claude_code_reporting_no_login_stops_the_wave(self): + outcome = self.wave("claude-code", self.claude('echo \'{"loggedIn": false}\'; exit 1\n')) + self.assertStopped(outcome, "claude", "claude auth login") + + def test_claude_code_exiting_zero_with_loggedin_false_still_stops_the_wave(self): + outcome = self.wave("claude-code", self.claude('echo \'{"loggedIn": false}\'\n')) + self.assertStopped(outcome, "claude", "claude auth login") + + def test_codex_login_status_failing_stops_the_wave(self): + script = 'case "$1" in --version) echo codex 1;; *) echo "Not logged in" >&2; exit 1;; esac\n' + self.assertStopped(self.wave("codex", self.codex(script)), "codex", "codex login") + + def test_codex_without_an_auth_file_stops_the_wave_even_when_the_cli_says_logged_in(self): + script = 'case "$1" in --version) echo codex 1;; *) echo "Logged in using ChatGPT";; esac\n' + self.assertStopped(self.wave("codex", self.codex(script, with_auth=False)), "codex", "codex login") + + def test_grok_with_an_expired_host_token_stops_the_wave(self): + self.assertStopped(self.wave("grok", self.grok("2020-01-01T00:00:00Z")), "grok", "grok login") + + def test_a_valid_login_lets_every_case_run(self): + script = 'case "$1" in --version) echo codex 1;; *) echo "Logged in using ChatGPT";; esac\n' + code, run_case, _ = self.wave("codex", self.codex(script)) + self.assertEqual((code, [c.args[1] for c in run_case.call_args_list]), (0, ["a", "b"])) + + def test_a_login_that_dies_mid_wave_stops_the_wave_after_its_error_verdict(self): + from harnesses import codex + died = LoginRequired(codex.LOGIN, "turn 0 exited 1") + stderr = io.StringIO() + with mock.patch.object(live, "preflight"), mock.patch.object(live, "run_case", side_effect=[died]) as run_case, redirect_stderr(stderr): + code = live.main(["run", "--harness", "codex", "--case", "a", "--case", "b", "--out", str(self.tmp / "out")]) + self.assertEqual((code, run_case.call_count), (3, 1)) + self.assertIn("Run `codex login`", stderr.getvalue()) + + +class AuthDeath(unittest.TestCase): + """A turn that ended on an auth failure is the harness failing, not the skill under test.""" + + def setUp(self): + tmp = tempfile.TemporaryDirectory(prefix="pstack-auth-death-") + self.addCleanup(tmp.cleanup) + self.tmp = Path(tmp.name).resolve() + + def record(self, stderr="", stream="", exit_code=1, timed_out=False): + (self.tmp / "err").write_text(stderr) + (self.tmp / "out").write_text(stream) + return {"exit_code": exit_code, "timed_out": timed_out, "stderr": str(self.tmp / "err"), "stream": str(self.tmp / "out")} + + def test_each_cli_expired_login_message_is_an_auth_death(self): + from harnesses import claude_code, codex, grok + cases = [(claude_code.LOGIN, {"stream": '{"type":"result","is_error":true,"result":"Invalid API key \u00b7 Please run /login"}\n'}), + (claude_code.LOGIN, {"stderr": "Login expired \u00b7 Please run /login"}), + (codex.LOGIN, {"stderr": "Your access token could not be refreshed because your refresh token was already used. Please log out and sign in again."}), + (grok.LOGIN, {"stderr": "Authentication required. Run `grok login` to re-authenticate."})] + for login, streams in cases: + with self.subTest(cli=login.cli, streams=list(streams)): + self.assertTrue(auth_death(login, self.record(**streams))) + + def test_a_turn_that_exited_zero_or_timed_out_or_died_for_another_reason_is_not_an_auth_death(self): + from harnesses import claude_code + login = claude_code.LOGIN + text = "Please run /login" + for name, record in (("exit zero", self.record(stderr=text, exit_code=0)), + ("timed out", self.record(stderr=text, exit_code=-9, timed_out=True)), + ("other failure", self.record(stderr="segmentation fault"))): + with self.subTest(name): + self.assertIsNone(auth_death(login, record)) + + def test_a_login_message_quoted_early_in_a_longer_stream_is_not_an_auth_death(self): + from harnesses import claude_code + stream = '{"type":"assistant","text":"the docs say Please run /login"}\n{"type":"result","is_error":true,"result":"max turns"}\n' + record = self.record(stream=stream) + self.assertIsNone(auth_death(claude_code.LOGIN, record)) + + def test_drive_ends_the_run_at_the_dying_turn_and_names_the_login(self): + from harnesses import claude_code + turns = [] + + def turn(run, text, index): + turns.append(index) + return {"index": index, **self.record(stderr="Not logged in \u00b7 Please run /login")} + + module = mock.Mock(SHARES_HOST_TMP=False, LOGIN=claude_code.LOGIN, turn=turn) + run = live.Run(self.tmp, "claude-code", {"id": "x", "fixture": "relay", "turns": ["one", "two"]}, "0" * 40, 60) + with mock.patch.object(live, "_write_record"): + login, detail = live.drive(module, run, object()) + self.assertEqual((login.cli, turns), ("claude", [0])) + self.assertIn("turn 0 exited 1", detail) + self.assertIn("Not logged in", detail) + + def test_drive_reports_a_login_that_died_in_prepare(self): + from harnesses import grok + module = mock.Mock(SHARES_HOST_TMP=False, LOGIN=grok.LOGIN, prepare=mock.Mock(side_effect=LoginRequired(grok.LOGIN, "token expired"))) + run = live.Run(self.tmp, "grok", {"id": "x", "fixture": "relay", "turns": ["one"]}, "0" * 40, 60) + self.assertEqual(live.drive(module, run, object()), (grok.LOGIN, "token expired")) + module.turn.assert_not_called() + + +@unittest.skipUnless((platform.system(), platform.release(), platform.machine()) == ("Darwin", "25.6.0", "arm64"), + "native parent grading requires the reviewed Darwin 25.6.0 arm64 runtime") +class ErrorRun(unittest.TestCase): + def test_a_run_that_dies_on_auth_is_graded_error_not_fail_and_stops_the_wave(self): + import report + from harnesses import codex + case = live.load_case("principle-steer-run") + case["turns"] = ["first", "second"] + case["tmp_lock"] = False + with tempfile.TemporaryDirectory(prefix="pstack-live-error-") as tmp: + out = Path(tmp).resolve() + + def turn(run, text, index): + (run.root / "err").write_text("Please log out and sign in again.") + (run.root / "out").write_bytes(b"") + return {"index": index, "session_id": "s", "argv": ["codex", text], "exit_code": 1, "timed_out": False, + "duration_s": 0.1, "stream": str(run.root / "out"), "stderr": str(run.root / "err")} + + output = io.StringIO() + with mock.patch.object(live, "load_case", return_value=case), mock.patch.object(codex, "prepare"), \ + mock.patch.object(codex, "turn", side_effect=turn) as turns, redirect_stdout(output): + with self.assertRaisesRegex(live.LoginRequired, "Run `codex login`"): + live.run_case("codex", case["id"], "HEAD", out, 0) + [root] = [p.parent for p in out.rglob("verdict.json")] + verdict = json.loads((root / "verdict.json").read_text()) + table = report.render(out, upstream=out) + self.assertEqual(turns.call_count, 1) + self.assertEqual({r["verdict"] for r in verdict["promises"].values()}, {"ERROR"}) + self.assertEqual(set(verdict["promises"]), set(case["promises"])) + self.assertEqual(verdict["error"]["relogin"], "codex login") + self.assertEqual(set(json.loads(output.getvalue())["verdicts"].values()), {"ERROR"}) + self.assertIn("codex pass rate: none graded (ERROR", table) + + class MakeProject(unittest.TestCase): def build(self, commit, branch=None, extra_steps=()): tmp = Path(tempfile.mkdtemp()) diff --git a/evals/promises/test_report.py b/evals/promises/test_report.py index 10e40ec2..7cc0b848 100644 --- a/evals/promises/test_report.py +++ b/evals/promises/test_report.py @@ -27,6 +27,31 @@ def test_the_claude_skill_listing_state_of_each_run_is_counted(self): table = report.render(Path(out), upstream=Path(out)) self.assertEqual(table.splitlines()[-1], "claude-code skill listing: absent 1, installed 2") + def run_dir(self, out, name, harness, verdicts): + run = Path(out) / name + run.mkdir() + (run / "verdict.json").write_text(json.dumps({"harness": harness, "promises": { + pid: {"verdict": v, "failures": [], "evidence": []} for pid, v in verdicts.items()}})) + + def test_an_error_run_drops_out_of_the_pass_rate_and_keeps_its_cell(self): + with tempfile.TemporaryDirectory() as out: + self.run_dir(out, "a", "claude-code", {"p1": "PASS", "p2": "FAIL"}) + self.run_dir(out, "b", "claude-code", {"p1": "ERROR", "p2": "ERROR"}) + self.run_dir(out, "c", "claude-code", {"p3": "INCONCLUSIVE"}) + self.run_dir(out, "d", "codex", {"p1": "ERROR"}) + table = report.render(Path(out), upstream=Path(out)) + self.assertIn("claude-code pass rate: 1 of 3 (ERROR 2 excluded)", table) + self.assertIn("codex pass rate: none graded (ERROR 1 excluded)", table) + self.assertIn("| `p1` | ERROR 1, PASS 1 | ERROR |", table) + self.assertIn("ERROR 3, FAIL 1, INCONCLUSIVE 1, PASS 1", table) + + def test_a_harness_with_no_error_prints_a_plain_pass_rate(self): + with tempfile.TemporaryDirectory() as out: + self.run_dir(out, "a", "grok", {"p1": "PASS", "p2": "PASS", "p3": "FAIL"}) + table = report.render(Path(out), upstream=Path(out)) + self.assertIn("grok pass rate: 2 of 3\n", table + "\n") + self.assertNotIn("excluded", table) + if __name__ == "__main__": unittest.main() From 12407d37d545590eaf778dbaf8a7a99f692ca77c Mon Sep 17 00:00:00 2001 From: mdsmithaustin <15272014+mdsmithaustin@users.noreply.github.com> Date: Mon, 12 Oct 2026 13:26:01 -0600 Subject: [PATCH 02/12] no-mistakes(document): Clarify live login checks and ERROR verdict documentation --- evals/promises/README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/evals/promises/README.md b/evals/promises/README.md index 8de7edbb..8aff6cf8 100644 --- a/evals/promises/README.md +++ b/evals/promises/README.md @@ -71,11 +71,11 @@ python3 evals/promises/live.py run --harness --c python3 evals/promises/live.py report --upstream-ref upstream/main ``` -Each run makes a fresh git repository from `fixtures//` and `histories//`. Each history step is committed in order, except a step with `"commit": false`, which stays as uncommitted work in the tree. A step with `"branch"` switches to that branch first, creating it at the current commit when it does not exist, so later steps land there and `main` keeps the fixture. It installs `skills/` from `--skills-at` (default `HEAD`) where the harness discovers project skills, and runs the case's turns headless. Each run directory gets `run.json`, `trace.json`, and `verdict.json`. Every adapter writes the same trace schema, with events, files read, worklist snapshots, spawns, and the final reply. A tool event carries its harness call id, so each result pairs with its own call. `oracles.py` grades each promise as `PASS`, `FAIL`, or `INCONCLUSIVE`, and `live.py` alone adds `ERROR` for a dead login. A verdict is `INCONCLUSIVE` when the trace cannot show the behavior, or when the harness reports `x_host_skill_hits`, which means the user's own skills touched the run. Some oracles also return `INCONCLUSIVE` when only a judge can decide, such as whether a reply's wording reports a review `pass`. That verdict carries `needs_judge: true` and an `excerpt` of the graded reply in `verdict.json`. In a multi-turn case the graded reply can come from an earlier turn than `final_reply`, and the verdict's evidence names that turn. When the excerpt is not enough, read that turn's text events in `trace.json` before you tally the run. +Each run makes a fresh git repository from `fixtures//` and `histories//`. Each history step is committed in order, except a step with `"commit": false`, which stays as uncommitted work in the tree. A step with `"branch"` switches to that branch first, creating it at the current commit when it does not exist, so later steps land there and `main` keeps the fixture. It installs `skills/` from `--skills-at` (default `HEAD`) where the harness discovers project skills, and runs the case's turns headless. A run that completes harvest and grading gets `run.json`, `trace.json`, and `verdict.json`. Every adapter writes the same trace schema, with events, files read, worklist snapshots, spawns, and the final reply. A tool event carries its harness call id, so each result pairs with its own call. `oracles.py` grades each promise as `PASS`, `FAIL`, or `INCONCLUSIVE`, and `live.py` alone adds `ERROR` for a dead login. A verdict is `INCONCLUSIVE` when the trace cannot show the behavior, or when the harness reports `x_host_skill_hits`, which means the user's own skills touched the run. Some oracles also return `INCONCLUSIVE` when only a judge can decide, such as whether a reply's wording reports a review `pass`. That verdict carries `needs_judge: true` and an `excerpt` of the graded reply in `verdict.json`. In a multi-turn case the graded reply can come from an earlier turn than `final_reply`, and the verdict's evidence names that turn. When the excerpt is not enough, read that turn's text events in `trace.json` before you tally the run. -Before the first case, `live.py run` asks each CLI whether its login is usable, at no model cost. A failed check stops the wave with exit 3 before any case runs, and the message names the CLI and its login command. Claude Code runs `claude auth status`. Codex needs the `auth.json` file and a passing `codex login status`. Grok refuses a host token that expires within five minutes, because the run's read-only provider never refreshes it. Claude Code and Codex refresh their own access tokens when they run, so their checks do not read token expiry. Hermes has no free status command, and its preparation step still refuses a missing `auth.json`. +Before the first case, `live.py run` checks the selected Claude Code, Codex, or Grok login at no model cost. A failed check stops the wave with exit 3 before any case runs, and the message names the CLI and its login command. Claude Code runs `claude auth status`. Codex needs the `auth.json` file and a passing `codex login status`. Grok refuses a host token that expires within five minutes, because the run's read-only provider never refreshes it. Claude Code and Codex refresh their own access tokens when they run, so their checks do not read token expiry. Hermes has no preflight check, and its preparation step still refuses a missing `auth.json`. -A run that dies on auth after the wave starts is graded `ERROR`, not `FAIL` or `INCONCLUSIVE`. A turn counts as an auth death when it exited nonzero without timing out and the tail of its stderr or the last line of its stream carries that CLI's login-failure text. `live.py` writes a `verdict.json` that marks every promise of the case `ERROR` and records the CLI and its login command under `error`. It skips harvest and grading, since a dead login says nothing about the skill, and it stops the wave after that run. `live.py report` shows `ERROR` in the table and leaves those verdicts out of each harness's `pass rate: of ` line. `INCONCLUSIVE` stays in the denominator. A run that died on auth but exited 0 still grades as before. +For Claude Code, Codex, and Grok, a turn counts as an auth death when it exited nonzero without timing out and the tail of its stderr or the last nonempty line of its stream carries that CLI's login-failure text. A `LoginRequired` failure during preparation also produces `ERROR`. `live.py` writes a `verdict.json` that marks every promise of the case `ERROR` and records the CLI and its login command under `error`. It skips harvest and grading, so the run has no `trace.json`, and stops the wave with exit 3 after that run. `live.py report` shows `ERROR` in the table and leaves those verdicts out of each harness's `pass rate: of ` line. This rate counts applicable promise verdicts, not runs. `INCONCLUSIVE` stays in the denominator. When every applicable verdict is `ERROR`, the rate reads `none graded`. A turn that exited 0 still grades as before, even if it contains login-failure text. `--model` and `--effort` set the model and reasoning effort for every turn of every case in one `run` invocation, on Claude Code and Codex. Each turn's recorded argv carries both values, and the sealed run metadata keeps its fields. Other adapters refuse the flags. From 3a92d0536dbbd6ba027b6e50f087e7e45575a1b7 Mon Sep 17 00:00:00 2001 From: mdsmithaustin <15272014+mdsmithaustin@users.noreply.github.com> Date: Mon, 12 Oct 2026 13:31:57 -0600 Subject: [PATCH 03/12] evals: read the Claude login status under the turn environment --- evals/promises/harnesses/claude_code.py | 9 ++++++--- evals/promises/test_live.py | 5 +++++ 2 files changed, 11 insertions(+), 3 deletions(-) diff --git a/evals/promises/harnesses/claude_code.py b/evals/promises/harnesses/claude_code.py index b8d9b730..a38147b1 100644 --- a/evals/promises/harnesses/claude_code.py +++ b/evals/promises/harnesses/claude_code.py @@ -201,12 +201,15 @@ def store(run): def preflight(): - """Ask the CLI for its login status. The CLI refreshes an expired access token itself when it runs, so this fails only - when it reports no login.""" + """Ask the CLI for its login status under the few variables a turn keeps, so an API key or config directory the host shell + exports cannot report a login the turns never see. The CLI refreshes an expired access token itself when it runs, so this + fails only when it reports no login.""" binary = CLAUDE_BINARY if CLAUDE_BINARY.is_file() else shutil.which("claude") if not binary: raise LoginRequired(LOGIN, "no claude binary to ask") - out = check_status(LOGIN, [binary, "auth", "status"]) + env = {"HOME": os.environ["HOME"], "USER": os.environ.get("USER", ""), "LOGNAME": os.environ.get("LOGNAME", ""), + "PATH": "/usr/bin:/bin", "LANG": "en_US.UTF-8", "TERM": "dumb", "CLAUDE_SECURESTORAGE_CONFIG_DIR": ""} + out = check_status(LOGIN, [binary, "auth", "status"], env=env) try: logged_in = json.loads(out).get("loggedIn") except ValueError: diff --git a/evals/promises/test_live.py b/evals/promises/test_live.py index 2e8f5f3b..55a7944c 100644 --- a/evals/promises/test_live.py +++ b/evals/promises/test_live.py @@ -417,6 +417,11 @@ def test_claude_code_exiting_zero_with_loggedin_false_still_stops_the_wave(self) outcome = self.wave("claude-code", self.claude('echo \'{"loggedIn": false}\'\n')) self.assertStopped(outcome, "claude", "claude auth login") + def test_claude_code_status_does_not_see_credentials_only_the_host_shell_exports(self): + script = 'if [ -n "$ANTHROPIC_API_KEY" ]; then echo \'{"loggedIn": true}\'; else echo \'{"loggedIn": false}\'; exit 1; fi\n' + outcome = self.wave("claude-code", self.claude(script), mock.patch.dict(os.environ, {"ANTHROPIC_API_KEY": "host-only"})) + self.assertStopped(outcome, "claude", "claude auth login") + def test_codex_login_status_failing_stops_the_wave(self): script = 'case "$1" in --version) echo codex 1;; *) echo "Not logged in" >&2; exit 1;; esac\n' self.assertStopped(self.wave("codex", self.codex(script)), "codex", "codex login") From 0ca9537f1697754886aaa365ff1e308359a860d4 Mon Sep 17 00:00:00 2001 From: mdsmithaustin <15272014+mdsmithaustin@users.noreply.github.com> Date: Mon, 12 Oct 2026 13:38:58 -0600 Subject: [PATCH 04/12] no-mistakes(document): Document Claude preflight environment isolation --- evals/promises/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/evals/promises/README.md b/evals/promises/README.md index 8aff6cf8..ce197b6b 100644 --- a/evals/promises/README.md +++ b/evals/promises/README.md @@ -73,7 +73,7 @@ python3 evals/promises/live.py report --upstream-ref upstream/main Each run makes a fresh git repository from `fixtures//` and `histories//`. Each history step is committed in order, except a step with `"commit": false`, which stays as uncommitted work in the tree. A step with `"branch"` switches to that branch first, creating it at the current commit when it does not exist, so later steps land there and `main` keeps the fixture. It installs `skills/` from `--skills-at` (default `HEAD`) where the harness discovers project skills, and runs the case's turns headless. A run that completes harvest and grading gets `run.json`, `trace.json`, and `verdict.json`. Every adapter writes the same trace schema, with events, files read, worklist snapshots, spawns, and the final reply. A tool event carries its harness call id, so each result pairs with its own call. `oracles.py` grades each promise as `PASS`, `FAIL`, or `INCONCLUSIVE`, and `live.py` alone adds `ERROR` for a dead login. A verdict is `INCONCLUSIVE` when the trace cannot show the behavior, or when the harness reports `x_host_skill_hits`, which means the user's own skills touched the run. Some oracles also return `INCONCLUSIVE` when only a judge can decide, such as whether a reply's wording reports a review `pass`. That verdict carries `needs_judge: true` and an `excerpt` of the graded reply in `verdict.json`. In a multi-turn case the graded reply can come from an earlier turn than `final_reply`, and the verdict's evidence names that turn. When the excerpt is not enough, read that turn's text events in `trace.json` before you tally the run. -Before the first case, `live.py run` checks the selected Claude Code, Codex, or Grok login at no model cost. A failed check stops the wave with exit 3 before any case runs, and the message names the CLI and its login command. Claude Code runs `claude auth status`. Codex needs the `auth.json` file and a passing `codex login status`. Grok refuses a host token that expires within five minutes, because the run's read-only provider never refreshes it. Claude Code and Codex refresh their own access tokens when they run, so their checks do not read token expiry. Hermes has no preflight check, and its preparation step still refuses a missing `auth.json`. +Before the first case, `live.py run` checks the selected Claude Code, Codex, or Grok login at no model cost. A failed check stops the wave with exit 3 before any case runs, and the message names the CLI and its login command. Claude Code runs `claude auth status` under the turn's minimal environment, without a host-exported `ANTHROPIC_API_KEY` that could mask a missing login. Codex needs the `auth.json` file and a passing `codex login status`. Grok refuses a host token that expires within five minutes, because the run's read-only provider never refreshes it. Claude Code and Codex refresh their own access tokens when they run, so their checks do not read token expiry. Hermes has no preflight check, and its preparation step still refuses a missing `auth.json`. For Claude Code, Codex, and Grok, a turn counts as an auth death when it exited nonzero without timing out and the tail of its stderr or the last nonempty line of its stream carries that CLI's login-failure text. A `LoginRequired` failure during preparation also produces `ERROR`. `live.py` writes a `verdict.json` that marks every promise of the case `ERROR` and records the CLI and its login command under `error`. It skips harvest and grading, so the run has no `trace.json`, and stops the wave with exit 3 after that run. `live.py report` shows `ERROR` in the table and leaves those verdicts out of each harness's `pass rate: of ` line. This rate counts applicable promise verdicts, not runs. `INCONCLUSIVE` stays in the denominator. When every applicable verdict is `ERROR`, the rate reads `none graded`. A turn that exited 0 still grades as before, even if it contains login-failure text. From 6cd6c8e903f7f23d19159743fb7176436194295c Mon Sep 17 00:00:00 2001 From: mdsmithaustin <15272014+mdsmithaustin@users.noreply.github.com> Date: Mon, 12 Oct 2026 14:15:12 -0600 Subject: [PATCH 05/12] evals: keep a real result when a login dies mid-run, and catch the Grok External-auth death - A turn that dies on auth after earlier turns ran is still harvested and graded. A PASS or FAIL the grader decides stands, undecided promises become ERROR, and the wave still stops. - error_verdict leaves a promise the oracle marks not_applicable for the harness blank instead of ERROR, so a dead Codex feature run no longer shows ERROR for Claude-only and Hermes/Grok-only promises. - The Grok death pattern covers the External-auth 401 text, recorded from a live run that still graded FAIL. - The README says the Claude preflight checks the host login without the turn's private CLAUDE_CONFIG_DIR. --- evals/promises/README.md | 4 +- evals/promises/harnesses/grok.py | 3 +- evals/promises/harnesses/login.py | 23 +++- evals/promises/live.py | 15 ++- evals/promises/test_live.py | 97 +++++++++++++- .../claude-code-401-last-line.jsonl | 1 + .../claude-code-after-ipc-trace.json | 125 ++++++++++++++++++ .../auth_death/grok-external-auth.stderr.txt | 28 ++++ .../auth_death/grok-external-auth.stream.json | 1 + 9 files changed, 280 insertions(+), 17 deletions(-) create mode 100644 evals/promises/testdata/auth_death/claude-code-401-last-line.jsonl create mode 100644 evals/promises/testdata/auth_death/claude-code-after-ipc-trace.json create mode 100644 evals/promises/testdata/auth_death/grok-external-auth.stderr.txt create mode 100644 evals/promises/testdata/auth_death/grok-external-auth.stream.json diff --git a/evals/promises/README.md b/evals/promises/README.md index ce197b6b..f8cbb60f 100644 --- a/evals/promises/README.md +++ b/evals/promises/README.md @@ -73,9 +73,9 @@ python3 evals/promises/live.py report --upstream-ref upstream/main Each run makes a fresh git repository from `fixtures//` and `histories//`. Each history step is committed in order, except a step with `"commit": false`, which stays as uncommitted work in the tree. A step with `"branch"` switches to that branch first, creating it at the current commit when it does not exist, so later steps land there and `main` keeps the fixture. It installs `skills/` from `--skills-at` (default `HEAD`) where the harness discovers project skills, and runs the case's turns headless. A run that completes harvest and grading gets `run.json`, `trace.json`, and `verdict.json`. Every adapter writes the same trace schema, with events, files read, worklist snapshots, spawns, and the final reply. A tool event carries its harness call id, so each result pairs with its own call. `oracles.py` grades each promise as `PASS`, `FAIL`, or `INCONCLUSIVE`, and `live.py` alone adds `ERROR` for a dead login. A verdict is `INCONCLUSIVE` when the trace cannot show the behavior, or when the harness reports `x_host_skill_hits`, which means the user's own skills touched the run. Some oracles also return `INCONCLUSIVE` when only a judge can decide, such as whether a reply's wording reports a review `pass`. That verdict carries `needs_judge: true` and an `excerpt` of the graded reply in `verdict.json`. In a multi-turn case the graded reply can come from an earlier turn than `final_reply`, and the verdict's evidence names that turn. When the excerpt is not enough, read that turn's text events in `trace.json` before you tally the run. -Before the first case, `live.py run` checks the selected Claude Code, Codex, or Grok login at no model cost. A failed check stops the wave with exit 3 before any case runs, and the message names the CLI and its login command. Claude Code runs `claude auth status` under the turn's minimal environment, without a host-exported `ANTHROPIC_API_KEY` that could mask a missing login. Codex needs the `auth.json` file and a passing `codex login status`. Grok refuses a host token that expires within five minutes, because the run's read-only provider never refreshes it. Claude Code and Codex refresh their own access tokens when they run, so their checks do not read token expiry. Hermes has no preflight check, and its preparation step still refuses a missing `auth.json`. +Before the first case, `live.py run` checks the selected Claude Code, Codex, or Grok login at no model cost. A failed check stops the wave with exit 3 before any case runs, and the message names the CLI and its login command. Claude Code runs `claude auth status` with the host `HOME`, `USER`, a minimal `PATH`, and an empty `CLAUDE_SECURESTORAGE_CONFIG_DIR`, so a host-exported `ANTHROPIC_API_KEY` cannot mask a missing login. It does not set the private `CLAUDE_CONFIG_DIR` each turn gets, since that directory exists only after a run is prepared, so it checks the host login. Codex needs the `auth.json` file and a passing `codex login status`. Grok refuses a host token that expires within five minutes, because the run's read-only provider never refreshes it. Claude Code and Codex refresh their own access tokens when they run, so their checks do not read token expiry. Hermes has no preflight check, and its preparation step still refuses a missing `auth.json`. -For Claude Code, Codex, and Grok, a turn counts as an auth death when it exited nonzero without timing out and the tail of its stderr or the last nonempty line of its stream carries that CLI's login-failure text. A `LoginRequired` failure during preparation also produces `ERROR`. `live.py` writes a `verdict.json` that marks every promise of the case `ERROR` and records the CLI and its login command under `error`. It skips harvest and grading, so the run has no `trace.json`, and stops the wave with exit 3 after that run. `live.py report` shows `ERROR` in the table and leaves those verdicts out of each harness's `pass rate: of ` line. This rate counts applicable promise verdicts, not runs. `INCONCLUSIVE` stays in the denominator. When every applicable verdict is `ERROR`, the rate reads `none graded`. A turn that exited 0 still grades as before, even if it contains login-failure text. +For Claude Code, Codex, and Grok, a turn counts as an auth death when it exited nonzero without timing out and the tail of its stderr or the last nonempty line of its stream carries that CLI's login-failure text. A `LoginRequired` failure during preparation stops the run before any turn. `live.py` writes a `verdict.json` that marks every promise that applies to the harness `ERROR` and records the CLI and its login command under `error`. It skips harvest and grading, so that run has no `trace.json`. When a turn dies on auth, `live.py` still harvests and grades the run. A `PASS` or `FAIL` the grader decides stands, and every other promise becomes `ERROR`. Running `live.py grade` on that run later re-grades it from its trace and writes a verdict without the `ERROR` marks. In both cases a promise the oracle marks `not_applicable` for the harness keeps that mark, so it stays out of the report, and the wave stops with exit 3 after the run. `live.py report` shows `ERROR` in the table and leaves those verdicts out of each harness's `pass rate: of ` line. This rate counts applicable promise verdicts, not runs. `INCONCLUSIVE` stays in the denominator. When every applicable verdict is `ERROR`, the rate reads `none graded`. A turn that exited 0 still grades as before, even if it contains login-failure text. `--model` and `--effort` set the model and reasoning effort for every turn of every case in one `run` invocation, on Claude Code and Codex. Each turn's recorded argv carries both values, and the sealed run metadata keeps its fields. Other adapters refuse the flags. diff --git a/evals/promises/harnesses/grok.py b/evals/promises/harnesses/grok.py index 00db3698..78e2b6b1 100644 --- a/evals/promises/harnesses/grok.py +++ b/evals/promises/harnesses/grok.py @@ -134,7 +134,8 @@ def host_token_expiry(auth): LOGIN = Login("grok", "grok login", re.compile( - r"Run `grok login`|auth token is invalid or expired|Not authenticated|Authentication required|no xAI credential", re.I)) + r"Run `grok login`|auth token is invalid or expired|Not authenticated|Authentication required|no xAI credential|" + r"Invalid or expired credentials|sign-in helper could not renew", re.I)) TOKEN_MARGIN = datetime.timedelta(minutes=5) diff --git a/evals/promises/harnesses/login.py b/evals/promises/harnesses/login.py index 14ca356c..7eb2f6f7 100644 --- a/evals/promises/harnesses/login.py +++ b/evals/promises/harnesses/login.py @@ -8,6 +8,7 @@ from grade_boundary import read_file ERROR = "ERROR" +DECIDED = ("PASS", "FAIL") STATUS_TIMEOUT_S = 60 DEATH_WINDOW = 4000 @@ -60,10 +61,22 @@ def _read(path): return "" -def error_verdict(case, harness, skills_at, login, detail): - """The verdict for a run a dead login ended. Every promise of the case is ERROR, so no skill result is claimed and - report.py leaves the run out of every pass rate.""" +def error_verdict(case, harness, skills_at, login, detail, graded=None): + """The verdict for a run a dead login ended. `graded` holds the grader's promise verdicts when turns ran before the + death. A verdict the grader decided (PASS or FAIL) stands, as does a promise that does not apply to the harness. Every + other promise is ERROR, so no skill result is claimed for it and report.py leaves it out of every pass rate.""" reason = f"{login.cli} login died: {detail}" + promises = {} + for pid in case["promises"]: + result = graded[pid] if graded else _applicability(pid, harness, case) + keep = result.get("not_applicable") or (graded and result["verdict"] in DECIDED) + promises[pid] = result if keep else {"verdict": ERROR, "failures": [reason], "evidence": []} return {"case": case["id"], "harness": harness, "skills_at": skills_at, - "error": {"cli": login.cli, "relogin": login.relogin, "detail": detail}, - "promises": {pid: {"verdict": ERROR, "failures": [reason], "evidence": []} for pid in case["promises"]}} + "error": {"cli": login.cli, "relogin": login.relogin, "detail": detail}, "promises": promises} + + +def _applicability(pid, harness, case): + """The oracle's own answer on a run with no trace. Only its `not_applicable` flag is read, so a promise that never + applies to this harness stays blank instead of turning ERROR.""" + import oracles + return oracles.check(pid, {"harness": harness}, case, None) diff --git a/evals/promises/live.py b/evals/promises/live.py index d8dd17f1..66ffe27c 100644 --- a/evals/promises/live.py +++ b/evals/promises/live.py @@ -17,7 +17,7 @@ from dataclasses import dataclass, field from pathlib import Path -from grade_boundary import create_file, grade, GradeRefused, read_file, _before_turns, _json, _lookup, _seal, _write_record +from grade_boundary import create_file, grade, GradeRefused, read_file, write_file, _before_turns, _json, _lookup, _seal, _write_record from harnesses.login import LoginRequired, auth_death, error_verdict HERE = Path(__file__).resolve().parent @@ -299,14 +299,15 @@ def run_case(harness, case_id, skills_at, out, index, hermes_retain_out=None, mo pair = None try: died = drive(module, run, authorization) - if died: - verdict = error_verdict(case, harness, commit, *died) - with create_file(root / "verdict.json") as stream: - stream.write(_json(verdict)) - else: + graded = None + if run.turns or not died: trace = {**module.harvest(run), "x_implicit_off": run.implicit_off} _seal(authorization, meta(run), trace) - verdict = grade(authorization) + graded = grade(authorization) + verdict = graded + if died: + verdict = error_verdict(case, harness, commit, *died, graded and graded["promises"]) + write_file(root / "verdict.json", _json(verdict)) except BaseException as original: owner = getattr(run, "_hermes_evidence", None) if owner is not None and retention is not None: diff --git a/evals/promises/test_live.py b/evals/promises/test_live.py index 55a7944c..29f58271 100644 --- a/evals/promises/test_live.py +++ b/evals/promises/test_live.py @@ -15,7 +15,9 @@ import live from grade_boundary import GradeRefused, _authorize_fixture -from harnesses.login import LoginRequired, auth_death +from harnesses.login import LoginRequired, auth_death, error_verdict + +FIXTURES = Path(__file__).resolve().parent / "testdata" / "auth_death" CODEX_CHAT = """I'm using `poteto-mode`. I'll identify the command first. @@ -471,6 +473,19 @@ def test_each_cli_expired_login_message_is_an_auth_death(self): with self.subTest(cli=login.cli, streams=list(streams)): self.assertTrue(auth_death(login, self.record(**streams))) + def test_the_grok_external_auth_failure_a_real_run_ended_on_is_an_auth_death(self): + from harnesses import grok + record = {"exit_code": 1, "timed_out": False, "stream": str(FIXTURES / "grok-external-auth.stream.json"), + "stderr": str(FIXTURES / "grok-external-auth.stderr.txt")} + self.assertTrue(auth_death(grok.LOGIN, record)) + self.assertTrue(auth_death(grok.LOGIN, {"exit_code": 1, "timed_out": False, "stream": record["stream"]})) + self.assertTrue(auth_death(grok.LOGIN, {"exit_code": 1, "timed_out": False, "stderr": record["stderr"]})) + + def test_the_claude_code_401_a_real_run_ended_on_is_an_auth_death(self): + from harnesses import claude_code + record = {"exit_code": 1, "timed_out": False, "stream": str(FIXTURES / "claude-code-401-last-line.jsonl")} + self.assertIn("Failed to authenticate", auth_death(claude_code.LOGIN, record)) + def test_a_turn_that_exited_zero_or_timed_out_or_died_for_another_reason_is_not_an_auth_death(self): from harnesses import claude_code login = claude_code.LOGIN @@ -529,8 +544,11 @@ def turn(run, text, index): return {"index": index, "session_id": "s", "argv": ["codex", text], "exit_code": 1, "timed_out": False, "duration_s": 0.1, "stream": str(run.root / "out"), "stderr": str(run.root / "err")} + def prepare(run): + (run.root / "launch.json").write_text(json.dumps({"path": "synthetic-no-model", "source": "test", "version": "test", "rejected": []})) + output = io.StringIO() - with mock.patch.object(live, "load_case", return_value=case), mock.patch.object(codex, "prepare"), \ + with mock.patch.object(live, "load_case", return_value=case), mock.patch.object(codex, "prepare", side_effect=prepare), \ mock.patch.object(codex, "turn", side_effect=turn) as turns, redirect_stdout(output): with self.assertRaisesRegex(live.LoginRequired, "Run `codex login`"): live.run_case("codex", case["id"], "HEAD", out, 0) @@ -544,6 +562,81 @@ def turn(run, text, index): self.assertEqual(set(json.loads(output.getvalue())["verdicts"].values()), {"ERROR"}) self.assertIn("codex pass rate: none graded (ERROR", table) + def test_a_login_that_dies_after_the_graded_behavior_keeps_the_graders_verdict(self): + import report + from harnesses import claude_code + case = live.load_case("feature-boundary-run") + case["tmp_lock"] = False + trace = json.loads((FIXTURES / "claude-code-after-ipc-trace.json").read_text()) + + def turn(run, text, index): + (run.root / "err").write_bytes(b"") + return {"index": index, "session_id": "s", "argv": ["claude", text], "exit_code": 1, "timed_out": False, + "duration_s": 582.6, "stream": str(FIXTURES / "claude-code-401-last-line.jsonl"), "stderr": str(run.root / "err")} + + with tempfile.TemporaryDirectory(prefix="pstack-live-late-death-") as tmp: + out = Path(tmp).resolve() + output = io.StringIO() + with mock.patch.object(live, "load_case", return_value=case), mock.patch.object(claude_code, "prepare"), \ + mock.patch.object(claude_code, "turn", side_effect=turn), \ + mock.patch.object(claude_code, "harvest", return_value=trace), redirect_stdout(output): + with self.assertRaisesRegex(live.LoginRequired, "Run `claude auth login`"): + live.run_case("claude-code", case["id"], "HEAD", out, 0) + [root] = [p.parent for p in out.rglob("verdict.json")] + verdict = json.loads((root / "verdict.json").read_text()) + sealed = json.loads((root / "trace.json").read_text()) + table = report.render(out, upstream=out) + promise = "poteto-mode-triggers-architect-on-boundary-crossing" + self.assertEqual(verdict["promises"][promise]["verdict"], "PASS") + self.assertEqual(verdict["error"]["cli"], "claude") + self.assertEqual(sealed["harness"], "claude-code") + self.assertEqual(json.loads(output.getvalue())["verdicts"], {promise: "PASS"}) + self.assertIn("claude-code pass rate: 1 of 1", table) + + +class ErrorVerdict(unittest.TestCase): + def test_pass_and_fail_survive_an_auth_death_and_only_the_undecided_become_error(self): + from harnesses import codex + case = {"id": "x", "promises": ["a", "b", "c", "d"]} + graded = {"a": {"verdict": "PASS", "failures": [], "evidence": ["e"]}, + "b": {"verdict": "FAIL", "failures": ["f"], "evidence": []}, + "c": {"verdict": "INCONCLUSIVE", "failures": ["run killed"], "evidence": []}, + "d": {"verdict": "INCONCLUSIVE", "failures": ["applies elsewhere"], "evidence": [], "not_applicable": True}} + verdict = error_verdict(case, "codex", "0" * 40, codex.LOGIN, "turn 0 exited 1", graded) + self.assertEqual({p: r["verdict"] for p, r in verdict["promises"].items()}, + {"a": "PASS", "b": "FAIL", "c": "ERROR", "d": "INCONCLUSIVE"}) + self.assertEqual(verdict["promises"]["a"], graded["a"]) + self.assertTrue(verdict["promises"]["d"]["not_applicable"]) + self.assertEqual(verdict["promises"]["c"]["failures"], ["codex login died: turn 0 exited 1"]) + self.assertEqual(verdict["error"]["relogin"], "codex login") + + def test_a_dead_codex_run_marks_error_only_the_promises_that_apply_to_codex(self): + import report + from harnesses import codex + case = live.load_case("feature-run") + verdict = error_verdict(case, "codex", "0" * 40, codex.LOGIN, "no login at auth.json") + gated = {pid for pid, r in verdict["promises"].items() if r.get("not_applicable")} + self.assertEqual(gated, {"claude-effort-applies-via-effort-agents", "persona-delivery-hermes-grok"}) + self.assertEqual({r["verdict"] for pid, r in verdict["promises"].items() if pid not in gated}, {"ERROR"}) + self.assertEqual(len(case["promises"]) - len(gated), 9) + with tempfile.TemporaryDirectory(prefix="pstack-error-verdict-") as tmp: + (Path(tmp) / "verdict.json").write_text(json.dumps(verdict)) + table = report.render(Path(tmp), upstream=Path(tmp)) + self.assertIn("codex pass rate: none graded (ERROR 9 excluded)", table) + self.assertNotIn("claude-effort-applies-via-effort-agents", table) + + def test_every_promise_of_every_live_case_answers_the_applicability_probe_on_every_harness(self): + import report + from harnesses import codex + for case_id in sorted(p.name for p in live.CASES.iterdir() if (p / "case.json").is_file()): + case = live.load_case(case_id) + if case.get("deferred") or case.get("kind", "live") != "live": + continue + for harness in report.HARNESSES: + with self.subTest(case=case_id, harness=harness): + verdict = error_verdict(case, harness, "0" * 40, codex.LOGIN, "probe") + self.assertEqual(set(verdict["promises"]), set(case["promises"])) + class MakeProject(unittest.TestCase): def build(self, commit, branch=None, extra_steps=()): diff --git a/evals/promises/testdata/auth_death/claude-code-401-last-line.jsonl b/evals/promises/testdata/auth_death/claude-code-401-last-line.jsonl new file mode 100644 index 00000000..c4dadd47 --- /dev/null +++ b/evals/promises/testdata/auth_death/claude-code-401-last-line.jsonl @@ -0,0 +1 @@ +{"duration_api_ms":833771,"stop_reason":"stop_sequence","session_id":"aa6aba22-2895-4986-849d-99d32e3c80a6","total_cost_usd":3.4189594,"usage":{"output_tokens_details":{"thinking_tokens":0},"input_tokens":0,"cache_creation_input_tokens":0,"cache_read_input_tokens":0,"output_tokens":0,"server_tool_use":{"web_search_requests":0,"web_fetch_requests":0},"service_tier":"standard","cache_creation":{"ephemeral_1h_input_tokens":0,"ephemeral_5m_input_tokens":0},"inference_geo":"","iterations":[],"speed":"standard","fallback_credit":null},"modelUsage":{"claude-opus-5-5":{"inputTokens":88,"outputTokens":57421,"cacheReadInputTokens":2174469,"cacheCreationInputTokens":158600,"webSearchRequests":0,"costUSD":2.6488437999999994,"contextWindow":1000000,"maxOutputTokens":128000,"thinkingTokens":28105,"canonicalModel":"claude-opus-5-5","provider":"firstParty","costBasis":"list"},"claude-sonnet-5-5":{"inputTokens":50,"outputTokens":36269,"cacheReadInputTokens":669728,"cacheCreationInputTokens":109352,"webSearchRequests":0,"costUSD":0.7701156,"contextWindow":1000000,"maxOutputTokens":128000,"thinkingTokens":15968,"canonicalModel":"claude-sonnet-5-5","provider":"firstParty","costBasis":"list"}},"permission_denials":[],"terminal_reason":"api_error","fast_mode_state":"off","fast_mode_disabled_reason":"sdk_opt_in_required","origin":{"kind":"task-notification","producer":"session-task"},"subagent_stats":{"spawned":6,"requested":{"background":6,"foreground":0,"unset":0},"started_in_background":6,"max_depth":1,"spawned_by_subagents":0,"completed":4,"failed":2,"killed":{"parent":0,"user":0,"system":0},"refused":{"depth_limit":0,"concurrency_limit":0,"budget":0},"by_type":{"general-purpose":5,"poteto-agent":1}},"is_error":true,"num_turns":1,"subtype":"success","api_error_status":401,"result":"Failed to authenticate. API Error: 401 OAuth access token has expired. Re-authenticate to continue.","type":"result","duration_ms":2352,"uuid":"26f6e691-27b1-4624-9c6e-674462ff2b16","queued_turn_count":0,"result_index":5} diff --git a/evals/promises/testdata/auth_death/claude-code-after-ipc-trace.json b/evals/promises/testdata/auth_death/claude-code-after-ipc-trace.json new file mode 100644 index 00000000..17d4c18b --- /dev/null +++ b/evals/promises/testdata/auth_death/claude-code-after-ipc-trace.json @@ -0,0 +1,125 @@ +{ + "harness": "claude-code", + "model": "claude-opus-5-5", + "exit_code": 1, + "duration_s": 582.6, + "events": [ + { + "seq": 0, + "turn": 0, + "kind": "user", + "text": "/poteto-mode imports from flaky partner feeds sometimes die halfway. retry each source with backoff, record the attempt count on every stored record, and let the cli set the retry policy. verify it end to end." + }, + { + "seq": 1, + "turn": 0, + "kind": "tool_call", + "name": "Bash", + "input": { + "command": "cat .claude/skills/poteto-mode/playbooks/feature.md && git log --oneline | head && git ls-files | grep -v '^.claude/' | head -50", + "description": "Read Feature playbook and list repo files" + }, + "id": "toolu_019JjbK79V23sQissTKH43Ww" + }, + { + "seq": 26, + "turn": 0, + "kind": "tool_call", + "name": "Agent", + "input": { + "description": "Architect runner 1 (fable)", + "subagent_type": "general-purpose", + "model": "fable", + "run_in_background": true, + "prompt": "You are candidate runner 1 in an architect design exploration. Read these files in full first, in this order:\n1. /private/tmp/pstack-collection-after-ipc-20261005/claude-code-after-ipc/relay-qb3al3g7/w/relay/.claude/skills/architect/SKILL.md (the workflow you're inside; you only do Phase B for one candidate)\n2. /private/tmp/pstack-collection-after-ipc-20261005/claude-code-after-ipc/relay-qb3al3g7/w/relay/.claude/skills/architect/references/runner-prompt.md (your instructions)\n3. /private/tmp/pstack-collection-after-ipc-20261005/claude-code-after-ipc/relay-qb3al3g7/w/relay/.claude/skills/architect/references/rationale-template.md and design-red-flags.md\n4. /private/tmp/pstack-collection-after-ipc-20261005/claude-code-after-ipc/relay-qb3al3g7/tmp/arena-relay-retry/grounding.md (the task and the traced subsystem)\nThen read the relay source files the grounding names (the repo is ~250 lines).\n\nRead-only on the repo: do not edit or write anything under the repo. Write your design package to /private/tmp/pstack-collection-after-ipc-20261005/claude-code-after-ipc/relay-qb3al3g7/tmp/arena-relay-retry/candidate-1/ as `design.md` (rationale per the template, including caller usage, the type/signature sketch with `raise NotImplementedError` bodies, the stored-file format, CLI flags and defaults, and an end-to-end verification plan through the real CLI). Answer every open design question in the grounding. Keep it to one or two pages.\n\nReply with at most 12 lines: the shape in brief, your answer to each open question, and the file path. No preamble." + }, + "id": "toolu_01Tj7EPNdeNAWNpL9TQw2se5" + }, + { + "seq": 28, + "turn": 0, + "kind": "tool_call", + "name": "Agent", + "input": { + "description": "Architect runner 2 (opus)", + "subagent_type": "general-purpose", + "model": "opus", + "run_in_background": true, + "prompt": "You are candidate runner 2 in an architect design exploration. Read these files in full first, in this order:\n1. /private/tmp/pstack-collection-after-ipc-20261005/claude-code-after-ipc/relay-qb3al3g7/w/relay/.claude/skills/architect/SKILL.md (the workflow you're inside; you only do Phase B for one candidate)\n2. /private/tmp/pstack-collection-after-ipc-20261005/claude-code-after-ipc/relay-qb3al3g7/w/relay/.claude/skills/architect/references/runner-prompt.md (your instructions)\n3. /private/tmp/pstack-collection-after-ipc-20261005/claude-code-after-ipc/relay-qb3al3g7/w/relay/.claude/skills/architect/references/rationale-template.md and design-red-flags.md\n4. /private/tmp/pstack-collection-after-ipc-20261005/claude-code-after-ipc/relay-qb3al3g7/tmp/arena-relay-retry/grounding.md (the task and the traced subsystem)\nThen read the relay source files the grounding names (the repo is ~250 lines).\n\nRead-only on the repo: do not edit or write anything under the repo. Write your design package to /private/tmp/pstack-collection-after-ipc-20261005/claude-code-after-ipc/relay-qb3al3g7/tmp/arena-relay-retry/candidate-2/ as `design.md` (rationale per the template, including caller usage, the type/signature sketch with `raise NotImplementedError` bodies, the stored-file format, CLI flags and defaults, and an end-to-end verification plan through the real CLI). Answer every open design question in the grounding. Keep it to one or two pages.\n\nReply with at most 12 lines: the shape in brief, your answer to each open question, and the file path. No preamble." + }, + "id": "toolu_015qrwepJ33JPD2xS4cUoP4o" + }, + { + "seq": 30, + "turn": 0, + "kind": "tool_call", + "name": "Agent", + "input": { + "description": "Architect runner 3 (sonnet)", + "subagent_type": "general-purpose", + "model": "sonnet", + "run_in_background": true, + "prompt": "You are candidate runner 3 in an architect design exploration. Read these files in full first, in this order:\n1. /private/tmp/pstack-collection-after-ipc-20261005/claude-code-after-ipc/relay-qb3al3g7/w/relay/.claude/skills/architect/SKILL.md (the workflow you're inside; you only do Phase B for one candidate)\n2. /private/tmp/pstack-collection-after-ipc-20261005/claude-code-after-ipc/relay-qb3al3g7/w/relay/.claude/skills/architect/references/runner-prompt.md (your instructions)\n3. /private/tmp/pstack-collection-after-ipc-20261005/claude-code-after-ipc/relay-qb3al3g7/w/relay/.claude/skills/architect/references/rationale-template.md and design-red-flags.md\n4. /private/tmp/pstack-collection-after-ipc-20261005/claude-code-after-ipc/relay-qb3al3g7/tmp/arena-relay-retry/grounding.md (the task and the traced subsystem)\nThen read the relay source files the grounding names (the repo is ~250 lines).\n\nRead-only on the repo: do not edit or write anything under the repo. Write your design package to /private/tmp/pstack-collection-after-ipc-20261005/claude-code-after-ipc/relay-qb3al3g7/tmp/arena-relay-retry/candidate-3/ as `design.md` (rationale per the template, including caller usage, the type/signature sketch with `raise NotImplementedError` bodies, the stored-file format, CLI flags and defaults, and an end-to-end verification plan through the real CLI). Answer every open design question in the grounding. Keep it to one or two pages.\n\nReply with at most 12 lines: the shape in brief, your answer to each open question, and the file path. No preamble." + }, + "id": "toolu_018tAPYCTZjALPprt8D9AR6r" + }, + { + "seq": 32, + "turn": 0, + "kind": "tool_call", + "name": "Agent", + "input": { + "description": "Architect runner 1 retry (session model)", + "subagent_type": "general-purpose", + "run_in_background": true, + "prompt": "You are candidate runner 1 in an architect design exploration. Read these files in full first, in this order:\n1. /private/tmp/pstack-collection-after-ipc-20261005/claude-code-after-ipc/relay-qb3al3g7/w/relay/.claude/skills/architect/SKILL.md (the workflow you're inside; you only do Phase B for one candidate)\n2. /private/tmp/pstack-collection-after-ipc-20261005/claude-code-after-ipc/relay-qb3al3g7/w/relay/.claude/skills/architect/references/runner-prompt.md (your instructions)\n3. /private/tmp/pstack-collection-after-ipc-20261005/claude-code-after-ipc/relay-qb3al3g7/w/relay/.claude/skills/architect/references/rationale-template.md and design-red-flags.md\n4. /private/tmp/pstack-collection-after-ipc-20261005/claude-code-after-ipc/relay-qb3al3g7/tmp/arena-relay-retry/grounding.md (the task and the traced subsystem)\nThen read the relay source files the grounding names (the repo is ~250 lines).\n\nAnother runner on your model is designing in parallel. To widen the design space, take the interpretation that `import` accepts several feeds per invocation, each source retried independently, and design the whole shape (types, per-source outcome, partial-failure semantics, exit status, stored format) around that. If you conclude that interpretation is wrong, say so with a reason and still sketch it as your alternative.\n\nRead-only on the repo: do not edit or write anything under the repo. Write your design package to /private/tmp/pstack-collection-after-ipc-20261005/claude-code-after-ipc/relay-qb3al3g7/tmp/arena-relay-retry/candidate-1/ as `design.md` (rationale per the template, including caller usage, the type/signature sketch with `raise NotImplementedError` bodies, the stored-file format, CLI flags and defaults, and an end-to-end verification plan through the real CLI). Answer every open design question in the grounding. Keep it to one or two pages.\n\nReply with at most 12 lines: the shape in brief, your answer to each open question, and the file path. No preamble." + }, + "id": "toolu_01GMyqerq3x2U8YSzfCTyNoS" + } + ], + "files_read": [], + "worklist": [], + "spawns": [ + { + "seq": 26, + "turn": 0, + "tool": "Agent", + "subagent_type": "general-purpose", + "persona": "general-purpose", + "model": "fable", + "effort": null, + "prompt_head": "You are candidate runner 1 in an architect design exploration. Read these files in full first, in this order:\n1. /private/tmp/pstack-collection-after-ipc-20261005/claude-code-after-ipc/relay-qb3al3g7/w/relay/.claude/skills/architect/SKILL.md (the workflow you're inside; you only do Phase B for one c" + }, + { + "seq": 28, + "turn": 0, + "tool": "Agent", + "subagent_type": "general-purpose", + "persona": "general-purpose", + "model": "opus", + "effort": null, + "prompt_head": "You are candidate runner 2 in an architect design exploration. Read these files in full first, in this order:\n1. /private/tmp/pstack-collection-after-ipc-20261005/claude-code-after-ipc/relay-qb3al3g7/w/relay/.claude/skills/architect/SKILL.md (the workflow you're inside; you only do Phase B for one c" + }, + { + "seq": 30, + "turn": 0, + "tool": "Agent", + "subagent_type": "general-purpose", + "persona": "general-purpose", + "model": "sonnet", + "effort": null, + "prompt_head": "You are candidate runner 3 in an architect design exploration. Read these files in full first, in this order:\n1. /private/tmp/pstack-collection-after-ipc-20261005/claude-code-after-ipc/relay-qb3al3g7/w/relay/.claude/skills/architect/SKILL.md (the workflow you're inside; you only do Phase B for one c" + }, + { + "seq": 32, + "turn": 0, + "tool": "Agent", + "subagent_type": "general-purpose", + "persona": "general-purpose", + "model": null, + "effort": null, + "prompt_head": "You are candidate runner 1 in an architect design exploration. Read these files in full first, in this order:\n1. /private/tmp/pstack-collection-after-ipc-20261005/claude-code-after-ipc/relay-qb3al3g7/w/relay/.claude/skills/architect/SKILL.md (the workflow you're inside; you only do Phase B for one c" + } + ], + "final_reply": "Failed to authenticate. API Error: 401 OAuth access token has expired. Re-authenticate to continue." +} \ No newline at end of file diff --git a/evals/promises/testdata/auth_death/grok-external-auth.stderr.txt b/evals/promises/testdata/auth_death/grok-external-auth.stderr.txt new file mode 100644 index 00000000..997b2955 --- /dev/null +++ b/evals/promises/testdata/auth_death/grok-external-auth.stderr.txt @@ -0,0 +1,28 @@ +Error: Internal error: { + "message": "Unauthorized (401) from https://cli-chat-proxy.grok.com/v1/responses: Invalid or expired credentials (auth_kind=none, x_xai_token_auth=xai-grok-cli, upstream=Unauthenticated, reason=no auth context)\n\n Model: grok-4.7\n Auth: External\n Version: 1.0.41\n Available: grok-4.7, grok-4.7-build-fast, grok-4.6, grok-4.5\n\nYour session expired and your sign-in helper could not renew it in the background. Run /login to sign in again.", + "promptUsage": { + "inputTokens": 1444814, + "outputTokens": 140967, + "totalTokens": 1585781, + "cachedReadTokens": 1104640, + "cacheCreationTokens": 0, + "reasoningTokens": 100924, + "modelCalls": 37, + "apiDurationMs": 1935728, + "costUsdTicks": 7066798000, + "modelUsage": { + "grok-4.7-build": { + "inputTokens": 1444814, + "outputTokens": 140967, + "totalTokens": 1585781, + "cachedReadTokens": 1104640, + "cacheCreationTokens": 0, + "reasoningTokens": 100924, + "modelCalls": 37, + "apiDurationMs": 1935728, + "costUsdTicks": 7066798000 + } + }, + "numTurns": 12 + } +} diff --git a/evals/promises/testdata/auth_death/grok-external-auth.stream.json b/evals/promises/testdata/auth_death/grok-external-auth.stream.json new file mode 100644 index 00000000..68b1f7db --- /dev/null +++ b/evals/promises/testdata/auth_death/grok-external-auth.stream.json @@ -0,0 +1 @@ +{"type":"error","message":"Internal error: {\n \"message\": \"Unauthorized (401) from https://cli-chat-proxy.grok.com/v1/responses: Invalid or expired credentials (auth_kind=none, x_xai_token_auth=xai-grok-cli, upstream=Unauthenticated, reason=no auth context)\\n\\n Model: grok-4.7\\n Auth: External\\n Version: 1.0.41\\n Available: grok-4.7, grok-4.7-build-fast, grok-4.6, grok-4.5\\n\\nYour session expired and your sign-in helper could not renew it in the background. Run /login to sign in again.\",\n \"promptUsage\": {\n \"inputTokens\": 1444814,\n \"outputTokens\": 140967,\n \"totalTokens\": 1585781,\n \"cachedReadTokens\": 1104640,\n \"cacheCreationTokens\": 0,\n \"reasoningTokens\": 100924,\n \"modelCalls\": 37,\n \"apiDurationMs\": 1935728,\n \"costUsdTicks\": 7066798000,\n \"modelUsage\": {\n \"grok-4.7-build\": {\n \"inputTokens\": 1444814,\n \"outputTokens\": 140967,\n \"totalTokens\": 1585781,\n \"cachedReadTokens\": 1104640,\n \"cacheCreationTokens\": 0,\n \"reasoningTokens\": 100924,\n \"modelCalls\": 37,\n \"apiDurationMs\": 1935728,\n \"costUsdTicks\": 7066798000\n }\n },\n \"numTurns\": 12\n }\n}","usage":{"input_tokens":340174,"cache_read_input_tokens":1104640,"cache_creation_input_tokens":0,"output_tokens":140967,"reasoning_tokens":100924,"total_tokens":1585781},"num_turns":12,"total_cost_usd":0.7066798,"total_cost_usd_ticks":7066798000,"modelUsage":{"grok-4.7-build":{"inputTokens":340174,"outputTokens":140967,"cacheReadInputTokens":1104640,"cacheCreationInputTokens":0,"modelCalls":37,"costUSD":0.7066798}}} From 05b90d075de04a57b6c6d0e8b083f604bdeb6105 Mon Sep 17 00:00:00 2001 From: mdsmithaustin <15272014+mdsmithaustin@users.noreply.github.com> Date: Mon, 12 Oct 2026 14:25:17 -0600 Subject: [PATCH 06/12] no-mistakes(document): Correct login fixture policy and preflight documentation --- evals/README.md | 6 +++--- evals/promises/harnesses/claude_code.py | 5 ++--- 2 files changed, 5 insertions(+), 6 deletions(-) diff --git a/evals/README.md b/evals/README.md index 073f17b5..4f52fc1f 100644 --- a/evals/README.md +++ b/evals/README.md @@ -10,9 +10,9 @@ each harness. See `promises/README.md`. Model answers, traces, judge transcripts, reports, and private cases stay outside Git. A committed case is tuning material, not sealed evidence. -Oracle regression fixtures under `promises/testdata/` are an exception. They -are reduced, sanitized traces that keep only what the oracle tests read, and -`promises/reduce_trace.py` rebuilds one from a raw trace. `promises/results/` +Oracle and login regression fixtures under `promises/testdata/` are an exception. +They are reduced, sanitized traces and CLI error records used by model-free tests. +`promises/reduce_trace.py` reduces raw traces for oracle fixtures. `promises/results/` is a second exception. It holds aggregated verdict matrices with no model answers. diff --git a/evals/promises/harnesses/claude_code.py b/evals/promises/harnesses/claude_code.py index a38147b1..1be5b39d 100644 --- a/evals/promises/harnesses/claude_code.py +++ b/evals/promises/harnesses/claude_code.py @@ -201,9 +201,8 @@ def store(run): def preflight(): - """Ask the CLI for its login status under the few variables a turn keeps, so an API key or config directory the host shell - exports cannot report a login the turns never see. The CLI refreshes an expired access token itself when it runs, so this - fails only when it reports no login.""" + """Check the host login before a private turn config exists. Keep the turn's host HOME and Keychain namespace, + but omit exported API keys and config overrides so they cannot mask a missing login.""" binary = CLAUDE_BINARY if CLAUDE_BINARY.is_file() else shutil.which("claude") if not binary: raise LoginRequired(LOGIN, "no claude binary to ask") From 6e082de636fc8113863e47c338ddc3a1ee47c075 Mon Sep 17 00:00:00 2001 From: mdsmithaustin <15272014+mdsmithaustin@users.noreply.github.com> Date: Mon, 12 Oct 2026 15:01:59 -0600 Subject: [PATCH 07/12] evals: build an auth-death verdict from its stored error record error_verdict now takes the error record a verdict.json carries, so a later re-grade can rebuild the verdict from the record alone. --- evals/promises/harnesses/login.py | 18 ++++++++++++------ evals/promises/live.py | 4 ++-- evals/promises/test_live.py | 8 ++++---- 3 files changed, 18 insertions(+), 12 deletions(-) diff --git a/evals/promises/harnesses/login.py b/evals/promises/harnesses/login.py index 7eb2f6f7..8ae251f9 100644 --- a/evals/promises/harnesses/login.py +++ b/evals/promises/harnesses/login.py @@ -61,18 +61,24 @@ def _read(path): return "" -def error_verdict(case, harness, skills_at, login, detail, graded=None): - """The verdict for a run a dead login ended. `graded` holds the grader's promise verdicts when turns ran before the - death. A verdict the grader decided (PASS or FAIL) stands, as does a promise that does not apply to the harness. Every - other promise is ERROR, so no skill result is claimed for it and report.py leaves it out of every pass rate.""" - reason = f"{login.cli} login died: {detail}" +def death_record(login, detail): + """The `error` record a verdict.json carries for a run a dead login ended.""" + return {"cli": login.cli, "relogin": login.relogin, "detail": detail} + + +def error_verdict(case, harness, skills_at, error, graded=None): + """The verdict for a run a dead login ended, from its `death_record`. `graded` holds the grader's promise verdicts when + turns ran before the death. A verdict the grader decided (PASS or FAIL) stands, as does a promise that does not apply + to the harness. Every other promise is ERROR, so no skill result is claimed for it and report.py leaves it out of + every pass rate.""" + reason = f"{error['cli']} login died: {error['detail']}" promises = {} for pid in case["promises"]: result = graded[pid] if graded else _applicability(pid, harness, case) keep = result.get("not_applicable") or (graded and result["verdict"] in DECIDED) promises[pid] = result if keep else {"verdict": ERROR, "failures": [reason], "evidence": []} return {"case": case["id"], "harness": harness, "skills_at": skills_at, - "error": {"cli": login.cli, "relogin": login.relogin, "detail": detail}, "promises": promises} + "error": error, "promises": promises} def _applicability(pid, harness, case): diff --git a/evals/promises/live.py b/evals/promises/live.py index 66ffe27c..1d6edd63 100644 --- a/evals/promises/live.py +++ b/evals/promises/live.py @@ -18,7 +18,7 @@ from pathlib import Path from grade_boundary import create_file, grade, GradeRefused, read_file, write_file, _before_turns, _json, _lookup, _seal, _write_record -from harnesses.login import LoginRequired, auth_death, error_verdict +from harnesses.login import LoginRequired, auth_death, death_record, error_verdict HERE = Path(__file__).resolve().parent ROOT = HERE.parents[1] @@ -306,7 +306,7 @@ def run_case(harness, case_id, skills_at, out, index, hermes_retain_out=None, mo graded = grade(authorization) verdict = graded if died: - verdict = error_verdict(case, harness, commit, *died, graded and graded["promises"]) + verdict = error_verdict(case, harness, commit, death_record(*died), graded and graded["promises"]) write_file(root / "verdict.json", _json(verdict)) except BaseException as original: owner = getattr(run, "_hermes_evidence", None) diff --git a/evals/promises/test_live.py b/evals/promises/test_live.py index 29f58271..0d9b5227 100644 --- a/evals/promises/test_live.py +++ b/evals/promises/test_live.py @@ -15,7 +15,7 @@ import live from grade_boundary import GradeRefused, _authorize_fixture -from harnesses.login import LoginRequired, auth_death, error_verdict +from harnesses.login import LoginRequired, auth_death, death_record, error_verdict FIXTURES = Path(__file__).resolve().parent / "testdata" / "auth_death" @@ -602,7 +602,7 @@ def test_pass_and_fail_survive_an_auth_death_and_only_the_undecided_become_error "b": {"verdict": "FAIL", "failures": ["f"], "evidence": []}, "c": {"verdict": "INCONCLUSIVE", "failures": ["run killed"], "evidence": []}, "d": {"verdict": "INCONCLUSIVE", "failures": ["applies elsewhere"], "evidence": [], "not_applicable": True}} - verdict = error_verdict(case, "codex", "0" * 40, codex.LOGIN, "turn 0 exited 1", graded) + verdict = error_verdict(case, "codex", "0" * 40, death_record(codex.LOGIN, "turn 0 exited 1"), graded) self.assertEqual({p: r["verdict"] for p, r in verdict["promises"].items()}, {"a": "PASS", "b": "FAIL", "c": "ERROR", "d": "INCONCLUSIVE"}) self.assertEqual(verdict["promises"]["a"], graded["a"]) @@ -614,7 +614,7 @@ def test_a_dead_codex_run_marks_error_only_the_promises_that_apply_to_codex(self import report from harnesses import codex case = live.load_case("feature-run") - verdict = error_verdict(case, "codex", "0" * 40, codex.LOGIN, "no login at auth.json") + verdict = error_verdict(case, "codex", "0" * 40, death_record(codex.LOGIN, "no login at auth.json")) gated = {pid for pid, r in verdict["promises"].items() if r.get("not_applicable")} self.assertEqual(gated, {"claude-effort-applies-via-effort-agents", "persona-delivery-hermes-grok"}) self.assertEqual({r["verdict"] for pid, r in verdict["promises"].items() if pid not in gated}, {"ERROR"}) @@ -634,7 +634,7 @@ def test_every_promise_of_every_live_case_answers_the_applicability_probe_on_eve continue for harness in report.HARNESSES: with self.subTest(case=case_id, harness=harness): - verdict = error_verdict(case, harness, "0" * 40, codex.LOGIN, "probe") + verdict = error_verdict(case, harness, "0" * 40, death_record(codex.LOGIN, "probe")) self.assertEqual(set(verdict["promises"]), set(case["promises"])) From beca8837dd6bd43a33bb0961c0f9dd6b032c3c08 Mon Sep 17 00:00:00 2001 From: mdsmithaustin <15272014+mdsmithaustin@users.noreply.github.com> Date: Mon, 12 Oct 2026 15:03:07 -0600 Subject: [PATCH 08/12] evals: red tests for the round-2 auth-death verdict gaps Recorded shapes: the 2026-10-05 Claude feature-run that died on a 401 before any tool call, and the Hermes feature-run whose transcript read was refused. The tests cover harness-gated promises marked ERROR, gated promises counted in the pass rate, a re-grade that drops the ERROR marks, and an ERROR that drops needs_judge. --- evals/promises/test_live.py | 83 ++++++++++++++++++ evals/promises/test_report.py | 11 +++ .../claude-code-never-started-trace.json | 27 ++++++ .../hermes-feature-run-verdict.json | 84 +++++++++++++++++++ 4 files changed, 205 insertions(+) create mode 100644 evals/promises/testdata/auth_death/claude-code-never-started-trace.json create mode 100644 evals/promises/testdata/harvest_refused/hermes-feature-run-verdict.json diff --git a/evals/promises/test_live.py b/evals/promises/test_live.py index 0d9b5227..2e31fae0 100644 --- a/evals/promises/test_live.py +++ b/evals/promises/test_live.py @@ -593,6 +593,62 @@ def turn(run, text, index): self.assertEqual(json.loads(output.getvalue())["verdicts"], {promise: "PASS"}) self.assertIn("claude-code pass rate: 1 of 1", table) + def claude_feature_run_that_never_started(self, out): + from harnesses import claude_code + case = live.load_case("feature-run") + case["tmp_lock"] = False + trace = json.loads((FIXTURES / "claude-code-never-started-trace.json").read_text()) + + def turn(run, text, index): + (run.root / "err").write_bytes(b"") + return {"index": index, "session_id": "s", "argv": ["claude", text], "exit_code": 1, "timed_out": False, + "duration_s": 4.7, "stream": str(FIXTURES / "claude-code-401-last-line.jsonl"), "stderr": str(run.root / "err")} + + output = io.StringIO() + with mock.patch.object(live, "load_case", return_value=case), mock.patch.object(claude_code, "prepare"), \ + mock.patch.object(claude_code, "turn", side_effect=turn), \ + mock.patch.object(claude_code, "harvest", return_value=trace), redirect_stdout(output): + with self.assertRaisesRegex(live.LoginRequired, "Run `claude auth login`"): + live.run_case("claude-code", case["id"], "HEAD", out, 0) + return json.loads(output.getvalue())["authorization_id"] + + def test_regrading_a_run_a_dead_login_ended_keeps_its_error_marks(self): + import report + with tempfile.TemporaryDirectory(prefix="pstack-live-regrade-") as tmp: + out = Path(tmp).resolve() + run_id = self.claude_feature_run_that_never_started(out) + with redirect_stdout(io.StringIO()): + code = live.main(["grade", run_id, "--out", str(out)]) + [root] = [p.parent for p in out.rglob("verdict.json")] + verdict = json.loads((root / "verdict.json").read_text()) + table = report.render(out, upstream=out) + self.assertEqual(code, 0) + self.assertEqual(verdict["error"]["relogin"], "claude auth login") + self.assertEqual({p: r["verdict"] for p, r in verdict["promises"].items() if not r.get("not_applicable")}, + {p: "ERROR" for p in verdict["promises"] if p != "persona-delivery-hermes-grok"}) + self.assertIn("claude-code pass rate: none graded (ERROR 10 excluded)", table) + + def test_regrading_a_run_whose_login_died_before_any_turn_is_refused_and_keeps_its_verdict(self): + from harnesses import codex + case = live.load_case("principle-steer-run") + case["tmp_lock"] = False + with tempfile.TemporaryDirectory(prefix="pstack-live-regrade-") as tmp: + out = Path(tmp).resolve() + output = io.StringIO() + with mock.patch.object(live, "load_case", return_value=case), redirect_stdout(output), \ + mock.patch.object(codex, "prepare", side_effect=LoginRequired(codex.LOGIN, "no login at auth.json")): + with self.assertRaises(live.LoginRequired): + live.run_case("codex", case["id"], "HEAD", out, 0) + [root] = [p.parent for p in out.rglob("verdict.json")] + before = (root / "verdict.json").read_bytes() + with redirect_stderr(io.StringIO()) as refusal: + code = live.main(["grade", json.loads(output.getvalue())["authorization_id"], "--out", str(out)]) + after = (root / "verdict.json").read_bytes() + self.assertEqual(code, 2) + self.assertIn("not sealed", refusal.getvalue()) + self.assertEqual(after, before) + self.assertEqual({r["verdict"] for r in json.loads(after)["promises"].values()}, {"ERROR"}) + class ErrorVerdict(unittest.TestCase): def test_pass_and_fail_survive_an_auth_death_and_only_the_undecided_become_error(self): @@ -610,6 +666,33 @@ def test_pass_and_fail_survive_an_auth_death_and_only_the_undecided_become_error self.assertEqual(verdict["promises"]["c"]["failures"], ["codex login died: turn 0 exited 1"]) self.assertEqual(verdict["error"]["relogin"], "codex login") + def test_a_claude_run_that_never_started_keeps_the_hermes_and_grok_promise_out(self): + import oracles + import report + from harnesses import claude_code + case = live.load_case("feature-run") + trace = json.loads((FIXTURES / "claude-code-never-started-trace.json").read_text()) + graded = {pid: oracles.check(pid, trace, case, None) for pid in case["promises"]} + self.assertIn("run never started", graded["persona-delivery-hermes-grok"]["failures"][0]) + verdict = error_verdict(case, "claude-code", "0" * 40, death_record(claude_code.LOGIN, "turn 0 exited 1"), graded) + gated = {pid for pid, r in verdict["promises"].items() if r.get("not_applicable")} + self.assertEqual(gated, {"persona-delivery-hermes-grok"}) + self.assertEqual({r["verdict"] for pid, r in verdict["promises"].items() if pid not in gated}, {"ERROR"}) + with tempfile.TemporaryDirectory(prefix="pstack-error-verdict-") as tmp: + (Path(tmp) / "verdict.json").write_text(json.dumps(verdict)) + table = report.render(Path(tmp), upstream=Path(tmp)) + self.assertIn("claude-code pass rate: none graded (ERROR 10 excluded)", table) + self.assertNotIn("persona-delivery-hermes-grok", table) + + def test_an_undecided_promise_that_needs_a_judge_keeps_the_flag_and_excerpt_on_its_error(self): + import oracles + from harnesses import codex + case = {"id": "x", "promises": ["a"]} + graded = {"a": oracles.inconclusive("wording needs a judge", "seq 4", needs_judge=True, excerpt="the reply says pass")} + verdict = error_verdict(case, "codex", "0" * 40, death_record(codex.LOGIN, "turn 1 exited 1"), graded) + self.assertEqual(verdict["promises"]["a"], {"verdict": "ERROR", "failures": ["codex login died: turn 1 exited 1"], "evidence": [], + "needs_judge": True, "excerpt": "the reply says pass"}) + def test_a_dead_codex_run_marks_error_only_the_promises_that_apply_to_codex(self): import report from harnesses import codex diff --git a/evals/promises/test_report.py b/evals/promises/test_report.py index 7cc0b848..a4c74ffc 100644 --- a/evals/promises/test_report.py +++ b/evals/promises/test_report.py @@ -45,6 +45,17 @@ def test_an_error_run_drops_out_of_the_pass_rate_and_keeps_its_cell(self): self.assertIn("| `p1` | ERROR 1, PASS 1 | ERROR |", table) self.assertIn("ERROR 3, FAIL 1, INCONCLUSIVE 1, PASS 1", table) + def test_a_run_whose_transcript_could_not_be_read_counts_only_the_promises_that_apply(self): + recorded = Path(__file__).resolve().parent / "testdata" / "harvest_refused" / "hermes-feature-run-verdict.json" + with tempfile.TemporaryDirectory() as out: + run = Path(out) / "tally" + run.mkdir() + (run / "verdict.json").write_bytes(recorded.read_bytes()) + table = report.render(Path(out), upstream=Path(out)) + self.assertIn("hermes pass rate: 0 of 10\n", table + "\n") + self.assertIn("| `persona-delivery-hermes-grok` | | | INCONCLUSIVE | | |", table) + self.assertNotIn("claude-effort-applies-via-effort-agents", table) + def test_a_harness_with_no_error_prints_a_plain_pass_rate(self): with tempfile.TemporaryDirectory() as out: self.run_dir(out, "a", "grok", {"p1": "PASS", "p2": "PASS", "p3": "FAIL"}) diff --git a/evals/promises/testdata/auth_death/claude-code-never-started-trace.json b/evals/promises/testdata/auth_death/claude-code-never-started-trace.json new file mode 100644 index 00000000..510352d4 --- /dev/null +++ b/evals/promises/testdata/auth_death/claude-code-never-started-trace.json @@ -0,0 +1,27 @@ +{ + "harness": "claude-code", + "cli_version": "2.1.289", + "model": "claude-opus-5-5", + "effort": "high", + "exit_code": 1, + "duration_s": 4.7, + "entry": "injected", + "events": [ + { + "seq": 0, + "turn": 0, + "kind": "user", + "text": "/poteto-mode add a --json flag to this command. text output stays byte-identical. verify both." + }, + { + "seq": 1, + "turn": 0, + "kind": "text", + "text": "Failed to authenticate. API Error: 401 OAuth access token has expired. Re-authenticate to continue." + } + ], + "files_read": [], + "worklist": [], + "spawns": [], + "final_reply": "Failed to authenticate. API Error: 401 OAuth access token has expired. Re-authenticate to continue." +} diff --git a/evals/promises/testdata/harvest_refused/hermes-feature-run-verdict.json b/evals/promises/testdata/harvest_refused/hermes-feature-run-verdict.json new file mode 100644 index 00000000..8b078bdf --- /dev/null +++ b/evals/promises/testdata/harvest_refused/hermes-feature-run-verdict.json @@ -0,0 +1,84 @@ +{ + "case": "feature-run", + "harness": "hermes", + "skills_at": "57e4409a6c4ebb7dfd27a5118caf478ef64bd769", + "promises": { + "poteto-mode-routes-to-playbook": { + "verdict": "INCONCLUSIVE", + "failures": [ + "the harness transcript could not be read: path-refused: recorded read crosses an unavailable fixture prefix" + ], + "evidence": [] + }, + "worklist-opens-with-playbook-steps": { + "verdict": "INCONCLUSIVE", + "failures": [ + "the harness transcript could not be read: path-refused: recorded read crosses an unavailable fixture prefix" + ], + "evidence": [] + }, + "worklist-in-native-task-tool": { + "verdict": "INCONCLUSIVE", + "failures": [ + "the harness transcript could not be read: path-refused: recorded read crosses an unavailable fixture prefix" + ], + "evidence": [] + }, + "skipped-step-stays-in-worklist": { + "verdict": "INCONCLUSIVE", + "failures": [ + "the harness transcript could not be read: path-refused: recorded read crosses an unavailable fixture prefix" + ], + "evidence": [] + }, + "reply-carries-commands-and-outputs": { + "verdict": "INCONCLUSIVE", + "failures": [ + "the harness transcript could not be read: path-refused: recorded read crosses an unavailable fixture prefix" + ], + "evidence": [] + }, + "reply-says-inconclusive-when-check-cannot-run": { + "verdict": "INCONCLUSIVE", + "failures": [ + "the harness transcript could not be read: path-refused: recorded read crosses an unavailable fixture prefix" + ], + "evidence": [] + }, + "prove-it-works-checks-real-artifact": { + "verdict": "INCONCLUSIVE", + "failures": [ + "the harness transcript could not be read: path-refused: recorded read crosses an unavailable fixture prefix" + ], + "evidence": [] + }, + "claude-effort-applies-via-effort-agents": { + "verdict": "INCONCLUSIVE", + "failures": [ + "the harness transcript could not be read: path-refused: recorded read crosses an unavailable fixture prefix" + ], + "evidence": [] + }, + "persona-delivery-hermes-grok": { + "verdict": "INCONCLUSIVE", + "failures": [ + "the harness transcript could not be read: path-refused: recorded read crosses an unavailable fixture prefix" + ], + "evidence": [] + }, + "poteto-mode-reads-principles-and-names-applied": { + "verdict": "INCONCLUSIVE", + "failures": [ + "the harness transcript could not be read: path-refused: recorded read crosses an unavailable fixture prefix" + ], + "evidence": [] + }, + "design-ladder-spares-small-changes": { + "verdict": "INCONCLUSIVE", + "failures": [ + "the harness transcript could not be read: path-refused: recorded read crosses an unavailable fixture prefix" + ], + "evidence": [] + } + } +} From ad95f19648ff6e1751a6de4f20004126b343671f Mon Sep 17 00:00:00 2001 From: mdsmithaustin <15272014+mdsmithaustin@users.noreply.github.com> Date: Mon, 12 Oct 2026 15:05:26 -0600 Subject: [PATCH 09/12] evals: keep the auth-death verdict through early returns and re-grades - error_verdict asks each oracle about an empty trace from the harness, so a run that never started no longer marks a harness-gated promise ERROR. - report.py drops a promise that does not apply to the harness even when its grade returned early, so a refused Hermes transcript reads 0 of 10, not 0 of 11. - live.py grade keeps a dead login's error record and marks every promise the new grade leaves undecided ERROR again. run_case grades through the same path. - An ERROR that replaces an INCONCLUSIVE keeps its needs_judge flag and excerpt. --- evals/promises/README.md | 2 +- evals/promises/harnesses/login.py | 31 +++++++++++++++++++------------ evals/promises/live.py | 28 +++++++++++++++++++++------- evals/promises/report.py | 4 ++-- evals/promises/test_live.py | 9 +++++---- 5 files changed, 48 insertions(+), 26 deletions(-) diff --git a/evals/promises/README.md b/evals/promises/README.md index f8cbb60f..cc31819f 100644 --- a/evals/promises/README.md +++ b/evals/promises/README.md @@ -75,7 +75,7 @@ Each run makes a fresh git repository from `fixtures//` and `histories/ of ` line. This rate counts applicable promise verdicts, not runs. `INCONCLUSIVE` stays in the denominator. When every applicable verdict is `ERROR`, the rate reads `none graded`. A turn that exited 0 still grades as before, even if it contains login-failure text. +For Claude Code, Codex, and Grok, a turn counts as an auth death when it exited nonzero without timing out and the tail of its stderr or the last nonempty line of its stream carries that CLI's login-failure text. A `LoginRequired` failure during preparation stops the run before any turn. `live.py` writes a `verdict.json` that marks every promise that applies to the harness `ERROR` and records the CLI and its login command under `error`. It skips harvest and grading, so that run has no `trace.json`. When a turn dies on auth, `live.py` still harvests and grades the run. A `PASS` or `FAIL` the grader decides stands, and every other promise becomes `ERROR`, keeping any `needs_judge` flag and `excerpt`. Running `live.py grade` on that run later re-grades it from its trace, keeps the `error` record, and marks `ERROR` again on every promise the new grade leaves undecided. A run whose login died before any turn has no sealed trace, so `live.py grade` refuses it with exit 2 and leaves its verdict unchanged. In each of these verdicts a promise that does not apply to the harness is marked `not_applicable`, so it stays out of the report, and the wave stops with exit 3 after the run. `live.py report` shows `ERROR` in the table and leaves those verdicts out of each harness's `pass rate: of ` line. This rate counts applicable promise verdicts, not runs. `live.py` and `live.py report` decide applicability by asking each oracle about an empty trace from the harness, because a grade that returns early carries no `not_applicable` mark. A grade returns early when the run never started, its transcript could not be read, or host skills touched it. `INCONCLUSIVE` stays in the denominator. When every applicable verdict is `ERROR`, the rate reads `none graded`. A turn that exited 0 still grades as before, even if it contains login-failure text. `--model` and `--effort` set the model and reasoning effort for every turn of every case in one `run` invocation, on Claude Code and Codex. Each turn's recorded argv carries both values, and the sealed run metadata keeps its fields. Other adapters refuse the flags. diff --git a/evals/promises/harnesses/login.py b/evals/promises/harnesses/login.py index 8ae251f9..aea1b102 100644 --- a/evals/promises/harnesses/login.py +++ b/evals/promises/harnesses/login.py @@ -70,19 +70,26 @@ def error_verdict(case, harness, skills_at, error, graded=None): """The verdict for a run a dead login ended, from its `death_record`. `graded` holds the grader's promise verdicts when turns ran before the death. A verdict the grader decided (PASS or FAIL) stands, as does a promise that does not apply to the harness. Every other promise is ERROR, so no skill result is claimed for it and report.py leaves it out of - every pass rate.""" + every pass rate. An ERROR keeps the judge flag and excerpt of the INCONCLUSIVE it replaces.""" reason = f"{error['cli']} login died: {error['detail']}" promises = {} for pid in case["promises"]: - result = graded[pid] if graded else _applicability(pid, harness, case) - keep = result.get("not_applicable") or (graded and result["verdict"] in DECIDED) - promises[pid] = result if keep else {"verdict": ERROR, "failures": [reason], "evidence": []} - return {"case": case["id"], "harness": harness, "skills_at": skills_at, - "error": error, "promises": promises} - - -def _applicability(pid, harness, case): - """The oracle's own answer on a run with no trace. Only its `not_applicable` flag is read, so a promise that never - applies to this harness stays blank instead of turning ERROR.""" + result = graded[pid] if graded else {} + gate = applicability(pid, harness) + if gate.get("not_applicable"): + promises[pid] = gate + elif result.get("verdict") in DECIDED: + promises[pid] = result + else: + judge = {k: result[k] for k in ("needs_judge", "excerpt") if k in result} + promises[pid] = {"verdict": ERROR, "failures": [reason], "evidence": [], **judge} + return {"case": case["id"], "harness": harness, "skills_at": skills_at, "error": error, "promises": promises} + + +def applicability(pid, harness): + """The oracle's answer on an empty trace from this harness. A harness-gated oracle marks itself `not_applicable` before + it reads the trace, but oracles.check returns before any oracle runs when a run never started, its transcript could + not be read, or host skills touched it, and that answer carries no flag. The empty trace reaches the gate however the + real grade returned.""" import oracles - return oracles.check(pid, {"harness": harness}, case, None) + return oracles.check(pid, {"harness": harness}, None, None) diff --git a/evals/promises/live.py b/evals/promises/live.py index 1d6edd63..4fbc5859 100644 --- a/evals/promises/live.py +++ b/evals/promises/live.py @@ -299,15 +299,13 @@ def run_case(harness, case_id, skills_at, out, index, hermes_retain_out=None, mo pair = None try: died = drive(module, run, authorization) - graded = None + if died: + verdict = error_verdict(case, harness, commit, death_record(*died)) + write_file(root / "verdict.json", _json(verdict)) if run.turns or not died: trace = {**module.harvest(run), "x_implicit_off": run.implicit_off} _seal(authorization, meta(run), trace) - graded = grade(authorization) - verdict = graded - if died: - verdict = error_verdict(case, harness, commit, death_record(*died), graded and graded["promises"]) - write_file(root / "verdict.json", _json(verdict)) + verdict = grade_run(authorization) except BaseException as original: owner = getattr(run, "_hermes_evidence", None) if owner is not None and retention is not None: @@ -337,6 +335,22 @@ def run_case(harness, case_id, skills_at, out, index, hermes_retain_out=None, mo return root +def grade_run(authorization): + """Grade a sealed run. When its verdict.json records a dead login under `error`, the new grade keeps that record and + every promise it leaves undecided goes back to ERROR, so a re-grade claims no skill result the run never produced.""" + _, state = authorization._controller.load(authorization._id) + target = Path(state["output"]["path"]) / "verdict.json" + try: + error = json.loads(read_file(target)).get("error") + except FileNotFoundError: + error = None + verdict = grade(authorization) + if error: + verdict = error_verdict(state["case"], verdict["harness"], verdict["skills_at"], error, verdict["promises"]) + write_file(target, _json(verdict)) + return verdict + + def meta(run): return {"harness": run.harness, "case": run.case["id"], "skills_at": run.skills_at, "project": str(run.project), "timeout_s": run.timeout_s, "turns": run.turns, "baseline": run.baseline} @@ -377,7 +391,7 @@ def main(argv=None): if args.cmd == "grade": for run_id in args.run_ids: try: - print(json.dumps(grade(_lookup(args.out, run_id)))) + print(json.dumps(grade_run(_lookup(args.out, run_id)))) except GradeRefused as refusal: print(json.dumps({"refusal": refusal.receipt}), file=sys.stderr) return 2 diff --git a/evals/promises/report.py b/evals/promises/report.py index 380484a1..00bff47d 100644 --- a/evals/promises/report.py +++ b/evals/promises/report.py @@ -3,7 +3,7 @@ from pathlib import Path import ledger -from harnesses.login import ERROR +from harnesses.login import ERROR, applicability HARNESSES = ("claude-code", "codex", "hermes", "grok") @@ -13,7 +13,7 @@ def collect(out): for path in sorted(Path(out).rglob("verdict.json")): record = json.loads(path.read_text()) for pid, result in record["promises"].items(): - if not result.get("not_applicable"): + if not result.get("not_applicable") and not applicability(pid, record["harness"]).get("not_applicable"): cells[(pid, record["harness"])].append((result["verdict"], str(path.parent))) for path in sorted(Path(out).rglob("install.json")): for row in json.loads(path.read_text()): diff --git a/evals/promises/test_live.py b/evals/promises/test_live.py index 2e31fae0..862c9334 100644 --- a/evals/promises/test_live.py +++ b/evals/promises/test_live.py @@ -653,16 +653,17 @@ def test_regrading_a_run_whose_login_died_before_any_turn_is_refused_and_keeps_i class ErrorVerdict(unittest.TestCase): def test_pass_and_fail_survive_an_auth_death_and_only_the_undecided_become_error(self): from harnesses import codex - case = {"id": "x", "promises": ["a", "b", "c", "d"]} + gated = "persona-delivery-hermes-grok" + case = {"id": "x", "promises": ["a", "b", "c", gated]} graded = {"a": {"verdict": "PASS", "failures": [], "evidence": ["e"]}, "b": {"verdict": "FAIL", "failures": ["f"], "evidence": []}, "c": {"verdict": "INCONCLUSIVE", "failures": ["run killed"], "evidence": []}, - "d": {"verdict": "INCONCLUSIVE", "failures": ["applies elsewhere"], "evidence": [], "not_applicable": True}} + gated: {"verdict": "INCONCLUSIVE", "failures": ["applies to hermes and grok only"], "evidence": [], "not_applicable": True}} verdict = error_verdict(case, "codex", "0" * 40, death_record(codex.LOGIN, "turn 0 exited 1"), graded) self.assertEqual({p: r["verdict"] for p, r in verdict["promises"].items()}, - {"a": "PASS", "b": "FAIL", "c": "ERROR", "d": "INCONCLUSIVE"}) + {"a": "PASS", "b": "FAIL", "c": "ERROR", gated: "INCONCLUSIVE"}) self.assertEqual(verdict["promises"]["a"], graded["a"]) - self.assertTrue(verdict["promises"]["d"]["not_applicable"]) + self.assertTrue(verdict["promises"][gated]["not_applicable"]) self.assertEqual(verdict["promises"]["c"]["failures"], ["codex login died: turn 0 exited 1"]) self.assertEqual(verdict["error"]["relogin"], "codex login") From 88eab78e1f61e7d7cb6c18d34e65edfb37f575fc Mon Sep 17 00:00:00 2001 From: mdsmithaustin <15272014+mdsmithaustin@users.noreply.github.com> Date: Mon, 12 Oct 2026 15:12:31 -0600 Subject: [PATCH 10/12] evals: drop the auth-death docstrings and name grade_run in the grade docs --- evals/promises/README.md | 2 +- evals/promises/harnesses/login.py | 9 --------- evals/promises/live.py | 2 -- 3 files changed, 1 insertion(+), 12 deletions(-) diff --git a/evals/promises/README.md b/evals/promises/README.md index cc31819f..e3b61805 100644 --- a/evals/promises/README.md +++ b/evals/promises/README.md @@ -107,7 +107,7 @@ A completed run prints `authorization_id`. To regrade that registered run, use t python3 evals/promises/live.py grade --out ``` -The grade CLI exits 2 and writes a refusal receipt to stderr when the ID is unknown or grading refuses. A successful grade returns the existing verdict dictionary and writes `verdict.json`. It preserves the JSON indentation and final newline. The Python entry point is `live.grade(authorization)`. It raises `GradeRefused` for infrastructure and unsafe-I/O refusals. A refusal has no promise verdict. Its separate receipt and worker logs remain under the controller's run and attempt directories. +The grade CLI exits 2 and writes a refusal receipt to stderr when the ID is unknown or grading refuses. A successful grade returns the existing verdict dictionary and writes `verdict.json`. It preserves the JSON indentation and final newline. The Python entry point is `live.grade(authorization)`. The CLI calls it through `live.grade_run(authorization)`, which also keeps a dead login's `error` record and its `ERROR` marks, as [Run live cases](#run-live-cases) describes. It raises `GradeRefused` for infrastructure and unsafe-I/O refusals. A refusal has no promise verdict. Its separate receipt and worker logs remain under the controller's run and attempt directories. Missing sealed `run.json`, `trace.json`, or retained prior `verdict.json` produces an `input_changed` refusal and an attempt receipt. The refusal leaves any existing output verdict unchanged. An intentionally absent project keeps its oracle result, and an absent output verdict still permits first publication. diff --git a/evals/promises/harnesses/login.py b/evals/promises/harnesses/login.py index aea1b102..52365c38 100644 --- a/evals/promises/harnesses/login.py +++ b/evals/promises/harnesses/login.py @@ -62,15 +62,10 @@ def _read(path): def death_record(login, detail): - """The `error` record a verdict.json carries for a run a dead login ended.""" return {"cli": login.cli, "relogin": login.relogin, "detail": detail} def error_verdict(case, harness, skills_at, error, graded=None): - """The verdict for a run a dead login ended, from its `death_record`. `graded` holds the grader's promise verdicts when - turns ran before the death. A verdict the grader decided (PASS or FAIL) stands, as does a promise that does not apply - to the harness. Every other promise is ERROR, so no skill result is claimed for it and report.py leaves it out of - every pass rate. An ERROR keeps the judge flag and excerpt of the INCONCLUSIVE it replaces.""" reason = f"{error['cli']} login died: {error['detail']}" promises = {} for pid in case["promises"]: @@ -87,9 +82,5 @@ def error_verdict(case, harness, skills_at, error, graded=None): def applicability(pid, harness): - """The oracle's answer on an empty trace from this harness. A harness-gated oracle marks itself `not_applicable` before - it reads the trace, but oracles.check returns before any oracle runs when a run never started, its transcript could - not be read, or host skills touched it, and that answer carries no flag. The empty trace reaches the gate however the - real grade returned.""" import oracles return oracles.check(pid, {"harness": harness}, None, None) diff --git a/evals/promises/live.py b/evals/promises/live.py index 4fbc5859..b7f786b8 100644 --- a/evals/promises/live.py +++ b/evals/promises/live.py @@ -336,8 +336,6 @@ def run_case(harness, case_id, skills_at, out, index, hermes_retain_out=None, mo def grade_run(authorization): - """Grade a sealed run. When its verdict.json records a dead login under `error`, the new grade keeps that record and - every promise it leaves undecided goes back to ERROR, so a re-grade claims no skill result the run never produced.""" _, state = authorization._controller.load(authorization._id) target = Path(state["output"]["path"]) / "verdict.json" try: From 74e702a43ffaeb20fc7102e46ced4ca84fb0d1cc Mon Sep 17 00:00:00 2001 From: mdsmithaustin <15272014+mdsmithaustin@users.noreply.github.com> Date: Mon, 12 Oct 2026 15:16:51 -0600 Subject: [PATCH 11/12] evals: grade an auth-death run before writing its verdict A harvest refusal after an auth death again stops the batch before the run gets a verdict.json, as the README says. Also moved the exit 3 sentence to the live-run sentences, and named recorded verdicts among the testdata fixtures. --- evals/README.md | 2 +- evals/promises/README.md | 2 +- evals/promises/live.py | 20 +++++++++++--------- evals/promises/test_live.py | 20 ++++++++++++++++++++ 4 files changed, 33 insertions(+), 11 deletions(-) diff --git a/evals/README.md b/evals/README.md index 4f52fc1f..1e117840 100644 --- a/evals/README.md +++ b/evals/README.md @@ -11,7 +11,7 @@ each harness. See `promises/README.md`. Model answers, traces, judge transcripts, reports, and private cases stay outside Git. A committed case is tuning material, not sealed evidence. Oracle and login regression fixtures under `promises/testdata/` are an exception. -They are reduced, sanitized traces and CLI error records used by model-free tests. +They are reduced, sanitized traces, CLI error records, and recorded verdicts used by model-free tests. `promises/reduce_trace.py` reduces raw traces for oracle fixtures. `promises/results/` is a second exception. It holds aggregated verdict matrices with no model answers. diff --git a/evals/promises/README.md b/evals/promises/README.md index e3b61805..05811863 100644 --- a/evals/promises/README.md +++ b/evals/promises/README.md @@ -75,7 +75,7 @@ Each run makes a fresh git repository from `fixtures//` and `histories/ of ` line. This rate counts applicable promise verdicts, not runs. `live.py` and `live.py report` decide applicability by asking each oracle about an empty trace from the harness, because a grade that returns early carries no `not_applicable` mark. A grade returns early when the run never started, its transcript could not be read, or host skills touched it. `INCONCLUSIVE` stays in the denominator. When every applicable verdict is `ERROR`, the rate reads `none graded`. A turn that exited 0 still grades as before, even if it contains login-failure text. +For Claude Code, Codex, and Grok, a turn counts as an auth death when it exited nonzero without timing out and the tail of its stderr or the last nonempty line of its stream carries that CLI's login-failure text. A `LoginRequired` failure during preparation stops the run before any turn. `live.py` writes a `verdict.json` that marks every promise that applies to the harness `ERROR` and records the CLI and its login command under `error`. It skips harvest and grading, so that run has no `trace.json`. When a turn dies on auth, `live.py` still harvests and grades the run. A `PASS` or `FAIL` the grader decides stands, and every other promise becomes `ERROR`, keeping any `needs_judge` flag and `excerpt`. Either way the wave stops with exit 3 after the run. Running `live.py grade` on that run later re-grades it from its trace, keeps the `error` record, and marks `ERROR` again on every promise the new grade leaves undecided. A run whose login died before any turn has no sealed trace, so `live.py grade` refuses it with exit 2 and leaves its verdict unchanged. In each of these verdicts a promise that does not apply to the harness is marked `not_applicable`, so it stays out of the report. `live.py report` shows `ERROR` in the table and leaves those verdicts out of each harness's `pass rate: of ` line. This rate counts applicable promise verdicts, not runs. `live.py` and `live.py report` decide applicability by asking each oracle about an empty trace from the harness, because a grade that returns early carries no `not_applicable` mark. A grade returns early when the run never started, its transcript could not be read, or host skills touched it. `INCONCLUSIVE` stays in the denominator. When every applicable verdict is `ERROR`, the rate reads `none graded`. A turn that exited 0 still grades as before, even if it contains login-failure text. `--model` and `--effort` set the model and reasoning effort for every turn of every case in one `run` invocation, on Claude Code and Codex. Each turn's recorded argv carries both values, and the sealed run metadata keeps its fields. Other adapters refuse the flags. diff --git a/evals/promises/live.py b/evals/promises/live.py index b7f786b8..76be5535 100644 --- a/evals/promises/live.py +++ b/evals/promises/live.py @@ -299,13 +299,14 @@ def run_case(harness, case_id, skills_at, out, index, hermes_retain_out=None, mo pair = None try: died = drive(module, run, authorization) - if died: - verdict = error_verdict(case, harness, commit, death_record(*died)) - write_file(root / "verdict.json", _json(verdict)) + error = died and death_record(*died) if run.turns or not died: trace = {**module.harvest(run), "x_implicit_off": run.implicit_off} _seal(authorization, meta(run), trace) - verdict = grade_run(authorization) + verdict = grade_run(authorization, error) + else: + verdict = error_verdict(case, harness, commit, error) + write_file(root / "verdict.json", _json(verdict)) except BaseException as original: owner = getattr(run, "_hermes_evidence", None) if owner is not None and retention is not None: @@ -335,13 +336,14 @@ def run_case(harness, case_id, skills_at, out, index, hermes_retain_out=None, mo return root -def grade_run(authorization): +def grade_run(authorization, error=None): _, state = authorization._controller.load(authorization._id) target = Path(state["output"]["path"]) / "verdict.json" - try: - error = json.loads(read_file(target)).get("error") - except FileNotFoundError: - error = None + if error is None: + try: + error = json.loads(read_file(target)).get("error") + except FileNotFoundError: + pass verdict = grade(authorization) if error: verdict = error_verdict(state["case"], verdict["harness"], verdict["skills_at"], error, verdict["promises"]) diff --git a/evals/promises/test_live.py b/evals/promises/test_live.py index 862c9334..6a8fff8c 100644 --- a/evals/promises/test_live.py +++ b/evals/promises/test_live.py @@ -628,6 +628,26 @@ def test_regrading_a_run_a_dead_login_ended_keeps_its_error_marks(self): {p: "ERROR" for p in verdict["promises"] if p != "persona-delivery-hermes-grok"}) self.assertIn("claude-code pass rate: none graded (ERROR 10 excluded)", table) + def test_a_harvest_refusal_after_an_auth_death_stops_the_batch_before_any_verdict(self): + from harnesses import claude_code + case = live.load_case("feature-run") + case["tmp_lock"] = False + + def turn(run, text, index): + (run.root / "err").write_bytes(b"") + return {"index": index, "session_id": "s", "argv": ["claude", text], "exit_code": 1, "timed_out": False, + "duration_s": 4.7, "stream": str(FIXTURES / "claude-code-401-last-line.jsonl"), "stderr": str(run.root / "err")} + + with tempfile.TemporaryDirectory(prefix="pstack-live-harvest-refused-") as tmp: + out = Path(tmp).resolve() + with mock.patch.object(live, "load_case", return_value=case), mock.patch.object(claude_code, "prepare"), \ + mock.patch.object(claude_code, "turn", side_effect=turn), \ + mock.patch.object(claude_code, "harvest", side_effect=GradeRefused("input_changed", "linked transcript")): + with self.assertRaisesRegex(GradeRefused, "linked transcript"): + live.run_case("claude-code", case["id"], "HEAD", out, 0) + written = sorted(p.name for p in out.rglob("*.json") if p.name in ("trace.json", "verdict.json")) + self.assertEqual(written, []) + def test_regrading_a_run_whose_login_died_before_any_turn_is_refused_and_keeps_its_verdict(self): from harnesses import codex case = live.load_case("principle-steer-run") From ef2b2f40f5d30657ddd60e25abd1962cdb0f9d0e Mon Sep 17 00:00:00 2001 From: mdsmithaustin <15272014+mdsmithaustin@users.noreply.github.com> Date: Mon, 12 Oct 2026 15:27:44 -0600 Subject: [PATCH 12/12] no-mistakes(document): Clarify auth-death harvest refusals and regrading documentation --- evals/promises/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/evals/promises/README.md b/evals/promises/README.md index 05811863..b5a833b3 100644 --- a/evals/promises/README.md +++ b/evals/promises/README.md @@ -75,7 +75,7 @@ Each run makes a fresh git repository from `fixtures//` and `histories/ of ` line. This rate counts applicable promise verdicts, not runs. `live.py` and `live.py report` decide applicability by asking each oracle about an empty trace from the harness, because a grade that returns early carries no `not_applicable` mark. A grade returns early when the run never started, its transcript could not be read, or host skills touched it. `INCONCLUSIVE` stays in the denominator. When every applicable verdict is `ERROR`, the rate reads `none graded`. A turn that exited 0 still grades as before, even if it contains login-failure text. +For Claude Code, Codex, and Grok, a turn counts as an auth death when it exited nonzero without timing out and the tail of its stderr or the last nonempty line of its stream carries that CLI's login-failure text. A `LoginRequired` failure during preparation stops the run before any turn. `live.py` writes a `verdict.json` that marks every promise that applies to the harness `ERROR` and records the CLI and its login command under `error`. It skips harvest and grading, so that run has no `trace.json`. When a turn dies on auth, `live.py` still harvests and grades the run. A `PASS` or `FAIL` the grader decides stands, and every other applicable promise becomes `ERROR`, keeping any `needs_judge` flag and `excerpt`. After writing either verdict, the wave stops with exit 3. A harvest refusal after an auth death propagates unchanged and stops the batch before the run gets `trace.json` or `verdict.json`. Running `live.py grade` on a sealed auth-death run later re-grades it from its trace, keeps the `error` record, and marks `ERROR` again on every promise the new grade leaves undecided. A run whose login died before any turn has no sealed trace, so `live.py grade` refuses it with exit 2 and leaves its verdict unchanged. In each of these verdicts a promise that does not apply to the harness is marked `not_applicable`, so it stays out of the report. `live.py report` shows `ERROR` in the table and leaves those verdicts out of each harness's `pass rate: of ` line. This rate counts applicable promise verdicts, not runs. `live.py` and `live.py report` decide applicability by asking each oracle about an empty trace from the harness, because a grade that returns early carries no `not_applicable` mark. A grade returns early when the run never started, its transcript could not be read, or host skills touched it. `INCONCLUSIVE` stays in the denominator. When every applicable verdict is `ERROR`, the rate reads `none graded`. A turn that exited 0 still grades as before, even if it contains login-failure text. `--model` and `--effort` set the model and reasoning effort for every turn of every case in one `run` invocation, on Claude Code and Codex. Each turn's recorded argv carries both values, and the sealed run metadata keeps its fields. Other adapters refuse the flags.