diff --git a/evals/README.md b/evals/README.md index 073f17b5..1e117840 100644 --- a/evals/README.md +++ b/evals/README.md @@ -10,9 +10,9 @@ each harness. See `promises/README.md`. Model answers, traces, judge transcripts, reports, and private cases stay outside Git. A committed case is tuning material, not sealed evidence. -Oracle regression fixtures under `promises/testdata/` are an exception. They -are reduced, sanitized traces that keep only what the oracle tests read, and -`promises/reduce_trace.py` rebuilds one from a raw trace. `promises/results/` +Oracle and login regression fixtures under `promises/testdata/` are an exception. +They are reduced, sanitized traces, CLI error records, and recorded verdicts used by model-free tests. +`promises/reduce_trace.py` reduces raw traces for oracle fixtures. `promises/results/` is a second exception. It holds aggregated verdict matrices with no model answers. diff --git a/evals/promises/README.md b/evals/promises/README.md index 83622542..b5a833b3 100644 --- a/evals/promises/README.md +++ b/evals/promises/README.md @@ -71,7 +71,11 @@ python3 evals/promises/live.py run --harness --c python3 evals/promises/live.py report --upstream-ref upstream/main ``` -Each run makes a fresh git repository from `fixtures//` and `histories//`. Each history step is committed in order, except a step with `"commit": false`, which stays as uncommitted work in the tree. A step with `"branch"` switches to that branch first, creating it at the current commit when it does not exist, so later steps land there and `main` keeps the fixture. It installs `skills/` from `--skills-at` (default `HEAD`) where the harness discovers project skills, and runs the case's turns headless. Each run directory gets `run.json`, `trace.json`, and `verdict.json`. Every adapter writes the same trace schema, with events, files read, worklist snapshots, spawns, and the final reply. A tool event carries its harness call id, so each result pairs with its own call. `oracles.py` grades each promise as `PASS`, `FAIL`, or `INCONCLUSIVE`. A verdict is `INCONCLUSIVE` when the trace cannot show the behavior, or when the harness reports `x_host_skill_hits`, which means the user's own skills touched the run. Some oracles also return `INCONCLUSIVE` when only a judge can decide, such as whether a reply's wording reports a review `pass`. That verdict carries `needs_judge: true` and an `excerpt` of the graded reply in `verdict.json`. In a multi-turn case the graded reply can come from an earlier turn than `final_reply`, and the verdict's evidence names that turn. When the excerpt is not enough, read that turn's text events in `trace.json` before you tally the run. +Each run makes a fresh git repository from `fixtures//` and `histories//`. Each history step is committed in order, except a step with `"commit": false`, which stays as uncommitted work in the tree. A step with `"branch"` switches to that branch first, creating it at the current commit when it does not exist, so later steps land there and `main` keeps the fixture. It installs `skills/` from `--skills-at` (default `HEAD`) where the harness discovers project skills, and runs the case's turns headless. A run that completes harvest and grading gets `run.json`, `trace.json`, and `verdict.json`. Every adapter writes the same trace schema, with events, files read, worklist snapshots, spawns, and the final reply. A tool event carries its harness call id, so each result pairs with its own call. `oracles.py` grades each promise as `PASS`, `FAIL`, or `INCONCLUSIVE`, and `live.py` alone adds `ERROR` for a dead login. A verdict is `INCONCLUSIVE` when the trace cannot show the behavior, or when the harness reports `x_host_skill_hits`, which means the user's own skills touched the run. Some oracles also return `INCONCLUSIVE` when only a judge can decide, such as whether a reply's wording reports a review `pass`. That verdict carries `needs_judge: true` and an `excerpt` of the graded reply in `verdict.json`. In a multi-turn case the graded reply can come from an earlier turn than `final_reply`, and the verdict's evidence names that turn. When the excerpt is not enough, read that turn's text events in `trace.json` before you tally the run. + +Before the first case, `live.py run` checks the selected Claude Code, Codex, or Grok login at no model cost. A failed check stops the wave with exit 3 before any case runs, and the message names the CLI and its login command. Claude Code runs `claude auth status` with the host `HOME`, `USER`, a minimal `PATH`, and an empty `CLAUDE_SECURESTORAGE_CONFIG_DIR`, so a host-exported `ANTHROPIC_API_KEY` cannot mask a missing login. It does not set the private `CLAUDE_CONFIG_DIR` each turn gets, since that directory exists only after a run is prepared, so it checks the host login. Codex needs the `auth.json` file and a passing `codex login status`. Grok refuses a host token that expires within five minutes, because the run's read-only provider never refreshes it. Claude Code and Codex refresh their own access tokens when they run, so their checks do not read token expiry. Hermes has no preflight check, and its preparation step still refuses a missing `auth.json`. + +For Claude Code, Codex, and Grok, a turn counts as an auth death when it exited nonzero without timing out and the tail of its stderr or the last nonempty line of its stream carries that CLI's login-failure text. A `LoginRequired` failure during preparation stops the run before any turn. `live.py` writes a `verdict.json` that marks every promise that applies to the harness `ERROR` and records the CLI and its login command under `error`. It skips harvest and grading, so that run has no `trace.json`. When a turn dies on auth, `live.py` still harvests and grades the run. A `PASS` or `FAIL` the grader decides stands, and every other applicable promise becomes `ERROR`, keeping any `needs_judge` flag and `excerpt`. After writing either verdict, the wave stops with exit 3. A harvest refusal after an auth death propagates unchanged and stops the batch before the run gets `trace.json` or `verdict.json`. Running `live.py grade` on a sealed auth-death run later re-grades it from its trace, keeps the `error` record, and marks `ERROR` again on every promise the new grade leaves undecided. A run whose login died before any turn has no sealed trace, so `live.py grade` refuses it with exit 2 and leaves its verdict unchanged. In each of these verdicts a promise that does not apply to the harness is marked `not_applicable`, so it stays out of the report. `live.py report` shows `ERROR` in the table and leaves those verdicts out of each harness's `pass rate: of ` line. This rate counts applicable promise verdicts, not runs. `live.py` and `live.py report` decide applicability by asking each oracle about an empty trace from the harness, because a grade that returns early carries no `not_applicable` mark. A grade returns early when the run never started, its transcript could not be read, or host skills touched it. `INCONCLUSIVE` stays in the denominator. When every applicable verdict is `ERROR`, the rate reads `none graded`. A turn that exited 0 still grades as before, even if it contains login-failure text. `--model` and `--effort` set the model and reasoning effort for every turn of every case in one `run` invocation, on Claude Code and Codex. Each turn's recorded argv carries both values, and the sealed run metadata keeps its fields. Other adapters refuse the flags. @@ -103,7 +107,7 @@ A completed run prints `authorization_id`. To regrade that registered run, use t python3 evals/promises/live.py grade --out ``` -The grade CLI exits 2 and writes a refusal receipt to stderr when the ID is unknown or grading refuses. A successful grade returns the existing verdict dictionary and writes `verdict.json`. It preserves the JSON indentation and final newline. The Python entry point is `live.grade(authorization)`. It raises `GradeRefused` for infrastructure and unsafe-I/O refusals. A refusal has no promise verdict. Its separate receipt and worker logs remain under the controller's run and attempt directories. +The grade CLI exits 2 and writes a refusal receipt to stderr when the ID is unknown or grading refuses. A successful grade returns the existing verdict dictionary and writes `verdict.json`. It preserves the JSON indentation and final newline. The Python entry point is `live.grade(authorization)`. The CLI calls it through `live.grade_run(authorization)`, which also keeps a dead login's `error` record and its `ERROR` marks, as [Run live cases](#run-live-cases) describes. It raises `GradeRefused` for infrastructure and unsafe-I/O refusals. A refusal has no promise verdict. Its separate receipt and worker logs remain under the controller's run and attempt directories. Missing sealed `run.json`, `trace.json`, or retained prior `verdict.json` produces an `input_changed` refusal and an attempt receipt. The refusal leaves any existing output verdict unchanged. An intentionally absent project keeps its oracle result, and an absent output verdict still permits first publication. diff --git a/evals/promises/harnesses/claude_code.py b/evals/promises/harnesses/claude_code.py index f29702db..1be5b39d 100644 --- a/evals/promises/harnesses/claude_code.py +++ b/evals/promises/harnesses/claude_code.py @@ -15,11 +15,15 @@ from pathlib import Path import live +from harnesses.login import Login, LoginRequired, check_status SKILLS_DIR = ".claude/skills" PRIVATE_DIRS = (".claude/",) SHARES_HOST_TMP = False +LOGIN = Login("claude", "claude auth login", re.compile( + r"Please run /login|Not logged in|Login expired|OAuth token (?:revoked|has expired)|Invalid API key|" + r"Failed to authenticate|Authentication required", re.I)) DEFAULT_EFFORT = "high" SETTING_SOURCES = "user,project" LISTING_SCRIPT = Path("setup-pstack/scripts/skill-listing.py") @@ -196,6 +200,23 @@ def store(run): return config_dir(run) / "projects" +def preflight(): + """Check the host login before a private turn config exists. Keep the turn's host HOME and Keychain namespace, + but omit exported API keys and config overrides so they cannot mask a missing login.""" + binary = CLAUDE_BINARY if CLAUDE_BINARY.is_file() else shutil.which("claude") + if not binary: + raise LoginRequired(LOGIN, "no claude binary to ask") + env = {"HOME": os.environ["HOME"], "USER": os.environ.get("USER", ""), "LOGNAME": os.environ.get("LOGNAME", ""), + "PATH": "/usr/bin:/bin", "LANG": "en_US.UTF-8", "TERM": "dumb", "CLAUDE_SECURESTORAGE_CONFIG_DIR": ""} + out = check_status(LOGIN, [binary, "auth", "status"], env=env) + try: + logged_in = json.loads(out).get("loggedIn") + except ValueError: + logged_in = None + if logged_in is not True: + raise LoginRequired(LOGIN, f"`claude auth status` reports loggedIn {logged_in!r}") + + def prepare(run): state = _bind_paths(run, _host_runtime()) script = state.paths.project / SKILLS_DIR / "pstack-harness/scripts/subagents.py" diff --git a/evals/promises/harnesses/codex.py b/evals/promises/harnesses/codex.py index a00811cb..b68bc7ab 100644 --- a/evals/promises/harnesses/codex.py +++ b/evals/promises/harnesses/codex.py @@ -9,6 +9,7 @@ import live from grade_boundary import GradeRefused, read_file, write_file +from harnesses.login import Login, LoginRequired, check_status SKILLS_DIR = ".agents/skills" PRIVATE_DIRS = [".agents/", ".codex/"] @@ -19,6 +20,9 @@ EFFORT = "high" COMPACT = "/compact" COMPACT_TOKEN_LIMIT = 2000 +LOGIN = Login("codex", "codex login", re.compile( + r"access token could not be refreshed|refresh token was (?:already used|rejected)|log out and sign in again|" + r"authentication session could not be refreshed|Not logged in", re.I)) ENV_KEPT = ("PATH", "LANG", "LC_ALL", "LC_CTYPE", "LC_MESSAGES", "TERM", "USER", "LOGNAME", "SHELL", "TZ", "SSL_CERT_FILE", "SSL_CERT_DIR", "HTTP_PROXY", "HTTPS_PROXY", "NO_PROXY", "http_proxy", "https_proxy", "no_proxy") @@ -57,13 +61,26 @@ def resolve_binary(env): raise RuntimeError(f"no working codex binary; rejected: {json.dumps(rejected)}") +def host_auth(): + return Path(os.environ.get("CODEX_HOME") or Path.home() / ".codex") / "auth.json" + + +def preflight(): + """Ask the CLI whether it holds a login. The CLI refreshes an expired access token itself when it runs, so the token's + own expiry is not checked here.""" + auth = host_auth() + if not auth.is_file(): + raise LoginRequired(LOGIN, f"no login at {auth}") + check_status(LOGIN, [resolve_binary(dict(os.environ))["path"], "login", "status"]) + + def prepare(run): root = run.root for name in ("home", "codex-home", "tmp", "transcripts"): (root / name).mkdir(exist_ok=True) - auth = Path(os.environ.get("CODEX_HOME") or Path.home() / ".codex") / "auth.json" + auth = host_auth() if not auth.is_file(): - raise RuntimeError(f"no codex login at {auth}") + raise LoginRequired(LOGIN, f"no login at {auth}") link = root / "codex-home" / "auth.json" if not link.is_symlink(): link.symlink_to(auth) diff --git a/evals/promises/harnesses/grok.py b/evals/promises/harnesses/grok.py index 448af118..78e2b6b1 100644 --- a/evals/promises/harnesses/grok.py +++ b/evals/promises/harnesses/grok.py @@ -11,6 +11,7 @@ import live from grade_boundary import GradeRefused, copy_file +from harnesses.login import Login, LoginRequired SKILLS_DIR = ".agents/skills" PRIVATE_DIRS = [".agents/"] @@ -132,6 +133,22 @@ def host_token_expiry(auth): return datetime.datetime.fromisoformat(entry["expires_at"].replace("Z", "+00:00")) if entry else None +LOGIN = Login("grok", "grok login", re.compile( + r"Run `grok login`|auth token is invalid or expired|Not authenticated|Authentication required|no xAI credential|" + r"Invalid or expired credentials|sign-in helper could not renew", re.I)) +TOKEN_MARGIN = datetime.timedelta(minutes=5) + + +def preflight(): + """Refuse a missing or expired host token. The run's read-only provider never refreshes it, so only `grok login` on the host does.""" + auth = host_grok_home() / "auth.json" + if not auth.is_file(): + raise LoginRequired(LOGIN, f"no login at {auth}") + expiry = host_token_expiry(auth) + if expiry is not None and expiry <= datetime.datetime.now(datetime.timezone.utc) + TOKEN_MARGIN: + raise LoginRequired(LOGIN, f"the host token expired at {expiry:%Y-%m-%dT%H:%MZ}, and the run's read-only provider never refreshes it") + + def prepare(run): root = run.root for name in ("home", "grok-home", "tmp", "transcripts"): @@ -139,13 +156,8 @@ def prepare(run): for name in HOME_DIRS: (home(run) / name).mkdir(exist_ok=True) (home(run) / ".gitconfig").write_text(GITCONFIG) + preflight() auth = host_grok_home() / "auth.json" - if not auth.is_file(): - raise RuntimeError(f"no grok login at {auth}") - expiry = host_token_expiry(auth) - if expiry is not None and expiry <= datetime.datetime.now(datetime.timezone.utc) + datetime.timedelta(minutes=5): - raise RuntimeError(f"the host Grok token expired at {expiry:%Y-%m-%dT%H:%MZ}; run `grok login` on the host, " - "since the run's read-only provider never refreshes it") provider = root / "auth-provider.py" provider.write_text(AUTH_PROVIDER.format(python=shutil.which("python3") or "/usr/bin/python3", auth=str(auth))) provider.chmod(0o755) @@ -159,7 +171,7 @@ def prepare(run): text=True, timeout=120) (root / "login.log").write_text(login.stdout + login.stderr) if login.returncode != 0: - raise RuntimeError(f"grok login failed under the run's own home: {(login.stdout + login.stderr)[-300:]}") + raise LoginRequired(LOGIN, f"`grok login` failed under the run's own home: {(login.stdout + login.stderr)[-300:]}") inspected = subprocess.run([launch["path"], "inspect", "--json"], cwd=run.project, env=env, capture_output=True, text=True, timeout=120, check=True) (root / "inspect.json").write_text(inspected.stdout) diff --git a/evals/promises/harnesses/login.py b/evals/promises/harnesses/login.py new file mode 100644 index 00000000..52365c38 --- /dev/null +++ b/evals/promises/harnesses/login.py @@ -0,0 +1,86 @@ +"""What a live wave needs from each CLI's login, shared by the adapters and live.py.""" + +import re +import subprocess +from dataclasses import dataclass +from pathlib import Path + +from grade_boundary import read_file + +ERROR = "ERROR" +DECIDED = ("PASS", "FAIL") +STATUS_TIMEOUT_S = 60 +DEATH_WINDOW = 4000 + + +@dataclass(frozen=True) +class Login: + cli: str + relogin: str + death: re.Pattern + + +class LoginRequired(RuntimeError): + def __init__(self, login, detail): + super().__init__(f"{login.cli} login is not usable: {detail}. Run `{login.relogin}`, then start the wave again") + self.login = login + self.detail = detail + + +def check_status(login, argv, env=None): + """Run the CLI's own login status command. It spends no model tokens.""" + try: + done = subprocess.run([str(a) for a in argv], env=env, capture_output=True, text=True, timeout=STATUS_TIMEOUT_S, + stdin=subprocess.DEVNULL) + except (OSError, subprocess.TimeoutExpired) as error: + raise LoginRequired(login, f"`{' '.join(map(str, argv))}` did not finish: {error!r}") from error + if done.returncode: + raise LoginRequired(login, f"`{' '.join(map(str, argv))}` exited {done.returncode}: {(done.stdout + done.stderr).strip()[-300:]}") + return done.stdout + + +def auth_death(login, record): + """The auth-failure text that ended a turn, or None. Only a turn that exited nonzero counts, and only its stderr tail + and last stream line, so a reply that quotes a login message in a run that finished is not read as a dead login.""" + if not record.get("exit_code") or record.get("timed_out"): + return None + tail = [] + if record.get("stderr"): + tail.append(_read(record["stderr"])[-DEATH_WINDOW:]) + if record.get("stream"): + lines = [line for line in _read(record["stream"]).splitlines() if line.strip()] + tail.append(lines[-1][-DEATH_WINDOW:] if lines else "") + found = login.death.search("\n".join(tail)) + return found.group(0) if found else None + + +def _read(path): + try: + return read_file(Path(path)).decode(errors="replace") + except FileNotFoundError: + return "" + + +def death_record(login, detail): + return {"cli": login.cli, "relogin": login.relogin, "detail": detail} + + +def error_verdict(case, harness, skills_at, error, graded=None): + reason = f"{error['cli']} login died: {error['detail']}" + promises = {} + for pid in case["promises"]: + result = graded[pid] if graded else {} + gate = applicability(pid, harness) + if gate.get("not_applicable"): + promises[pid] = gate + elif result.get("verdict") in DECIDED: + promises[pid] = result + else: + judge = {k: result[k] for k in ("needs_judge", "excerpt") if k in result} + promises[pid] = {"verdict": ERROR, "failures": [reason], "evidence": [], **judge} + return {"case": case["id"], "harness": harness, "skills_at": skills_at, "error": error, "promises": promises} + + +def applicability(pid, harness): + import oracles + return oracles.check(pid, {"harness": harness}, None, None) diff --git a/evals/promises/live.py b/evals/promises/live.py index 23295d1c..76be5535 100644 --- a/evals/promises/live.py +++ b/evals/promises/live.py @@ -17,7 +17,8 @@ from dataclasses import dataclass, field from pathlib import Path -from grade_boundary import create_file, grade, GradeRefused, read_file, _before_turns, _lookup, _seal, _write_record +from grade_boundary import create_file, grade, GradeRefused, read_file, write_file, _before_turns, _json, _lookup, _seal, _write_record +from harnesses.login import LoginRequired, auth_death, death_record, error_verdict HERE = Path(__file__).resolve().parent ROOT = HERE.parents[1] @@ -64,6 +65,13 @@ def adapter(name): return importlib.import_module(HARNESSES[name]) +def preflight(harness): + """Refuse a wave whose login is gone before any case spends a token. An adapter with no `preflight` has no free check.""" + check = getattr(adapter(harness), "preflight", None) + if check: + check() + + def load_case(case_id): case = json.loads((CASES / case_id / "case.json").read_text(encoding="utf-8")) case["id"] = case_id @@ -241,6 +249,28 @@ def fixture_lock(fixture): return handle +def drive(module, run, authorization): + """Prepare the run and play its turns. Returns (login, detail) when a login died before or during a turn, else None.""" + try: + module.prepare(run) + except LoginRequired as expired: + return expired.login, expired.detail + login = getattr(module, "LOGIN", None) + lock = fixture_lock(run.case["fixture"]) if module.SHARES_HOST_TMP and run.case.get("tmp_lock", True) else None + try: + for i, text in enumerate(run.case["turns"]): + record = module.turn(run, text, i) + run.turns.append(record) + _write_record(authorization, meta(run)) + dead = login and auth_death(login, record) + if dead: + return login, f"turn {i} exited {record['exit_code']} with {dead!r}" + finally: + if lock: + lock.close() + return None + + def run_case(harness, case_id, skills_at, out, index, hermes_retain_out=None, model=None, effort=None): if hermes_retain_out is not None and harness != "hermes": raise ValueError("--hermes-retain-out requires --harness hermes") @@ -268,19 +298,15 @@ def run_case(harness, case_id, skills_at, out, index, hermes_retain_out=None, mo authorization = _before_turns(run) pair = None try: - module.prepare(run) - lock = fixture_lock(case["fixture"]) if module.SHARES_HOST_TMP and case.get("tmp_lock", True) else None - try: - for i, text in enumerate(case["turns"]): - record = module.turn(run, text, i) - run.turns.append(record) - _write_record(authorization, meta(run)) - finally: - if lock: - lock.close() - trace = {**module.harvest(run), "x_implicit_off": run.implicit_off} - _seal(authorization, meta(run), trace) - verdict = grade(authorization) + died = drive(module, run, authorization) + error = died and death_record(*died) + if run.turns or not died: + trace = {**module.harvest(run), "x_implicit_off": run.implicit_off} + _seal(authorization, meta(run), trace) + verdict = grade_run(authorization, error) + else: + verdict = error_verdict(case, harness, commit, error) + write_file(root / "verdict.json", _json(verdict)) except BaseException as original: owner = getattr(run, "_hermes_evidence", None) if owner is not None and retention is not None: @@ -305,9 +331,26 @@ def run_case(harness, case_id, skills_at, out, index, hermes_retain_out=None, mo print(json.dumps({"run": str(root), "authorization_id": authorization._id, "case": case_id, "harness": harness, "n": index, **({"hermes_pair": str(pair.root)} if pair is not None else {}), "verdicts": {p: v["verdict"] for p, v in verdict["promises"].items()}})) + if died: + raise LoginRequired(*died) return root +def grade_run(authorization, error=None): + _, state = authorization._controller.load(authorization._id) + target = Path(state["output"]["path"]) / "verdict.json" + if error is None: + try: + error = json.loads(read_file(target)).get("error") + except FileNotFoundError: + pass + verdict = grade(authorization) + if error: + verdict = error_verdict(state["case"], verdict["harness"], verdict["skills_at"], error, verdict["promises"]) + write_file(target, _json(verdict)) + return verdict + + def meta(run): return {"harness": run.harness, "case": run.case["id"], "skills_at": run.skills_at, "project": str(run.project), "timeout_s": run.timeout_s, "turns": run.turns, "baseline": run.baseline} @@ -336,14 +379,19 @@ def main(argv=None): if args.cmd == "run": out = Path(args.out) out.mkdir(parents=True, exist_ok=True) - for case_id in args.case: - for n in range(args.runs): - run_case(args.harness, case_id, args.skills_at, out, n, args.hermes_retain_out, args.model, args.effort) + try: + preflight(args.harness) + for case_id in args.case: + for n in range(args.runs): + run_case(args.harness, case_id, args.skills_at, out, n, args.hermes_retain_out, args.model, args.effort) + except LoginRequired as expired: + print(f"wave stopped: {expired}", file=sys.stderr) + return 3 return 0 if args.cmd == "grade": for run_id in args.run_ids: try: - print(json.dumps(grade(_lookup(args.out, run_id)))) + print(json.dumps(grade_run(_lookup(args.out, run_id)))) except GradeRefused as refusal: print(json.dumps({"refusal": refusal.receipt}), file=sys.stderr) return 2 diff --git a/evals/promises/report.py b/evals/promises/report.py index 69ceb348..00bff47d 100644 --- a/evals/promises/report.py +++ b/evals/promises/report.py @@ -3,6 +3,7 @@ from pathlib import Path import ledger +from harnesses.login import ERROR, applicability HARNESSES = ("claude-code", "codex", "hermes", "grok") @@ -12,7 +13,7 @@ def collect(out): for path in sorted(Path(out).rglob("verdict.json")): record = json.loads(path.read_text()) for pid, result in record["promises"].items(): - if not result.get("not_applicable"): + if not result.get("not_applicable") and not applicability(pid, record["harness"]).get("not_applicable"): cells[(pid, record["harness"])].append((result["verdict"], str(path.parent))) for path in sorted(Path(out).rglob("install.json")): for row in json.loads(path.read_text()): @@ -41,6 +42,22 @@ def cell(results): return ", ".join(f"{v} {n}" for v, n in sorted(counts.items())) +def pass_rates(cells): + """One line per harness. An ERROR verdict is a run a dead login ended, so it leaves the denominator.""" + counts = collections.defaultdict(collections.Counter) + for (_, harness), results in cells.items(): + counts[harness].update(v for v, _ in results) + lines = [] + for harness in HARNESSES: + tally = counts.get(harness) + if not tally: + continue + graded = sum(n for v, n in tally.items() if v != ERROR) + rate = f"{tally['PASS']} of {graded}" if graded else "none graded" + lines.append(f"{harness} pass rate: {rate}" + (f" (ERROR {tally[ERROR]} excluded)" if tally[ERROR] else "")) + return lines + + def render(out, upstream=None): book = ledger.load() cells = collect(out) @@ -60,5 +77,6 @@ def render(out, upstream=None): lines.append("") lines.append(f"{len(pids)} promises, {sum(totals.values())} verdicts: " + ", ".join(f"{v} {n}" for v, n in sorted(totals.items()))) + lines += pass_rates(cells) lines += listing_states(out) return "\n".join(lines) diff --git a/evals/promises/test_live.py b/evals/promises/test_live.py index 39032146..6a8fff8c 100644 --- a/evals/promises/test_live.py +++ b/evals/promises/test_live.py @@ -10,11 +10,14 @@ import unittest from pathlib import Path from unittest import mock -from contextlib import redirect_stdout +from contextlib import redirect_stderr, redirect_stdout from contextlib import nullcontext import live from grade_boundary import GradeRefused, _authorize_fixture +from harnesses.login import LoginRequired, auth_death, death_record, error_verdict + +FIXTURES = Path(__file__).resolve().parent / "testdata" / "auth_death" CODEX_CHAT = """I'm using `poteto-mode`. I'll identify the command first. @@ -333,7 +336,8 @@ def test_model_and_effort_refuse_harnesses_without_an_override(self): live.run_case(harness, "principle-steer-run", "HEAD", Path("/unused"), 0, model="m") def test_the_cli_passes_model_and_effort_to_every_case(self): - with tempfile.TemporaryDirectory(prefix="pstack-overrides-") as tmp, mock.patch.object(live, "run_case") as run_case: + with tempfile.TemporaryDirectory(prefix="pstack-overrides-") as tmp, mock.patch.object(live, "run_case") as run_case, \ + mock.patch.object(live, "preflight"): live.main(["run", "--harness", "codex", "--case", "a", "--case", "b", "--model", "gpt-6-luna", "--effort", "xhigh", "--out", tmp]) self.assertEqual([c.args[1] for c in run_case.call_args_list], ["a", "b"]) self.assertEqual({c.args[-2:] for c in run_case.call_args_list}, {("gpt-6-luna", "xhigh")}) @@ -357,6 +361,387 @@ def test_codex_and_claude_turns_carry_the_override_in_argv(self): self.assertEqual(argv[-5:], ("--effort", "high", "--model", "claude-sonnet-5-5", "go")) +def fake_cli(directory, name, script): + path = Path(directory) / name + path.write_text("#!/bin/sh\n" + script) + path.chmod(0o755) + return path + + +class LoginPreflight(unittest.TestCase): + """A wave stops on a login the CLI reports gone, before any case spends a token.""" + + def setUp(self): + tmp = tempfile.TemporaryDirectory(prefix="pstack-preflight-") + self.addCleanup(tmp.cleanup) + self.tmp = Path(tmp.name).resolve() + + def wave(self, harness, *patches): + stderr = io.StringIO() + with mock.patch.object(live, "run_case") as run_case, redirect_stderr(stderr): + for patch in patches: + self.enterContext(patch) + code = live.main(["run", "--harness", harness, "--case", "a", "--case", "b", "--out", str(self.tmp / "out")]) + return code, run_case, stderr.getvalue() + + def claude(self, script): + from harnesses import claude_code + return mock.patch.object(claude_code, "CLAUDE_BINARY", fake_cli(self.tmp, "claude", script)) + + def codex(self, script, with_auth=True): + home = self.tmp / "codex-home" + home.mkdir(exist_ok=True) + if with_auth: + (home / "auth.json").write_text("{}") + binary = fake_cli(self.tmp, "codex", script) + return mock.patch.dict(os.environ, {"CODEX_HOME": str(home), "CODEX_BIN": str(binary)}) + + def grok(self, expires_at): + from harnesses import grok + home = self.tmp / "grok-home" + home.mkdir(exist_ok=True) + (home / "auth.json").write_text(json.dumps({"xai": {"key": "k", "expires_at": expires_at}})) + return mock.patch.object(grok, "host_grok_home", return_value=home) + + def assertStopped(self, outcome, cli, command): + code, run_case, stderr = outcome + self.assertEqual(code, 3) + run_case.assert_not_called() + self.assertEqual(list((self.tmp / "out").iterdir()), []) + self.assertIn(f"{cli} login is not usable", stderr) + self.assertIn(f"Run `{command}`", stderr) + + def test_claude_code_reporting_no_login_stops_the_wave(self): + outcome = self.wave("claude-code", self.claude('echo \'{"loggedIn": false}\'; exit 1\n')) + self.assertStopped(outcome, "claude", "claude auth login") + + def test_claude_code_exiting_zero_with_loggedin_false_still_stops_the_wave(self): + outcome = self.wave("claude-code", self.claude('echo \'{"loggedIn": false}\'\n')) + self.assertStopped(outcome, "claude", "claude auth login") + + def test_claude_code_status_does_not_see_credentials_only_the_host_shell_exports(self): + script = 'if [ -n "$ANTHROPIC_API_KEY" ]; then echo \'{"loggedIn": true}\'; else echo \'{"loggedIn": false}\'; exit 1; fi\n' + outcome = self.wave("claude-code", self.claude(script), mock.patch.dict(os.environ, {"ANTHROPIC_API_KEY": "host-only"})) + self.assertStopped(outcome, "claude", "claude auth login") + + def test_codex_login_status_failing_stops_the_wave(self): + script = 'case "$1" in --version) echo codex 1;; *) echo "Not logged in" >&2; exit 1;; esac\n' + self.assertStopped(self.wave("codex", self.codex(script)), "codex", "codex login") + + def test_codex_without_an_auth_file_stops_the_wave_even_when_the_cli_says_logged_in(self): + script = 'case "$1" in --version) echo codex 1;; *) echo "Logged in using ChatGPT";; esac\n' + self.assertStopped(self.wave("codex", self.codex(script, with_auth=False)), "codex", "codex login") + + def test_grok_with_an_expired_host_token_stops_the_wave(self): + self.assertStopped(self.wave("grok", self.grok("2020-01-01T00:00:00Z")), "grok", "grok login") + + def test_a_valid_login_lets_every_case_run(self): + script = 'case "$1" in --version) echo codex 1;; *) echo "Logged in using ChatGPT";; esac\n' + code, run_case, _ = self.wave("codex", self.codex(script)) + self.assertEqual((code, [c.args[1] for c in run_case.call_args_list]), (0, ["a", "b"])) + + def test_a_login_that_dies_mid_wave_stops_the_wave_after_its_error_verdict(self): + from harnesses import codex + died = LoginRequired(codex.LOGIN, "turn 0 exited 1") + stderr = io.StringIO() + with mock.patch.object(live, "preflight"), mock.patch.object(live, "run_case", side_effect=[died]) as run_case, redirect_stderr(stderr): + code = live.main(["run", "--harness", "codex", "--case", "a", "--case", "b", "--out", str(self.tmp / "out")]) + self.assertEqual((code, run_case.call_count), (3, 1)) + self.assertIn("Run `codex login`", stderr.getvalue()) + + +class AuthDeath(unittest.TestCase): + """A turn that ended on an auth failure is the harness failing, not the skill under test.""" + + def setUp(self): + tmp = tempfile.TemporaryDirectory(prefix="pstack-auth-death-") + self.addCleanup(tmp.cleanup) + self.tmp = Path(tmp.name).resolve() + + def record(self, stderr="", stream="", exit_code=1, timed_out=False): + (self.tmp / "err").write_text(stderr) + (self.tmp / "out").write_text(stream) + return {"exit_code": exit_code, "timed_out": timed_out, "stderr": str(self.tmp / "err"), "stream": str(self.tmp / "out")} + + def test_each_cli_expired_login_message_is_an_auth_death(self): + from harnesses import claude_code, codex, grok + cases = [(claude_code.LOGIN, {"stream": '{"type":"result","is_error":true,"result":"Invalid API key \u00b7 Please run /login"}\n'}), + (claude_code.LOGIN, {"stderr": "Login expired \u00b7 Please run /login"}), + (codex.LOGIN, {"stderr": "Your access token could not be refreshed because your refresh token was already used. Please log out and sign in again."}), + (grok.LOGIN, {"stderr": "Authentication required. Run `grok login` to re-authenticate."})] + for login, streams in cases: + with self.subTest(cli=login.cli, streams=list(streams)): + self.assertTrue(auth_death(login, self.record(**streams))) + + def test_the_grok_external_auth_failure_a_real_run_ended_on_is_an_auth_death(self): + from harnesses import grok + record = {"exit_code": 1, "timed_out": False, "stream": str(FIXTURES / "grok-external-auth.stream.json"), + "stderr": str(FIXTURES / "grok-external-auth.stderr.txt")} + self.assertTrue(auth_death(grok.LOGIN, record)) + self.assertTrue(auth_death(grok.LOGIN, {"exit_code": 1, "timed_out": False, "stream": record["stream"]})) + self.assertTrue(auth_death(grok.LOGIN, {"exit_code": 1, "timed_out": False, "stderr": record["stderr"]})) + + def test_the_claude_code_401_a_real_run_ended_on_is_an_auth_death(self): + from harnesses import claude_code + record = {"exit_code": 1, "timed_out": False, "stream": str(FIXTURES / "claude-code-401-last-line.jsonl")} + self.assertIn("Failed to authenticate", auth_death(claude_code.LOGIN, record)) + + def test_a_turn_that_exited_zero_or_timed_out_or_died_for_another_reason_is_not_an_auth_death(self): + from harnesses import claude_code + login = claude_code.LOGIN + text = "Please run /login" + for name, record in (("exit zero", self.record(stderr=text, exit_code=0)), + ("timed out", self.record(stderr=text, exit_code=-9, timed_out=True)), + ("other failure", self.record(stderr="segmentation fault"))): + with self.subTest(name): + self.assertIsNone(auth_death(login, record)) + + def test_a_login_message_quoted_early_in_a_longer_stream_is_not_an_auth_death(self): + from harnesses import claude_code + stream = '{"type":"assistant","text":"the docs say Please run /login"}\n{"type":"result","is_error":true,"result":"max turns"}\n' + record = self.record(stream=stream) + self.assertIsNone(auth_death(claude_code.LOGIN, record)) + + def test_drive_ends_the_run_at_the_dying_turn_and_names_the_login(self): + from harnesses import claude_code + turns = [] + + def turn(run, text, index): + turns.append(index) + return {"index": index, **self.record(stderr="Not logged in \u00b7 Please run /login")} + + module = mock.Mock(SHARES_HOST_TMP=False, LOGIN=claude_code.LOGIN, turn=turn) + run = live.Run(self.tmp, "claude-code", {"id": "x", "fixture": "relay", "turns": ["one", "two"]}, "0" * 40, 60) + with mock.patch.object(live, "_write_record"): + login, detail = live.drive(module, run, object()) + self.assertEqual((login.cli, turns), ("claude", [0])) + self.assertIn("turn 0 exited 1", detail) + self.assertIn("Not logged in", detail) + + def test_drive_reports_a_login_that_died_in_prepare(self): + from harnesses import grok + module = mock.Mock(SHARES_HOST_TMP=False, LOGIN=grok.LOGIN, prepare=mock.Mock(side_effect=LoginRequired(grok.LOGIN, "token expired"))) + run = live.Run(self.tmp, "grok", {"id": "x", "fixture": "relay", "turns": ["one"]}, "0" * 40, 60) + self.assertEqual(live.drive(module, run, object()), (grok.LOGIN, "token expired")) + module.turn.assert_not_called() + + +@unittest.skipUnless((platform.system(), platform.release(), platform.machine()) == ("Darwin", "25.6.0", "arm64"), + "native parent grading requires the reviewed Darwin 25.6.0 arm64 runtime") +class ErrorRun(unittest.TestCase): + def test_a_run_that_dies_on_auth_is_graded_error_not_fail_and_stops_the_wave(self): + import report + from harnesses import codex + case = live.load_case("principle-steer-run") + case["turns"] = ["first", "second"] + case["tmp_lock"] = False + with tempfile.TemporaryDirectory(prefix="pstack-live-error-") as tmp: + out = Path(tmp).resolve() + + def turn(run, text, index): + (run.root / "err").write_text("Please log out and sign in again.") + (run.root / "out").write_bytes(b"") + return {"index": index, "session_id": "s", "argv": ["codex", text], "exit_code": 1, "timed_out": False, + "duration_s": 0.1, "stream": str(run.root / "out"), "stderr": str(run.root / "err")} + + def prepare(run): + (run.root / "launch.json").write_text(json.dumps({"path": "synthetic-no-model", "source": "test", "version": "test", "rejected": []})) + + output = io.StringIO() + with mock.patch.object(live, "load_case", return_value=case), mock.patch.object(codex, "prepare", side_effect=prepare), \ + mock.patch.object(codex, "turn", side_effect=turn) as turns, redirect_stdout(output): + with self.assertRaisesRegex(live.LoginRequired, "Run `codex login`"): + live.run_case("codex", case["id"], "HEAD", out, 0) + [root] = [p.parent for p in out.rglob("verdict.json")] + verdict = json.loads((root / "verdict.json").read_text()) + table = report.render(out, upstream=out) + self.assertEqual(turns.call_count, 1) + self.assertEqual({r["verdict"] for r in verdict["promises"].values()}, {"ERROR"}) + self.assertEqual(set(verdict["promises"]), set(case["promises"])) + self.assertEqual(verdict["error"]["relogin"], "codex login") + self.assertEqual(set(json.loads(output.getvalue())["verdicts"].values()), {"ERROR"}) + self.assertIn("codex pass rate: none graded (ERROR", table) + + def test_a_login_that_dies_after_the_graded_behavior_keeps_the_graders_verdict(self): + import report + from harnesses import claude_code + case = live.load_case("feature-boundary-run") + case["tmp_lock"] = False + trace = json.loads((FIXTURES / "claude-code-after-ipc-trace.json").read_text()) + + def turn(run, text, index): + (run.root / "err").write_bytes(b"") + return {"index": index, "session_id": "s", "argv": ["claude", text], "exit_code": 1, "timed_out": False, + "duration_s": 582.6, "stream": str(FIXTURES / "claude-code-401-last-line.jsonl"), "stderr": str(run.root / "err")} + + with tempfile.TemporaryDirectory(prefix="pstack-live-late-death-") as tmp: + out = Path(tmp).resolve() + output = io.StringIO() + with mock.patch.object(live, "load_case", return_value=case), mock.patch.object(claude_code, "prepare"), \ + mock.patch.object(claude_code, "turn", side_effect=turn), \ + mock.patch.object(claude_code, "harvest", return_value=trace), redirect_stdout(output): + with self.assertRaisesRegex(live.LoginRequired, "Run `claude auth login`"): + live.run_case("claude-code", case["id"], "HEAD", out, 0) + [root] = [p.parent for p in out.rglob("verdict.json")] + verdict = json.loads((root / "verdict.json").read_text()) + sealed = json.loads((root / "trace.json").read_text()) + table = report.render(out, upstream=out) + promise = "poteto-mode-triggers-architect-on-boundary-crossing" + self.assertEqual(verdict["promises"][promise]["verdict"], "PASS") + self.assertEqual(verdict["error"]["cli"], "claude") + self.assertEqual(sealed["harness"], "claude-code") + self.assertEqual(json.loads(output.getvalue())["verdicts"], {promise: "PASS"}) + self.assertIn("claude-code pass rate: 1 of 1", table) + + def claude_feature_run_that_never_started(self, out): + from harnesses import claude_code + case = live.load_case("feature-run") + case["tmp_lock"] = False + trace = json.loads((FIXTURES / "claude-code-never-started-trace.json").read_text()) + + def turn(run, text, index): + (run.root / "err").write_bytes(b"") + return {"index": index, "session_id": "s", "argv": ["claude", text], "exit_code": 1, "timed_out": False, + "duration_s": 4.7, "stream": str(FIXTURES / "claude-code-401-last-line.jsonl"), "stderr": str(run.root / "err")} + + output = io.StringIO() + with mock.patch.object(live, "load_case", return_value=case), mock.patch.object(claude_code, "prepare"), \ + mock.patch.object(claude_code, "turn", side_effect=turn), \ + mock.patch.object(claude_code, "harvest", return_value=trace), redirect_stdout(output): + with self.assertRaisesRegex(live.LoginRequired, "Run `claude auth login`"): + live.run_case("claude-code", case["id"], "HEAD", out, 0) + return json.loads(output.getvalue())["authorization_id"] + + def test_regrading_a_run_a_dead_login_ended_keeps_its_error_marks(self): + import report + with tempfile.TemporaryDirectory(prefix="pstack-live-regrade-") as tmp: + out = Path(tmp).resolve() + run_id = self.claude_feature_run_that_never_started(out) + with redirect_stdout(io.StringIO()): + code = live.main(["grade", run_id, "--out", str(out)]) + [root] = [p.parent for p in out.rglob("verdict.json")] + verdict = json.loads((root / "verdict.json").read_text()) + table = report.render(out, upstream=out) + self.assertEqual(code, 0) + self.assertEqual(verdict["error"]["relogin"], "claude auth login") + self.assertEqual({p: r["verdict"] for p, r in verdict["promises"].items() if not r.get("not_applicable")}, + {p: "ERROR" for p in verdict["promises"] if p != "persona-delivery-hermes-grok"}) + self.assertIn("claude-code pass rate: none graded (ERROR 10 excluded)", table) + + def test_a_harvest_refusal_after_an_auth_death_stops_the_batch_before_any_verdict(self): + from harnesses import claude_code + case = live.load_case("feature-run") + case["tmp_lock"] = False + + def turn(run, text, index): + (run.root / "err").write_bytes(b"") + return {"index": index, "session_id": "s", "argv": ["claude", text], "exit_code": 1, "timed_out": False, + "duration_s": 4.7, "stream": str(FIXTURES / "claude-code-401-last-line.jsonl"), "stderr": str(run.root / "err")} + + with tempfile.TemporaryDirectory(prefix="pstack-live-harvest-refused-") as tmp: + out = Path(tmp).resolve() + with mock.patch.object(live, "load_case", return_value=case), mock.patch.object(claude_code, "prepare"), \ + mock.patch.object(claude_code, "turn", side_effect=turn), \ + mock.patch.object(claude_code, "harvest", side_effect=GradeRefused("input_changed", "linked transcript")): + with self.assertRaisesRegex(GradeRefused, "linked transcript"): + live.run_case("claude-code", case["id"], "HEAD", out, 0) + written = sorted(p.name for p in out.rglob("*.json") if p.name in ("trace.json", "verdict.json")) + self.assertEqual(written, []) + + def test_regrading_a_run_whose_login_died_before_any_turn_is_refused_and_keeps_its_verdict(self): + from harnesses import codex + case = live.load_case("principle-steer-run") + case["tmp_lock"] = False + with tempfile.TemporaryDirectory(prefix="pstack-live-regrade-") as tmp: + out = Path(tmp).resolve() + output = io.StringIO() + with mock.patch.object(live, "load_case", return_value=case), redirect_stdout(output), \ + mock.patch.object(codex, "prepare", side_effect=LoginRequired(codex.LOGIN, "no login at auth.json")): + with self.assertRaises(live.LoginRequired): + live.run_case("codex", case["id"], "HEAD", out, 0) + [root] = [p.parent for p in out.rglob("verdict.json")] + before = (root / "verdict.json").read_bytes() + with redirect_stderr(io.StringIO()) as refusal: + code = live.main(["grade", json.loads(output.getvalue())["authorization_id"], "--out", str(out)]) + after = (root / "verdict.json").read_bytes() + self.assertEqual(code, 2) + self.assertIn("not sealed", refusal.getvalue()) + self.assertEqual(after, before) + self.assertEqual({r["verdict"] for r in json.loads(after)["promises"].values()}, {"ERROR"}) + + +class ErrorVerdict(unittest.TestCase): + def test_pass_and_fail_survive_an_auth_death_and_only_the_undecided_become_error(self): + from harnesses import codex + gated = "persona-delivery-hermes-grok" + case = {"id": "x", "promises": ["a", "b", "c", gated]} + graded = {"a": {"verdict": "PASS", "failures": [], "evidence": ["e"]}, + "b": {"verdict": "FAIL", "failures": ["f"], "evidence": []}, + "c": {"verdict": "INCONCLUSIVE", "failures": ["run killed"], "evidence": []}, + gated: {"verdict": "INCONCLUSIVE", "failures": ["applies to hermes and grok only"], "evidence": [], "not_applicable": True}} + verdict = error_verdict(case, "codex", "0" * 40, death_record(codex.LOGIN, "turn 0 exited 1"), graded) + self.assertEqual({p: r["verdict"] for p, r in verdict["promises"].items()}, + {"a": "PASS", "b": "FAIL", "c": "ERROR", gated: "INCONCLUSIVE"}) + self.assertEqual(verdict["promises"]["a"], graded["a"]) + self.assertTrue(verdict["promises"][gated]["not_applicable"]) + self.assertEqual(verdict["promises"]["c"]["failures"], ["codex login died: turn 0 exited 1"]) + self.assertEqual(verdict["error"]["relogin"], "codex login") + + def test_a_claude_run_that_never_started_keeps_the_hermes_and_grok_promise_out(self): + import oracles + import report + from harnesses import claude_code + case = live.load_case("feature-run") + trace = json.loads((FIXTURES / "claude-code-never-started-trace.json").read_text()) + graded = {pid: oracles.check(pid, trace, case, None) for pid in case["promises"]} + self.assertIn("run never started", graded["persona-delivery-hermes-grok"]["failures"][0]) + verdict = error_verdict(case, "claude-code", "0" * 40, death_record(claude_code.LOGIN, "turn 0 exited 1"), graded) + gated = {pid for pid, r in verdict["promises"].items() if r.get("not_applicable")} + self.assertEqual(gated, {"persona-delivery-hermes-grok"}) + self.assertEqual({r["verdict"] for pid, r in verdict["promises"].items() if pid not in gated}, {"ERROR"}) + with tempfile.TemporaryDirectory(prefix="pstack-error-verdict-") as tmp: + (Path(tmp) / "verdict.json").write_text(json.dumps(verdict)) + table = report.render(Path(tmp), upstream=Path(tmp)) + self.assertIn("claude-code pass rate: none graded (ERROR 10 excluded)", table) + self.assertNotIn("persona-delivery-hermes-grok", table) + + def test_an_undecided_promise_that_needs_a_judge_keeps_the_flag_and_excerpt_on_its_error(self): + import oracles + from harnesses import codex + case = {"id": "x", "promises": ["a"]} + graded = {"a": oracles.inconclusive("wording needs a judge", "seq 4", needs_judge=True, excerpt="the reply says pass")} + verdict = error_verdict(case, "codex", "0" * 40, death_record(codex.LOGIN, "turn 1 exited 1"), graded) + self.assertEqual(verdict["promises"]["a"], {"verdict": "ERROR", "failures": ["codex login died: turn 1 exited 1"], "evidence": [], + "needs_judge": True, "excerpt": "the reply says pass"}) + + def test_a_dead_codex_run_marks_error_only_the_promises_that_apply_to_codex(self): + import report + from harnesses import codex + case = live.load_case("feature-run") + verdict = error_verdict(case, "codex", "0" * 40, death_record(codex.LOGIN, "no login at auth.json")) + gated = {pid for pid, r in verdict["promises"].items() if r.get("not_applicable")} + self.assertEqual(gated, {"claude-effort-applies-via-effort-agents", "persona-delivery-hermes-grok"}) + self.assertEqual({r["verdict"] for pid, r in verdict["promises"].items() if pid not in gated}, {"ERROR"}) + self.assertEqual(len(case["promises"]) - len(gated), 9) + with tempfile.TemporaryDirectory(prefix="pstack-error-verdict-") as tmp: + (Path(tmp) / "verdict.json").write_text(json.dumps(verdict)) + table = report.render(Path(tmp), upstream=Path(tmp)) + self.assertIn("codex pass rate: none graded (ERROR 9 excluded)", table) + self.assertNotIn("claude-effort-applies-via-effort-agents", table) + + def test_every_promise_of_every_live_case_answers_the_applicability_probe_on_every_harness(self): + import report + from harnesses import codex + for case_id in sorted(p.name for p in live.CASES.iterdir() if (p / "case.json").is_file()): + case = live.load_case(case_id) + if case.get("deferred") or case.get("kind", "live") != "live": + continue + for harness in report.HARNESSES: + with self.subTest(case=case_id, harness=harness): + verdict = error_verdict(case, harness, "0" * 40, death_record(codex.LOGIN, "probe")) + self.assertEqual(set(verdict["promises"]), set(case["promises"])) + + class MakeProject(unittest.TestCase): def build(self, commit, branch=None, extra_steps=()): tmp = Path(tempfile.mkdtemp()) diff --git a/evals/promises/test_report.py b/evals/promises/test_report.py index 10e40ec2..a4c74ffc 100644 --- a/evals/promises/test_report.py +++ b/evals/promises/test_report.py @@ -27,6 +27,42 @@ def test_the_claude_skill_listing_state_of_each_run_is_counted(self): table = report.render(Path(out), upstream=Path(out)) self.assertEqual(table.splitlines()[-1], "claude-code skill listing: absent 1, installed 2") + def run_dir(self, out, name, harness, verdicts): + run = Path(out) / name + run.mkdir() + (run / "verdict.json").write_text(json.dumps({"harness": harness, "promises": { + pid: {"verdict": v, "failures": [], "evidence": []} for pid, v in verdicts.items()}})) + + def test_an_error_run_drops_out_of_the_pass_rate_and_keeps_its_cell(self): + with tempfile.TemporaryDirectory() as out: + self.run_dir(out, "a", "claude-code", {"p1": "PASS", "p2": "FAIL"}) + self.run_dir(out, "b", "claude-code", {"p1": "ERROR", "p2": "ERROR"}) + self.run_dir(out, "c", "claude-code", {"p3": "INCONCLUSIVE"}) + self.run_dir(out, "d", "codex", {"p1": "ERROR"}) + table = report.render(Path(out), upstream=Path(out)) + self.assertIn("claude-code pass rate: 1 of 3 (ERROR 2 excluded)", table) + self.assertIn("codex pass rate: none graded (ERROR 1 excluded)", table) + self.assertIn("| `p1` | ERROR 1, PASS 1 | ERROR |", table) + self.assertIn("ERROR 3, FAIL 1, INCONCLUSIVE 1, PASS 1", table) + + def test_a_run_whose_transcript_could_not_be_read_counts_only_the_promises_that_apply(self): + recorded = Path(__file__).resolve().parent / "testdata" / "harvest_refused" / "hermes-feature-run-verdict.json" + with tempfile.TemporaryDirectory() as out: + run = Path(out) / "tally" + run.mkdir() + (run / "verdict.json").write_bytes(recorded.read_bytes()) + table = report.render(Path(out), upstream=Path(out)) + self.assertIn("hermes pass rate: 0 of 10\n", table + "\n") + self.assertIn("| `persona-delivery-hermes-grok` | | | INCONCLUSIVE | | |", table) + self.assertNotIn("claude-effort-applies-via-effort-agents", table) + + def test_a_harness_with_no_error_prints_a_plain_pass_rate(self): + with tempfile.TemporaryDirectory() as out: + self.run_dir(out, "a", "grok", {"p1": "PASS", "p2": "PASS", "p3": "FAIL"}) + table = report.render(Path(out), upstream=Path(out)) + self.assertIn("grok pass rate: 2 of 3\n", table + "\n") + self.assertNotIn("excluded", table) + if __name__ == "__main__": unittest.main() diff --git a/evals/promises/testdata/auth_death/claude-code-401-last-line.jsonl b/evals/promises/testdata/auth_death/claude-code-401-last-line.jsonl new file mode 100644 index 00000000..c4dadd47 --- /dev/null +++ b/evals/promises/testdata/auth_death/claude-code-401-last-line.jsonl @@ -0,0 +1 @@ +{"duration_api_ms":833771,"stop_reason":"stop_sequence","session_id":"aa6aba22-2895-4986-849d-99d32e3c80a6","total_cost_usd":3.4189594,"usage":{"output_tokens_details":{"thinking_tokens":0},"input_tokens":0,"cache_creation_input_tokens":0,"cache_read_input_tokens":0,"output_tokens":0,"server_tool_use":{"web_search_requests":0,"web_fetch_requests":0},"service_tier":"standard","cache_creation":{"ephemeral_1h_input_tokens":0,"ephemeral_5m_input_tokens":0},"inference_geo":"","iterations":[],"speed":"standard","fallback_credit":null},"modelUsage":{"claude-opus-5-5":{"inputTokens":88,"outputTokens":57421,"cacheReadInputTokens":2174469,"cacheCreationInputTokens":158600,"webSearchRequests":0,"costUSD":2.6488437999999994,"contextWindow":1000000,"maxOutputTokens":128000,"thinkingTokens":28105,"canonicalModel":"claude-opus-5-5","provider":"firstParty","costBasis":"list"},"claude-sonnet-5-5":{"inputTokens":50,"outputTokens":36269,"cacheReadInputTokens":669728,"cacheCreationInputTokens":109352,"webSearchRequests":0,"costUSD":0.7701156,"contextWindow":1000000,"maxOutputTokens":128000,"thinkingTokens":15968,"canonicalModel":"claude-sonnet-5-5","provider":"firstParty","costBasis":"list"}},"permission_denials":[],"terminal_reason":"api_error","fast_mode_state":"off","fast_mode_disabled_reason":"sdk_opt_in_required","origin":{"kind":"task-notification","producer":"session-task"},"subagent_stats":{"spawned":6,"requested":{"background":6,"foreground":0,"unset":0},"started_in_background":6,"max_depth":1,"spawned_by_subagents":0,"completed":4,"failed":2,"killed":{"parent":0,"user":0,"system":0},"refused":{"depth_limit":0,"concurrency_limit":0,"budget":0},"by_type":{"general-purpose":5,"poteto-agent":1}},"is_error":true,"num_turns":1,"subtype":"success","api_error_status":401,"result":"Failed to authenticate. API Error: 401 OAuth access token has expired. Re-authenticate to continue.","type":"result","duration_ms":2352,"uuid":"26f6e691-27b1-4624-9c6e-674462ff2b16","queued_turn_count":0,"result_index":5} diff --git a/evals/promises/testdata/auth_death/claude-code-after-ipc-trace.json b/evals/promises/testdata/auth_death/claude-code-after-ipc-trace.json new file mode 100644 index 00000000..17d4c18b --- /dev/null +++ b/evals/promises/testdata/auth_death/claude-code-after-ipc-trace.json @@ -0,0 +1,125 @@ +{ + "harness": "claude-code", + "model": "claude-opus-5-5", + "exit_code": 1, + "duration_s": 582.6, + "events": [ + { + "seq": 0, + "turn": 0, + "kind": "user", + "text": "/poteto-mode imports from flaky partner feeds sometimes die halfway. retry each source with backoff, record the attempt count on every stored record, and let the cli set the retry policy. verify it end to end." + }, + { + "seq": 1, + "turn": 0, + "kind": "tool_call", + "name": "Bash", + "input": { + "command": "cat .claude/skills/poteto-mode/playbooks/feature.md && git log --oneline | head && git ls-files | grep -v '^.claude/' | head -50", + "description": "Read Feature playbook and list repo files" + }, + "id": "toolu_019JjbK79V23sQissTKH43Ww" + }, + { + "seq": 26, + "turn": 0, + "kind": "tool_call", + "name": "Agent", + "input": { + "description": "Architect runner 1 (fable)", + "subagent_type": "general-purpose", + "model": "fable", + "run_in_background": true, + "prompt": "You are candidate runner 1 in an architect design exploration. Read these files in full first, in this order:\n1. /private/tmp/pstack-collection-after-ipc-20261005/claude-code-after-ipc/relay-qb3al3g7/w/relay/.claude/skills/architect/SKILL.md (the workflow you're inside; you only do Phase B for one candidate)\n2. /private/tmp/pstack-collection-after-ipc-20261005/claude-code-after-ipc/relay-qb3al3g7/w/relay/.claude/skills/architect/references/runner-prompt.md (your instructions)\n3. /private/tmp/pstack-collection-after-ipc-20261005/claude-code-after-ipc/relay-qb3al3g7/w/relay/.claude/skills/architect/references/rationale-template.md and design-red-flags.md\n4. /private/tmp/pstack-collection-after-ipc-20261005/claude-code-after-ipc/relay-qb3al3g7/tmp/arena-relay-retry/grounding.md (the task and the traced subsystem)\nThen read the relay source files the grounding names (the repo is ~250 lines).\n\nRead-only on the repo: do not edit or write anything under the repo. Write your design package to /private/tmp/pstack-collection-after-ipc-20261005/claude-code-after-ipc/relay-qb3al3g7/tmp/arena-relay-retry/candidate-1/ as `design.md` (rationale per the template, including caller usage, the type/signature sketch with `raise NotImplementedError` bodies, the stored-file format, CLI flags and defaults, and an end-to-end verification plan through the real CLI). Answer every open design question in the grounding. Keep it to one or two pages.\n\nReply with at most 12 lines: the shape in brief, your answer to each open question, and the file path. No preamble." + }, + "id": "toolu_01Tj7EPNdeNAWNpL9TQw2se5" + }, + { + "seq": 28, + "turn": 0, + "kind": "tool_call", + "name": "Agent", + "input": { + "description": "Architect runner 2 (opus)", + "subagent_type": "general-purpose", + "model": "opus", + "run_in_background": true, + "prompt": "You are candidate runner 2 in an architect design exploration. Read these files in full first, in this order:\n1. /private/tmp/pstack-collection-after-ipc-20261005/claude-code-after-ipc/relay-qb3al3g7/w/relay/.claude/skills/architect/SKILL.md (the workflow you're inside; you only do Phase B for one candidate)\n2. /private/tmp/pstack-collection-after-ipc-20261005/claude-code-after-ipc/relay-qb3al3g7/w/relay/.claude/skills/architect/references/runner-prompt.md (your instructions)\n3. /private/tmp/pstack-collection-after-ipc-20261005/claude-code-after-ipc/relay-qb3al3g7/w/relay/.claude/skills/architect/references/rationale-template.md and design-red-flags.md\n4. /private/tmp/pstack-collection-after-ipc-20261005/claude-code-after-ipc/relay-qb3al3g7/tmp/arena-relay-retry/grounding.md (the task and the traced subsystem)\nThen read the relay source files the grounding names (the repo is ~250 lines).\n\nRead-only on the repo: do not edit or write anything under the repo. Write your design package to /private/tmp/pstack-collection-after-ipc-20261005/claude-code-after-ipc/relay-qb3al3g7/tmp/arena-relay-retry/candidate-2/ as `design.md` (rationale per the template, including caller usage, the type/signature sketch with `raise NotImplementedError` bodies, the stored-file format, CLI flags and defaults, and an end-to-end verification plan through the real CLI). Answer every open design question in the grounding. Keep it to one or two pages.\n\nReply with at most 12 lines: the shape in brief, your answer to each open question, and the file path. No preamble." + }, + "id": "toolu_015qrwepJ33JPD2xS4cUoP4o" + }, + { + "seq": 30, + "turn": 0, + "kind": "tool_call", + "name": "Agent", + "input": { + "description": "Architect runner 3 (sonnet)", + "subagent_type": "general-purpose", + "model": "sonnet", + "run_in_background": true, + "prompt": "You are candidate runner 3 in an architect design exploration. Read these files in full first, in this order:\n1. /private/tmp/pstack-collection-after-ipc-20261005/claude-code-after-ipc/relay-qb3al3g7/w/relay/.claude/skills/architect/SKILL.md (the workflow you're inside; you only do Phase B for one candidate)\n2. /private/tmp/pstack-collection-after-ipc-20261005/claude-code-after-ipc/relay-qb3al3g7/w/relay/.claude/skills/architect/references/runner-prompt.md (your instructions)\n3. /private/tmp/pstack-collection-after-ipc-20261005/claude-code-after-ipc/relay-qb3al3g7/w/relay/.claude/skills/architect/references/rationale-template.md and design-red-flags.md\n4. /private/tmp/pstack-collection-after-ipc-20261005/claude-code-after-ipc/relay-qb3al3g7/tmp/arena-relay-retry/grounding.md (the task and the traced subsystem)\nThen read the relay source files the grounding names (the repo is ~250 lines).\n\nRead-only on the repo: do not edit or write anything under the repo. Write your design package to /private/tmp/pstack-collection-after-ipc-20261005/claude-code-after-ipc/relay-qb3al3g7/tmp/arena-relay-retry/candidate-3/ as `design.md` (rationale per the template, including caller usage, the type/signature sketch with `raise NotImplementedError` bodies, the stored-file format, CLI flags and defaults, and an end-to-end verification plan through the real CLI). Answer every open design question in the grounding. Keep it to one or two pages.\n\nReply with at most 12 lines: the shape in brief, your answer to each open question, and the file path. No preamble." + }, + "id": "toolu_018tAPYCTZjALPprt8D9AR6r" + }, + { + "seq": 32, + "turn": 0, + "kind": "tool_call", + "name": "Agent", + "input": { + "description": "Architect runner 1 retry (session model)", + "subagent_type": "general-purpose", + "run_in_background": true, + "prompt": "You are candidate runner 1 in an architect design exploration. Read these files in full first, in this order:\n1. /private/tmp/pstack-collection-after-ipc-20261005/claude-code-after-ipc/relay-qb3al3g7/w/relay/.claude/skills/architect/SKILL.md (the workflow you're inside; you only do Phase B for one candidate)\n2. /private/tmp/pstack-collection-after-ipc-20261005/claude-code-after-ipc/relay-qb3al3g7/w/relay/.claude/skills/architect/references/runner-prompt.md (your instructions)\n3. /private/tmp/pstack-collection-after-ipc-20261005/claude-code-after-ipc/relay-qb3al3g7/w/relay/.claude/skills/architect/references/rationale-template.md and design-red-flags.md\n4. /private/tmp/pstack-collection-after-ipc-20261005/claude-code-after-ipc/relay-qb3al3g7/tmp/arena-relay-retry/grounding.md (the task and the traced subsystem)\nThen read the relay source files the grounding names (the repo is ~250 lines).\n\nAnother runner on your model is designing in parallel. To widen the design space, take the interpretation that `import` accepts several feeds per invocation, each source retried independently, and design the whole shape (types, per-source outcome, partial-failure semantics, exit status, stored format) around that. If you conclude that interpretation is wrong, say so with a reason and still sketch it as your alternative.\n\nRead-only on the repo: do not edit or write anything under the repo. Write your design package to /private/tmp/pstack-collection-after-ipc-20261005/claude-code-after-ipc/relay-qb3al3g7/tmp/arena-relay-retry/candidate-1/ as `design.md` (rationale per the template, including caller usage, the type/signature sketch with `raise NotImplementedError` bodies, the stored-file format, CLI flags and defaults, and an end-to-end verification plan through the real CLI). Answer every open design question in the grounding. Keep it to one or two pages.\n\nReply with at most 12 lines: the shape in brief, your answer to each open question, and the file path. No preamble." + }, + "id": "toolu_01GMyqerq3x2U8YSzfCTyNoS" + } + ], + "files_read": [], + "worklist": [], + "spawns": [ + { + "seq": 26, + "turn": 0, + "tool": "Agent", + "subagent_type": "general-purpose", + "persona": "general-purpose", + "model": "fable", + "effort": null, + "prompt_head": "You are candidate runner 1 in an architect design exploration. Read these files in full first, in this order:\n1. /private/tmp/pstack-collection-after-ipc-20261005/claude-code-after-ipc/relay-qb3al3g7/w/relay/.claude/skills/architect/SKILL.md (the workflow you're inside; you only do Phase B for one c" + }, + { + "seq": 28, + "turn": 0, + "tool": "Agent", + "subagent_type": "general-purpose", + "persona": "general-purpose", + "model": "opus", + "effort": null, + "prompt_head": "You are candidate runner 2 in an architect design exploration. Read these files in full first, in this order:\n1. /private/tmp/pstack-collection-after-ipc-20261005/claude-code-after-ipc/relay-qb3al3g7/w/relay/.claude/skills/architect/SKILL.md (the workflow you're inside; you only do Phase B for one c" + }, + { + "seq": 30, + "turn": 0, + "tool": "Agent", + "subagent_type": "general-purpose", + "persona": "general-purpose", + "model": "sonnet", + "effort": null, + "prompt_head": "You are candidate runner 3 in an architect design exploration. Read these files in full first, in this order:\n1. /private/tmp/pstack-collection-after-ipc-20261005/claude-code-after-ipc/relay-qb3al3g7/w/relay/.claude/skills/architect/SKILL.md (the workflow you're inside; you only do Phase B for one c" + }, + { + "seq": 32, + "turn": 0, + "tool": "Agent", + "subagent_type": "general-purpose", + "persona": "general-purpose", + "model": null, + "effort": null, + "prompt_head": "You are candidate runner 1 in an architect design exploration. Read these files in full first, in this order:\n1. /private/tmp/pstack-collection-after-ipc-20261005/claude-code-after-ipc/relay-qb3al3g7/w/relay/.claude/skills/architect/SKILL.md (the workflow you're inside; you only do Phase B for one c" + } + ], + "final_reply": "Failed to authenticate. API Error: 401 OAuth access token has expired. Re-authenticate to continue." +} \ No newline at end of file diff --git a/evals/promises/testdata/auth_death/claude-code-never-started-trace.json b/evals/promises/testdata/auth_death/claude-code-never-started-trace.json new file mode 100644 index 00000000..510352d4 --- /dev/null +++ b/evals/promises/testdata/auth_death/claude-code-never-started-trace.json @@ -0,0 +1,27 @@ +{ + "harness": "claude-code", + "cli_version": "2.1.289", + "model": "claude-opus-5-5", + "effort": "high", + "exit_code": 1, + "duration_s": 4.7, + "entry": "injected", + "events": [ + { + "seq": 0, + "turn": 0, + "kind": "user", + "text": "/poteto-mode add a --json flag to this command. text output stays byte-identical. verify both." + }, + { + "seq": 1, + "turn": 0, + "kind": "text", + "text": "Failed to authenticate. API Error: 401 OAuth access token has expired. Re-authenticate to continue." + } + ], + "files_read": [], + "worklist": [], + "spawns": [], + "final_reply": "Failed to authenticate. API Error: 401 OAuth access token has expired. Re-authenticate to continue." +} diff --git a/evals/promises/testdata/auth_death/grok-external-auth.stderr.txt b/evals/promises/testdata/auth_death/grok-external-auth.stderr.txt new file mode 100644 index 00000000..997b2955 --- /dev/null +++ b/evals/promises/testdata/auth_death/grok-external-auth.stderr.txt @@ -0,0 +1,28 @@ +Error: Internal error: { + "message": "Unauthorized (401) from https://cli-chat-proxy.grok.com/v1/responses: Invalid or expired credentials (auth_kind=none, x_xai_token_auth=xai-grok-cli, upstream=Unauthenticated, reason=no auth context)\n\n Model: grok-4.7\n Auth: External\n Version: 1.0.41\n Available: grok-4.7, grok-4.7-build-fast, grok-4.6, grok-4.5\n\nYour session expired and your sign-in helper could not renew it in the background. Run /login to sign in again.", + "promptUsage": { + "inputTokens": 1444814, + "outputTokens": 140967, + "totalTokens": 1585781, + "cachedReadTokens": 1104640, + "cacheCreationTokens": 0, + "reasoningTokens": 100924, + "modelCalls": 37, + "apiDurationMs": 1935728, + "costUsdTicks": 7066798000, + "modelUsage": { + "grok-4.7-build": { + "inputTokens": 1444814, + "outputTokens": 140967, + "totalTokens": 1585781, + "cachedReadTokens": 1104640, + "cacheCreationTokens": 0, + "reasoningTokens": 100924, + "modelCalls": 37, + "apiDurationMs": 1935728, + "costUsdTicks": 7066798000 + } + }, + "numTurns": 12 + } +} diff --git a/evals/promises/testdata/auth_death/grok-external-auth.stream.json b/evals/promises/testdata/auth_death/grok-external-auth.stream.json new file mode 100644 index 00000000..68b1f7db --- /dev/null +++ b/evals/promises/testdata/auth_death/grok-external-auth.stream.json @@ -0,0 +1 @@ +{"type":"error","message":"Internal error: {\n \"message\": \"Unauthorized (401) from https://cli-chat-proxy.grok.com/v1/responses: Invalid or expired credentials (auth_kind=none, x_xai_token_auth=xai-grok-cli, upstream=Unauthenticated, reason=no auth context)\\n\\n Model: grok-4.7\\n Auth: External\\n Version: 1.0.41\\n Available: grok-4.7, grok-4.7-build-fast, grok-4.6, grok-4.5\\n\\nYour session expired and your sign-in helper could not renew it in the background. Run /login to sign in again.\",\n \"promptUsage\": {\n \"inputTokens\": 1444814,\n \"outputTokens\": 140967,\n \"totalTokens\": 1585781,\n \"cachedReadTokens\": 1104640,\n \"cacheCreationTokens\": 0,\n \"reasoningTokens\": 100924,\n \"modelCalls\": 37,\n \"apiDurationMs\": 1935728,\n \"costUsdTicks\": 7066798000,\n \"modelUsage\": {\n \"grok-4.7-build\": {\n \"inputTokens\": 1444814,\n \"outputTokens\": 140967,\n \"totalTokens\": 1585781,\n \"cachedReadTokens\": 1104640,\n \"cacheCreationTokens\": 0,\n \"reasoningTokens\": 100924,\n \"modelCalls\": 37,\n \"apiDurationMs\": 1935728,\n \"costUsdTicks\": 7066798000\n }\n },\n \"numTurns\": 12\n }\n}","usage":{"input_tokens":340174,"cache_read_input_tokens":1104640,"cache_creation_input_tokens":0,"output_tokens":140967,"reasoning_tokens":100924,"total_tokens":1585781},"num_turns":12,"total_cost_usd":0.7066798,"total_cost_usd_ticks":7066798000,"modelUsage":{"grok-4.7-build":{"inputTokens":340174,"outputTokens":140967,"cacheReadInputTokens":1104640,"cacheCreationInputTokens":0,"modelCalls":37,"costUSD":0.7066798}}} diff --git a/evals/promises/testdata/harvest_refused/hermes-feature-run-verdict.json b/evals/promises/testdata/harvest_refused/hermes-feature-run-verdict.json new file mode 100644 index 00000000..8b078bdf --- /dev/null +++ b/evals/promises/testdata/harvest_refused/hermes-feature-run-verdict.json @@ -0,0 +1,84 @@ +{ + "case": "feature-run", + "harness": "hermes", + "skills_at": "57e4409a6c4ebb7dfd27a5118caf478ef64bd769", + "promises": { + "poteto-mode-routes-to-playbook": { + "verdict": "INCONCLUSIVE", + "failures": [ + "the harness transcript could not be read: path-refused: recorded read crosses an unavailable fixture prefix" + ], + "evidence": [] + }, + "worklist-opens-with-playbook-steps": { + "verdict": "INCONCLUSIVE", + "failures": [ + "the harness transcript could not be read: path-refused: recorded read crosses an unavailable fixture prefix" + ], + "evidence": [] + }, + "worklist-in-native-task-tool": { + "verdict": "INCONCLUSIVE", + "failures": [ + "the harness transcript could not be read: path-refused: recorded read crosses an unavailable fixture prefix" + ], + "evidence": [] + }, + "skipped-step-stays-in-worklist": { + "verdict": "INCONCLUSIVE", + "failures": [ + "the harness transcript could not be read: path-refused: recorded read crosses an unavailable fixture prefix" + ], + "evidence": [] + }, + "reply-carries-commands-and-outputs": { + "verdict": "INCONCLUSIVE", + "failures": [ + "the harness transcript could not be read: path-refused: recorded read crosses an unavailable fixture prefix" + ], + "evidence": [] + }, + "reply-says-inconclusive-when-check-cannot-run": { + "verdict": "INCONCLUSIVE", + "failures": [ + "the harness transcript could not be read: path-refused: recorded read crosses an unavailable fixture prefix" + ], + "evidence": [] + }, + "prove-it-works-checks-real-artifact": { + "verdict": "INCONCLUSIVE", + "failures": [ + "the harness transcript could not be read: path-refused: recorded read crosses an unavailable fixture prefix" + ], + "evidence": [] + }, + "claude-effort-applies-via-effort-agents": { + "verdict": "INCONCLUSIVE", + "failures": [ + "the harness transcript could not be read: path-refused: recorded read crosses an unavailable fixture prefix" + ], + "evidence": [] + }, + "persona-delivery-hermes-grok": { + "verdict": "INCONCLUSIVE", + "failures": [ + "the harness transcript could not be read: path-refused: recorded read crosses an unavailable fixture prefix" + ], + "evidence": [] + }, + "poteto-mode-reads-principles-and-names-applied": { + "verdict": "INCONCLUSIVE", + "failures": [ + "the harness transcript could not be read: path-refused: recorded read crosses an unavailable fixture prefix" + ], + "evidence": [] + }, + "design-ladder-spares-small-changes": { + "verdict": "INCONCLUSIVE", + "failures": [ + "the harness transcript could not be read: path-refused: recorded read crosses an unavailable fixture prefix" + ], + "evidence": [] + } + } +}