fix(evals): handle login failures in live promise evaluations

evals/README.md

@@ -10,9 +10,9 @@ each harness. See `promises/README.md`.
 
 Model answers, traces, judge transcripts, reports, and private cases stay
 outside Git. A committed case is tuning material, not sealed evidence.
-Oracle regression fixtures under `promises/testdata/` are an exception. They
-are reduced, sanitized traces that keep only what the oracle tests read, and
-`promises/reduce_trace.py` rebuilds one from a raw trace. `promises/results/`
+Oracle and login regression fixtures under `promises/testdata/` are an exception.
+They are reduced, sanitized traces, CLI error records, and recorded verdicts used by model-free tests.
+`promises/reduce_trace.py` reduces raw traces for oracle fixtures. `promises/results/`
 is a second exception. It holds aggregated verdict matrices with no model
 answers.
 

evals/promises/README.md

@@ -71,7 +71,11 @@ python3 evals/promises/live.py run --harness <claude-code|codex|hermes|grok> --c
 python3 evals/promises/live.py report <dir> --upstream-ref upstream/main
 ```
 
-Each run makes a fresh git repository from `fixtures/<name>/` and `histories/<name>/`. Each history step is committed in order, except a step with `"commit": false`, which stays as uncommitted work in the tree. A step with `"branch"` switches to that branch first, creating it at the current commit when it does not exist, so later steps land there and `main` keeps the fixture. It installs `skills/` from `--skills-at` (default `HEAD`) where the harness discovers project skills, and runs the case's turns headless. Each run directory gets `run.json`, `trace.json`, and `verdict.json`. Every adapter writes the same trace schema, with events, files read, worklist snapshots, spawns, and the final reply. A tool event carries its harness call id, so each result pairs with its own call. `oracles.py` grades each promise as `PASS`, `FAIL`, or `INCONCLUSIVE`. A verdict is `INCONCLUSIVE` when the trace cannot show the behavior, or when the harness reports `x_host_skill_hits`, which means the user's own skills touched the run. Some oracles also return `INCONCLUSIVE` when only a judge can decide, such as whether a reply's wording reports a review `pass`. That verdict carries `needs_judge: true` and an `excerpt` of the graded reply in `verdict.json`. In a multi-turn case the graded reply can come from an earlier turn than `final_reply`, and the verdict's evidence names that turn. When the excerpt is not enough, read that turn's text events in `trace.json` before you tally the run.
+Each run makes a fresh git repository from `fixtures/<name>/` and `histories/<name>/`. Each history step is committed in order, except a step with `"commit": false`, which stays as uncommitted work in the tree. A step with `"branch"` switches to that branch first, creating it at the current commit when it does not exist, so later steps land there and `main` keeps the fixture. It installs `skills/` from `--skills-at` (default `HEAD`) where the harness discovers project skills, and runs the case's turns headless. A run that completes harvest and grading gets `run.json`, `trace.json`, and `verdict.json`. Every adapter writes the same trace schema, with events, files read, worklist snapshots, spawns, and the final reply. A tool event carries its harness call id, so each result pairs with its own call. `oracles.py` grades each promise as `PASS`, `FAIL`, or `INCONCLUSIVE`, and `live.py` alone adds `ERROR` for a dead login. A verdict is `INCONCLUSIVE` when the trace cannot show the behavior, or when the harness reports `x_host_skill_hits`, which means the user's own skills touched the run. Some oracles also return `INCONCLUSIVE` when only a judge can decide, such as whether a reply's wording reports a review `pass`. That verdict carries `needs_judge: true` and an `excerpt` of the graded reply in `verdict.json`. In a multi-turn case the graded reply can come from an earlier turn than `final_reply`, and the verdict's evidence names that turn. When the excerpt is not enough, read that turn's text events in `trace.json` before you tally the run.
+
+Before the first case, `live.py run` checks the selected Claude Code, Codex, or Grok login at no model cost. A failed check stops the wave with exit 3 before any case runs, and the message names the CLI and its login command. Claude Code runs `claude auth status` with the host `HOME`, `USER`, a minimal `PATH`, and an empty `CLAUDE_SECURESTORAGE_CONFIG_DIR`, so a host-exported `ANTHROPIC_API_KEY` cannot mask a missing login. It does not set the private `CLAUDE_CONFIG_DIR` each turn gets, since that directory exists only after a run is prepared, so it checks the host login. Codex needs the `auth.json` file and a passing `codex login status`. Grok refuses a host token that expires within five minutes, because the run's read-only provider never refreshes it. Claude Code and Codex refresh their own access tokens when they run, so their checks do not read token expiry. Hermes has no preflight check, and its preparation step still refuses a missing `auth.json`.
+
+For Claude Code, Codex, and Grok, a turn counts as an auth death when it exited nonzero without timing out and the tail of its stderr or the last nonempty line of its stream carries that CLI's login-failure text. A `LoginRequired` failure during preparation stops the run before any turn. `live.py` writes a `verdict.json` that marks every promise that applies to the harness `ERROR` and records the CLI and its login command under `error`. It skips harvest and grading, so that run has no `trace.json`. When a turn dies on auth, `live.py` still harvests and grades the run. A `PASS` or `FAIL` the grader decides stands, and every other applicable promise becomes `ERROR`, keeping any `needs_judge` flag and `excerpt`. After writing either verdict, the wave stops with exit 3. A harvest refusal after an auth death propagates unchanged and stops the batch before the run gets `trace.json` or `verdict.json`. Running `live.py grade` on a sealed auth-death run later re-grades it from its trace, keeps the `error` record, and marks `ERROR` again on every promise the new grade leaves undecided. A run whose login died before any turn has no sealed trace, so `live.py grade` refuses it with exit 2 and leaves its verdict unchanged. In each of these verdicts a promise that does not apply to the harness is marked `not_applicable`, so it stays out of the report. `live.py report` shows `ERROR` in the table and leaves those verdicts out of each harness's `pass rate: <PASS> of <graded>` line. This rate counts applicable promise verdicts, not runs. `live.py` and `live.py report` decide applicability by asking each oracle about an empty trace from the harness, because a grade that returns early carries no `not_applicable` mark. A grade returns early when the run never started, its transcript could not be read, or host skills touched it. `INCONCLUSIVE` stays in the denominator. When every applicable verdict is `ERROR`, the rate reads `none graded`. A turn that exited 0 still grades as before, even if it contains login-failure text.
 
 `--model` and `--effort` set the model and reasoning effort for every turn of every case in one `run` invocation, on Claude Code and Codex. Each turn's recorded argv carries both values, and the sealed run metadata keeps its fields. Other adapters refuse the flags.
 
@@ -103,7 +107,7 @@ A completed run prints `authorization_id`. To regrade that registered run, use t
 python3 evals/promises/live.py grade --out <original-out> <authorization-id>
 ```
 
-The grade CLI exits 2 and writes a refusal receipt to stderr when the ID is unknown or grading refuses. A successful grade returns the existing verdict dictionary and writes `verdict.json`. It preserves the JSON indentation and final newline. The Python entry point is `live.grade(authorization)`. It raises `GradeRefused` for infrastructure and unsafe-I/O refusals. A refusal has no promise verdict. Its separate receipt and worker logs remain under the controller's run and attempt directories.
+The grade CLI exits 2 and writes a refusal receipt to stderr when the ID is unknown or grading refuses. A successful grade returns the existing verdict dictionary and writes `verdict.json`. It preserves the JSON indentation and final newline. The Python entry point is `live.grade(authorization)`. The CLI calls it through `live.grade_run(authorization)`, which also keeps a dead login's `error` record and its `ERROR` marks, as [Run live cases](#run-live-cases) describes. It raises `GradeRefused` for infrastructure and unsafe-I/O refusals. A refusal has no promise verdict. Its separate receipt and worker logs remain under the controller's run and attempt directories.
 
 Missing sealed `run.json`, `trace.json`, or retained prior `verdict.json` produces an `input_changed` refusal and an attempt receipt. The refusal leaves any existing output verdict unchanged. An intentionally absent project keeps its oracle result, and an absent output verdict still permits first publication.
 

evals/promises/harnesses/claude_code.py

@@ -15,11 +15,15 @@
 from pathlib import Path
 
 import live
+from harnesses.login import Login, LoginRequired, check_status
 
 SKILLS_DIR = ".claude/skills"
 PRIVATE_DIRS = (".claude/",)
 SHARES_HOST_TMP = False
 
+LOGIN = Login("claude", "claude auth login", re.compile(
+    r"Please run /login|Not logged in|Login expired|OAuth token (?:revoked|has expired)|Invalid API key|"
+    r"Failed to authenticate|Authentication required", re.I))
 DEFAULT_EFFORT = "high"
 SETTING_SOURCES = "user,project"
 LISTING_SCRIPT = Path("setup-pstack/scripts/skill-listing.py")
@@ -196,6 +200,23 @@ def store(run):
     return config_dir(run) / "projects"
 
 
+def preflight():
+    """Check the host login before a private turn config exists. Keep the turn's host HOME and Keychain namespace,
+    but omit exported API keys and config overrides so they cannot mask a missing login."""
+    binary = CLAUDE_BINARY if CLAUDE_BINARY.is_file() else shutil.which("claude")
+    if not binary:
+        raise LoginRequired(LOGIN, "no claude binary to ask")
+    env = {"HOME": os.environ["HOME"], "USER": os.environ.get("USER", ""), "LOGNAME": os.environ.get("LOGNAME", ""),
+           "PATH": "/usr/bin:/bin", "LANG": "en_US.UTF-8", "TERM": "dumb", "CLAUDE_SECURESTORAGE_CONFIG_DIR": ""}
+    out = check_status(LOGIN, [binary, "auth", "status"], env=env)
+    try:
+        logged_in = json.loads(out).get("loggedIn")
+    except ValueError:
+        logged_in = None
+    if logged_in is not True:
+        raise LoginRequired(LOGIN, f"`claude auth status` reports loggedIn {logged_in!r}")
+
+
 def prepare(run):
     state = _bind_paths(run, _host_runtime())
     script = state.paths.project / SKILLS_DIR / "pstack-harness/scripts/subagents.py"

evals/promises/harnesses/codex.py

@@ -9,6 +9,7 @@
 
 import live
 from grade_boundary import GradeRefused, read_file, write_file
+from harnesses.login import Login, LoginRequired, check_status
 
 SKILLS_DIR = ".agents/skills"
 PRIVATE_DIRS = [".agents/", ".codex/"]
@@ -19,6 +20,9 @@
 EFFORT = "high"
 COMPACT = "/compact"
 COMPACT_TOKEN_LIMIT = 2000
+LOGIN = Login("codex", "codex login", re.compile(
+    r"access token could not be refreshed|refresh token was (?:already used|rejected)|log out and sign in again|"
+    r"authentication session could not be refreshed|Not logged in", re.I))
 ENV_KEPT = ("PATH", "LANG", "LC_ALL", "LC_CTYPE", "LC_MESSAGES", "TERM", "USER", "LOGNAME", "SHELL", "TZ",
             "SSL_CERT_FILE", "SSL_CERT_DIR", "HTTP_PROXY", "HTTPS_PROXY", "NO_PROXY",
             "http_proxy", "https_proxy", "no_proxy")
@@ -57,13 +61,26 @@ def resolve_binary(env):
     raise RuntimeError(f"no working codex binary; rejected: {json.dumps(rejected)}")
 
 
+def host_auth():
+    return Path(os.environ.get("CODEX_HOME") or Path.home() / ".codex") / "auth.json"
+
+
+def preflight():
+    """Ask the CLI whether it holds a login. The CLI refreshes an expired access token itself when it runs, so the token's
+    own expiry is not checked here."""
+    auth = host_auth()
+    if not auth.is_file():
+        raise LoginRequired(LOGIN, f"no login at {auth}")
+    check_status(LOGIN, [resolve_binary(dict(os.environ))["path"], "login", "status"])
+
+
 def prepare(run):
     root = run.root
     for name in ("home", "codex-home", "tmp", "transcripts"):
         (root / name).mkdir(exist_ok=True)
-    auth = Path(os.environ.get("CODEX_HOME") or Path.home() / ".codex") / "auth.json"
+    auth = host_auth()
     if not auth.is_file():
-        raise RuntimeError(f"no codex login at {auth}")
+        raise LoginRequired(LOGIN, f"no login at {auth}")
     link = root / "codex-home" / "auth.json"
     if not link.is_symlink():
         link.symlink_to(auth)

evals/promises/harnesses/grok.py

@@ -11,6 +11,7 @@
 
 import live
 from grade_boundary import GradeRefused, copy_file
+from harnesses.login import Login, LoginRequired
 
 SKILLS_DIR = ".agents/skills"
 PRIVATE_DIRS = [".agents/"]
@@ -132,20 +133,31 @@ def host_token_expiry(auth):
     return datetime.datetime.fromisoformat(entry["expires_at"].replace("Z", "+00:00")) if entry else None
 
 
+LOGIN = Login("grok", "grok login", re.compile(
+    r"Run `grok login`|auth token is invalid or expired|Not authenticated|Authentication required|no xAI credential|"
+    r"Invalid or expired credentials|sign-in helper could not renew", re.I))
+TOKEN_MARGIN = datetime.timedelta(minutes=5)
+
+
+def preflight():
+    """Refuse a missing or expired host token. The run's read-only provider never refreshes it, so only `grok login` on the host does."""
+    auth = host_grok_home() / "auth.json"
+    if not auth.is_file():
+        raise LoginRequired(LOGIN, f"no login at {auth}")
+    expiry = host_token_expiry(auth)
+    if expiry is not None and expiry <= datetime.datetime.now(datetime.timezone.utc) + TOKEN_MARGIN:
+        raise LoginRequired(LOGIN, f"the host token expired at {expiry:%Y-%m-%dT%H:%MZ}, and the run's read-only provider never refreshes it")
+
+
 def prepare(run):
     root = run.root
     for name in ("home", "grok-home", "tmp", "transcripts"):
         (root / name).mkdir(exist_ok=True)
     for name in HOME_DIRS:
         (home(run) / name).mkdir(exist_ok=True)
     (home(run) / ".gitconfig").write_text(GITCONFIG)
+    preflight()
     auth = host_grok_home() / "auth.json"
-    if not auth.is_file():
-        raise RuntimeError(f"no grok login at {auth}")
-    expiry = host_token_expiry(auth)
-    if expiry is not None and expiry <= datetime.datetime.now(datetime.timezone.utc) + datetime.timedelta(minutes=5):
-        raise RuntimeError(f"the host Grok token expired at {expiry:%Y-%m-%dT%H:%MZ}; run `grok login` on the host, "
-                           "since the run's read-only provider never refreshes it")
     provider = root / "auth-provider.py"
     provider.write_text(AUTH_PROVIDER.format(python=shutil.which("python3") or "/usr/bin/python3", auth=str(auth)))
     provider.chmod(0o755)
@@ -159,7 +171,7 @@ def prepare(run):
                            text=True, timeout=120)
     (root / "login.log").write_text(login.stdout + login.stderr)
     if login.returncode != 0:
-        raise RuntimeError(f"grok login failed under the run's own home: {(login.stdout + login.stderr)[-300:]}")
+        raise LoginRequired(LOGIN, f"`grok login` failed under the run's own home: {(login.stdout + login.stderr)[-300:]}")
     inspected = subprocess.run([launch["path"], "inspect", "--json"], cwd=run.project, env=env,
                                capture_output=True, text=True, timeout=120, check=True)
     (root / "inspect.json").write_text(inspected.stdout)

evals/promises/harnesses/login.py

@@ -0,0 +1,86 @@
+"""What a live wave needs from each CLI's login, shared by the adapters and live.py."""
+
+import re
+import subprocess
+from dataclasses import dataclass
+from pathlib import Path
+
+from grade_boundary import read_file
+
+ERROR = "ERROR"
+DECIDED = ("PASS", "FAIL")
+STATUS_TIMEOUT_S = 60
+DEATH_WINDOW = 4000
+
+
+@dataclass(frozen=True)
+class Login:
+    cli: str
+    relogin: str
+    death: re.Pattern
+
+
+class LoginRequired(RuntimeError):
+    def __init__(self, login, detail):
+        super().__init__(f"{login.cli} login is not usable: {detail}. Run `{login.relogin}`, then start the wave again")
+        self.login = login
+        self.detail = detail
+
+
+def check_status(login, argv, env=None):
+    """Run the CLI's own login status command. It spends no model tokens."""
+    try:
+        done = subprocess.run([str(a) for a in argv], env=env, capture_output=True, text=True, timeout=STATUS_TIMEOUT_S,
+                              stdin=subprocess.DEVNULL)
+    except (OSError, subprocess.TimeoutExpired) as error:
+        raise LoginRequired(login, f"`{' '.join(map(str, argv))}` did not finish: {error!r}") from error
+    if done.returncode:
+        raise LoginRequired(login, f"`{' '.join(map(str, argv))}` exited {done.returncode}: {(done.stdout + done.stderr).strip()[-300:]}")
+    return done.stdout
+
+
+def auth_death(login, record):
+    """The auth-failure text that ended a turn, or None. Only a turn that exited nonzero counts, and only its stderr tail
+    and last stream line, so a reply that quotes a login message in a run that finished is not read as a dead login."""
+    if not record.get("exit_code") or record.get("timed_out"):
+        return None
+    tail = []
+    if record.get("stderr"):
+        tail.append(_read(record["stderr"])[-DEATH_WINDOW:])
+    if record.get("stream"):
+        lines = [line for line in _read(record["stream"]).splitlines() if line.strip()]
+        tail.append(lines[-1][-DEATH_WINDOW:] if lines else "")
+    found = login.death.search("\n".join(tail))
+    return found.group(0) if found else None
+
+
+def _read(path):
+    try:
+        return read_file(Path(path)).decode(errors="replace")
+    except FileNotFoundError:
+        return ""
+
+
+def death_record(login, detail):
+    return {"cli": login.cli, "relogin": login.relogin, "detail": detail}
+
+
+def error_verdict(case, harness, skills_at, error, graded=None):
+    reason = f"{error['cli']} login died: {error['detail']}"
+    promises = {}
+    for pid in case["promises"]:
+        result = graded[pid] if graded else {}
+        gate = applicability(pid, harness)
+        if gate.get("not_applicable"):
+            promises[pid] = gate
+        elif result.get("verdict") in DECIDED:
+            promises[pid] = result
+        else:
+            judge = {k: result[k] for k in ("needs_judge", "excerpt") if k in result}
+            promises[pid] = {"verdict": ERROR, "failures": [reason], "evidence": [], **judge}
+    return {"case": case["id"], "harness": harness, "skills_at": skills_at, "error": error, "promises": promises}
+
+
+def applicability(pid, harness):
+    import oracles
+    return oracles.check(pid, {"harness": harness}, None, None)