fix(evals): handle login failures in live promise evaluations by mdsmithaustin · Pull Request #198 · mdsmithaustin/pstack

mdsmithaustin · 2026-10-12T19:26:46Z

What Changed

Add login preflight checks for Claude Code, Codex, and Grok; stop live evaluation waves with exit 3 and a login command when authentication fails.
Record applicable, undecided promises as ERROR after login failures, preserve decided PASS/FAIL results, and retain authentication errors when regrading.
Show ERROR counts in reports, exclude them from pass rates, and filter out promises that do not apply to each harness.

Validation

Risk Assessment

✅ Low: The change is limited to eval tooling and does exactly the four round-2 items: applicability comes from an empty-trace probe, the report counts only applicable promises, a re-grade keeps the stored error record, and an ERROR keeps needs_judge and its excerpt. Tracing the never-started, re-grade, unsealed-refusal and harvest-refusal paths found no wrong result, and oracles.py is untouched.

Testing

Targeted tests passed, and seven scenarios passed through real grading/reporting code using isolated recorded and synthetic inputs. CLI transcripts and persisted verdicts provide evidence. No paid wave, full suite, or other pipeline phase ran; disposable files were removed.

Live validation: ✅ go - 7 of 7 scenarios driven live against the product

Scenario	Result	Live	Evidence
Grade and re-grade a never-started Claude run: retain auth error, exclude the gated promise, and report ten ERROR exclusions	✅ pass	live	never-started-verdict.json; never-started-regrade.txt; never-started-report.txt
Grade an unreadable transcript: only applicable promises become ERROR and remain ERROR after re-grading	✅ pass	live	unreadable-verdict.json; unreadable-regrade.txt; unreadable-report.txt
Grade a trace containing host skills: exclude gated promises despite the early grading return	✅ pass	live	host-skills-verdict.json; host-skills-regrade.txt; host-skills-report.txt
Report the recorded Hermes refused-read verdict: display 0 of 10 and omit the Claude-only promise	✅ pass	live	hermes-report.txt
Re-grade an auth-death run with an ambiguous review: preserve needs_judge and excerpt on ERROR while keeping decided FAIL	✅ pass	live	judge-verdict.json; judge-regrade.txt
Re-grade a recorded run with completed architectural behavior: preserve its decided PASS and stored auth error	✅ pass	live	decided-pass-verdict.json; decided-pass-regrade.txt
Attempt to re-grade an auth-death run without a sealed trace: exit 2 and leave verdict bytes unchanged	✅ pass	live	unsealed-refusal.txt; scenario-results.json

Evidence: Real grade/report CLI transcripts

Scenario: never-started-regrade
$ ~/.local/share/mise/installs/python/3.14/bin/python3 evals/promises/live.py grade 6b30506dc380413daddda2ef1e13591a --out ~/.no-mistakes/worktrees/fbee15f974d9/01M4RZVNV9H2M034P4M7JDTZ1D/.phase-test/data/never-started
exit: 0
{"case": "feature-run", "harness": "claude-code", "skills_at": "HEAD", "error": {"cli": "claude", "relogin": "claude auth login", "detail": "recorded turn exited 1 on OAuth 401"}, "promises": {"poteto-mode-routes-to-playbook": {"verdict": "ERROR", "failures": ["claude login died: recorded turn exited 1 on OAuth 401"], "evidence": []}, "worklist-opens-with-playbook-steps": {"verdict": "ERROR", "failures": ["claude login died: recorded turn exited 1 on OAuth 401"], "evidence": []}, "worklist-in-native-task-tool": {"verdict": "ERROR", "failures": ["claude login died: recorded turn exited 1 on OAuth 401"], "evidence": []}, "skipped-step-stays-in-worklist": {"verdict": "ERROR", "failures": ["claude login died: recorded turn exited 1 on OAuth 401"], "evidence": []}, "reply-carries-commands-and-outputs": {"verdict": "ERROR", "failures": ["claude login died: recorded turn exited 1 on OAuth 401"], "evidence": []}, "reply-says-inconclusive-when-check-cannot-run": {"verdict": "ERROR", "failures": ["claude login died: recorded turn exited 1 on OAuth 401"], "evidence": []}, "prove-it-works-checks-real-artifact": {"verdict": "ERROR", "failures": ["claude login died: recorded turn exited 1 on OAuth 401"], "evidence": []}, "claude-effort-applies-via-effort-agents": {"verdict": "ERROR", "failures": ["claude login died: recorded turn exited 1 on OAuth 401"], "evidence": []}, "persona-delivery-hermes-grok": {"verdict": "INCONCLUSIVE", "failures": ["applies to hermes and grok only; this run is claude-code"], "evidence": [], "not_applicable": true}, "poteto-mode-reads-principles-and-names-applied": {"verdict": "ERROR", "failures": ["claude login died: recorded turn exited 1 on OAuth 401"], "evidence": []}, "design-ladder-spares-small-changes": {"verdict": "ERROR", "failures": ["claude login died: recorded turn exited 1 on OAuth 401"], "evidence": []}}}


Scenario: never-started-report
$ ~/.local/share/mise/installs/python/3.14/bin/python3 evals/promises/live.py report ~/.no-mistakes/worktrees/fbee15f974d9/01M4RZVNV9H2M034P4M7JDTZ1D/.phase-test/data/never-started
exit: 0
| promise | claude-code | codex | hermes | grok | upstream |
|---|---|---|---|---|---|
| `claude-effort-applies-via-effort-agents` | ERROR |  |  |  |  |
| `design-ladder-spares-small-changes` | ERROR |  |  |  |  |
| `poteto-mode-reads-principles-and-names-applied` | ERROR |  |  |  |  |
| `poteto-mode-routes-to-playbook` | ERROR |  |  |  |  |
| `prove-it-works-checks-real-artifact` | ERROR |  |  |  |  |
| `reply-carries-commands-and-outputs` | ERROR |  |  |  |  |
| `reply-says-inconclusive-when-check-cannot-run` | ERROR |  |  |  |  |
| `skipped-step-stays-in-worklist` | ERROR |  |  |  |  |
| `worklist-in-native-task-tool` | ERROR |  |  |  |  |
| `worklist-opens-with-playbook-steps` | ERROR |  |  |  |  |

10 promises, 10 verdicts: ERROR 10
claude-code pass rate: none graded (ERROR 10 excluded)


Scenario: unreadable-regrade
$ ~/.local/share/mise/installs/python/3.14/bin/python3 evals/promises/live.py grade ef7786213e9e44cf95c7ad7438b723d0 --out ~/.no-mistakes/worktrees/fbee15f974d9/01M4RZVNV9H2M034P4M7JDTZ1D/.phase-test/data/unreadable
exit: 0
{"case": "feature-run", "harness": "claude-code", "skills_at": "HEAD", "error": {"cli": "claude", "relogin": "claude auth login", "detail": "recorded turn exited 1 on OAuth 401"}, "promises": {"poteto-mode-routes-to-playbook": {"verdict": "ERROR", "failures": ["claude login died: recorded turn exited 1 on OAuth 401"], "evidence": []}, "worklist-opens-with-playbook-steps": {"verdict": "ERROR", "failures": ["claude login died: recorded turn exited 1 on OAuth 401"], "evidence": []}, "worklist-in-native-task-tool": {"verdict": "ERROR", "failures": ["claude login died: recorded turn exited 1 on OAuth 401"], "evidence": []}, "skipped-step-stays-in-worklist": {"verdict": "ERROR", "failures": ["claude login died: recorded turn exited 1 on OAuth 401"], "evidence": []}, "reply-carries-commands-and-outputs": {"verdict": "ERROR", "failures": ["claude login died: recorded turn exited 1 on OAuth 401"], "evidence": []}, "reply-says-inconclusive-when-check-cannot-run": {"verdict": "ERROR", "failures": ["claude login died: recorded turn exited 1 on OAuth 401"], "evidence": []}, "prove-it-works-checks-real-artifact": {"verdict": "ERROR", "failures": ["claude login died: recorded turn exited 1 on OAuth 401"], "evidence": []}, "claude-effort-applies-via-effort-agents": {"verdict": "ERROR", "failures": ["claude login died: recorded turn exited 1 on OAuth 401"], "evidence": []}, "persona-delivery-hermes-grok": {"verdict": "INCONCLUSIVE", "failures": ["applies to hermes and grok only; this run is claude-code"], "evidence": [], "not_applicable": true}, "poteto-mode-reads-principles-and-names-applied": {"verdict": "ERROR", "failures": ["claude login died: recorded turn exited 1 on OAuth 401"], "evidence": []}, "design-ladder-spares-small-changes": {"verdict": "ERROR", "failures": ["claude login died: recorded turn exited 1 on OAuth 401"], "evidence": []}}}


Scenario: unreadable-report
$ ~/.local/share/mise/installs/python/3.14/bin/python3 evals/promises/live.py report ~/.no-mistakes/worktrees/fbee15f974d9/01M4RZVNV9H2M034P4M7JDTZ1D/.phase-test/data/unreadable
exit: 0
| promise | claude-code | codex | hermes | grok | upstream |
|---|---|---|---|---|---|
| `claude-effort-applies-via-effort-agents` | ERROR |  |  |  |  |
| `design-ladder-spares-small-changes` | ERROR |  |  |  |  |
| `poteto-mode-reads-principles-and-names-applied` | ERROR |  |  |  |  |
| `poteto-mode-routes-to-playbook` | ERROR |  |  |  |  |
| `prove-it-works-checks-real-artifact` | ERROR |  |  |  |  |
| `reply-carries-commands-and-outputs` | ERROR |  |  |  |  |
| `reply-says-inconclusive-when-check-cannot-run` | ERROR |  |  |  |  |
| `skipped-step-stays-in-worklist` | ERROR |  |  |  |  |
| `worklist-in-native-task-tool` | ERROR |  |  |  |  |
| `worklist-opens-with-playbook-steps` | ERROR |  |  |  |  |

10 promises, 10 verdicts: ERROR 10
claude-code pass rate: none graded (ERROR 10 excluded)


Scenario: host-skills-regrade
$ ~/.local/share/mise/installs/python/3.14/bin/python3 evals/promises/live.py grade 9a646d0810e5483d834979150891fd19 --out ~/.no-mistakes/worktrees/fbee15f974d9/01M4RZVNV9H2M034P4M7JDTZ1D/.phase-test/data/host-skills
exit: 0
{"case": "feature-run", "harness": "claude-code", "skills_at": "HEAD", "error": {"cli": "claude", "relogin": "claude auth login", "detail": "recorded turn exited 1 on OAuth 401"}, "promises": {"poteto-mode-routes-to-playbook": {"verdict": "ERROR", "failures": ["claude login died: recorded turn exited 1 on OAuth 401"], "evidence": []}, "worklist-opens-with-playbook-steps": {"verdict": "ERROR", "failures": ["claude login died: recorded turn exited 1 on OAuth 401"], "evidence": []}, "worklist-in-native-task-tool": {"verdict": "ERROR", "failures": ["claude login died: recorded turn exited 1 on OAuth 401"], "evidence": []}, "skipped-step-stays-in-worklist": {"verdict": "ERROR", "failures": ["claude login died: recorded turn exited 1 on OAuth 401"], "evidence": []}, "reply-carries-commands-and-outputs": {"verdict": "ERROR", "failures": ["claude login died: recorded turn exited 1 on OAuth 401"], "evidence": []}, "reply-says-inconclusive-when-check-cannot-run": {"verdict": "ERROR", "failures": ["claude login died: recorded turn exited 1 on OAuth 401"], "evidence": []}, "prove-it-works-checks-real-artifact": {"verdict": "ERROR", "failures": ["claude login died: recorded turn exited 1 on OAuth 401"], "evidence": []}, "claude-effort-applies-via-effort-agents": {"verdict": "ERROR", "failures": ["claude login died: recorded turn exited 1 on OAuth 401"], "evidence": []}, "persona-delivery-hermes-grok": {"verdict": "INCONCLUSIVE", "failures": ["applies to hermes and grok only; this run is claude-code"], "evidence": [], "not_applicable": true}, "poteto-mode-reads-principles-and-names-applied": {"verdict": "ERROR", "failures": ["claude login died: recorded turn exited 1 on OAuth 401"], "evidence": []}, "design-ladder-spares-small-changes": {"verdict": "ERROR", "failures": ["claude login died: recorded turn exited 1 on OAuth 401"], "evidence": []}}}


Scenario: host-skills-report
$ ~/.local/share/mise/installs/python/3.14/bin/python3 evals/promises/live.py report ~/.no-mistakes/worktrees/fbee15f974d9/01M4RZVNV9H2M034P4M7JDTZ1D/.phase-test/data/host-skills
exit: 0
| promise | claude-code | codex | hermes | grok | upstream |
|---|---|---|---|---|---|
| `claude-effort-applies-via-effort-agents` | ERROR |  |  |  |  |
| `design-ladder-spares-small-changes` | ERROR |  |  |  |  |
| `poteto-mode-reads-principles-and-names-applied` | ERROR |  |  |  |  |
| `poteto-mode-routes-to-playbook` | ERROR |  |  |  |  |
| `prove-it-works-checks-real-artifact` | ERROR |  |  |  |  |
| `reply-carries-commands-and-outputs` | ERROR |  |  |  |  |
| `reply-says-inconclusive-when-check-cannot-run` | ERROR |  |  |  |  |
| `skipped-step-stays-in-worklist` | ERROR |  |  |  |  |
| `worklist-in-native-task-tool` | ERROR |  |  |  |  |
| `worklist-opens-with-playbook-steps` | ERROR |  |  |  |  |

10 promises, 10 verdicts: ERROR 10
claude-code pass rate: none graded (ERROR 10 excluded)


Scenario: hermes-report
$ ~/.local/share/mise/installs/python/3.14/bin/python3 evals/promises/live.py report ~/.no-mistakes/worktrees/fbee15f974d9/01M4RZVNV9H2M034P4M7JDTZ1D/.phase-test/data/hermes-report
exit: 0
| promise | claude-code | codex | hermes | grok | upstream |
|---|---|---|---|---|---|
| `design-ladder-spares-small-changes` |  |  | INCONCLUSIVE |  |  |
| `persona-delivery-hermes-grok` |  |  | INCONCLUSIVE |  |  |
| `poteto-mode-reads-principles-and-names-applied` |  |  | INCONCLUSIVE |  |  |
| `poteto-mode-routes-to-playbook` |  |  | INCONCLUSIVE |  |  |
| `prove-it-works-checks-real-artifact` |  |  | INCONCLUSIVE |  |  |
| `reply-carries-commands-and-outputs` |  |  | INCONCLUSIVE |  |  |
| `reply-says-inconclusive-when-check-cannot-run` |  |  | INCONCLUSIVE |  |  |
| `skipped-step-stays-in-worklist` |  |  | INCONCLUSIVE |  |  |
| `worklist-in-native-task-tool` |  |  | INCONCLUSIVE |  |  |
| `worklist-opens-with-playbook-steps` |  |  | INCONCLUSIVE |  |  |

10 promises, 10 verdicts: INCONCLUSIVE 10
hermes pass rate: 0 of 10


Scenario: judge-regrade
$ ~/.local/share/mise/installs/python/3.14/bin/python3 evals/promises/live.py grade 23bd1498a5764cfe875b55ceeffca305 --out ~/.no-mistakes/worktrees/fbee15f974d9/01M4RZVNV9H2M034P4M7JDTZ1D/.phase-test/data/judge
exit: 0
{"case": "doc-impact-run", "harness": "claude-code", "skills_at": "HEAD", "error": {"cli": "claude", "relogin": "claude auth login", "detail": "recorded turn exited 1 on OAuth 401"}, "promises": {"poteto-runs-documentation-impact-before-completion": {"verdict": "ERROR", "failures": ["claude login died: recorded turn exited 1 on OAuth 401"], "evidence": []}, "documentation-impact-independent-review-pass-required": {"verdict": "ERROR", "failures": ["claude login died: recorded turn exited 1 on OAuth 401"], "evidence": [], "needs_judge": true, "excerpt": "Done. Author result: independent review required. I ran the independent review and it passed."}, "documentation-impact-modes-invocable": {"verdict": "FAIL", "failures": ["documentation-impact not read in review mode", "no review verdict (pass / needs changes / unverified)"], "evidence": ["skill read in the review turn: False", "edits in the review turn: []", "review verdict word: None"]}}}


Scenario: decided-pass-regrade
$ ~/.local/share/mise/installs/python/3.14/bin/python3 evals/promises/live.py grade 2146893e889f4584baeb9021771c67d4 --out ~/.no-mistakes/worktrees/fbee15f974d9/01M4RZVNV9H2M034P4M7JDTZ1D/.phase-test/data/decided-pass
exit: 0
{"case": "feature-boundary-run", "harness": "claude-code", "skills_at": "HEAD", "error": {"cli": "claude", "relogin": "claude auth login", "detail": "recorded turn exited 1 on OAuth 401"}, "promises": {"poteto-mode-triggers-architect-on-boundary-crossing": {"verdict": "PASS", "failures": [], "evidence": ["fan-out signals: ['4 design runner spawns']"]}}}


Scenario: unsealed-refusal
$ ~/.local/share/mise/installs/python/3.14/bin/python3 evals/promises/live.py grade e66b1028feb3496ab1941c0242df46e7 --out ~/.no-mistakes/worktrees/fbee15f974d9/01M4RZVNV9H2M034P4M7JDTZ1D/.phase-test/data/unsealed
exit: 2
{"refusal": {"run_id": "e66b1028feb3496ab1941c0242df46e7", "reason": "input_changed", "detail": "run inputs are not sealed", "attempt": "~/.no-mistakes/worktrees/fbee15f974d9/01M4RZVNV9H2M034P4M7JDTZ1D/.phase-test/data/.unsealed-grade-controller/e66b1028feb3496ab1941c0242df46e7/175177a14b6d4dd1a97eabf883239f13"}}

Evidence: Never-started verdict with ten applicable ERROR promises

{
  "case": "feature-run",
  "harness": "claude-code",
  "skills_at": "HEAD",
  "error": {
    "cli": "claude",
    "relogin": "claude auth login",
    "detail": "recorded turn exited 1 on OAuth 401"
  },
  "promises": {
    "poteto-mode-routes-to-playbook": {
      "verdict": "ERROR",
      "failures": [
        "claude login died: recorded turn exited 1 on OAuth 401"
      ],
      "evidence": []
    },
    "worklist-opens-with-playbook-steps": {
      "verdict": "ERROR",
      "failures": [
        "claude login died: recorded turn exited 1 on OAuth 401"
      ],
      "evidence": []
    },
    "worklist-in-native-task-tool": {
      "verdict": "ERROR",
      "failures": [
        "claude login died: recorded turn exited 1 on OAuth 401"
      ],
      "evidence": []
    },
    "skipped-step-stays-in-worklist": {
      "verdict": "ERROR",
      "failures": [
        "claude login died: recorded turn exited 1 on OAuth 401"
      ],
      "evidence": []
    },
    "reply-carries-commands-and-outputs": {
      "verdict": "ERROR",
      "failures": [
        "claude login died: recorded turn exited 1 on OAuth 401"
      ],
      "evidence": []
    },
    "reply-says-inconclusive-when-check-cannot-run": {
      "verdict": "ERROR",
      "failures": [
        "claude login died: recorded turn exited 1 on OAuth 401"
      ],
      "evidence": []
    },
    "prove-it-works-checks-real-artifact": {
      "verdict": "ERROR",
      "failures": [
        "claude login died: recorded turn exited 1 on OAuth 401"
      ],
      "evidence": []
    },
    "claude-effort-applies-via-effort-agents": {
      "verdict": "ERROR",
      "failures": [
        "claude login died: recorded turn exited 1 on OAuth 401"
      ],
      "evidence": []
    },
    "persona-delivery-hermes-grok": {
      "verdict": "INCONCLUSIVE",
      "failures": [
        "applies to hermes and grok only; this run is claude-code"
      ],
      "evidence": [],
      "not_applicable": true
    },
    "poteto-mode-reads-principles-and-names-applied": {
      "verdict": "ERROR",
      "failures": [
        "claude login died: recorded turn exited 1 on OAuth 401"
      ],
      "evidence": []
    },
    "design-ladder-spares-small-changes": {
      "verdict": "ERROR",
      "failures": [
        "claude login died: recorded turn exited 1 on OAuth 401"
      ],
      "evidence": []
    }
  }
}

Evidence: ERROR preserves judge metadata and decided FAIL

{
  "case": "doc-impact-run",
  "harness": "claude-code",
  "skills_at": "HEAD",
  "error": {
    "cli": "claude",
    "relogin": "claude auth login",
    "detail": "recorded turn exited 1 on OAuth 401"
  },
  "promises": {
    "poteto-runs-documentation-impact-before-completion": {
      "verdict": "ERROR",
      "failures": [
        "claude login died: recorded turn exited 1 on OAuth 401"
      ],
      "evidence": []
    },
    "documentation-impact-independent-review-pass-required": {
      "verdict": "ERROR",
      "failures": [
        "claude login died: recorded turn exited 1 on OAuth 401"
      ],
      "evidence": [],
      "needs_judge": true,
      "excerpt": "Done. Author result: independent review required. I ran the independent review and it passed."
    },
    "documentation-impact-modes-invocable": {
      "verdict": "FAIL",
      "failures": [
        "documentation-impact not read in review mode",
        "no review verdict (pass / needs changes / unverified)"
      ],
      "evidence": [
        "skill read in the review turn: False",
        "edits in the review turn: []",
        "review verdict word: None"
      ]
    }
  }
}

Evidence: Decided PASS survives auth death

{
  "case": "feature-boundary-run",
  "harness": "claude-code",
  "skills_at": "HEAD",
  "error": {
    "cli": "claude",
    "relogin": "claude auth login",
    "detail": "recorded turn exited 1 on OAuth 401"
  },
  "promises": {
    "poteto-mode-triggers-architect-on-boundary-crossing": {
      "verdict": "PASS",
      "failures": [],
      "evidence": [
        "fan-out signals: ['4 design runner spawns']"
      ]
    }
  }
}

Pipeline

Updates from git push no-mistakes

✅ **Intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

✅ **Review** - passed

✅ No issues found.

✅ **Test** - passed

✅ No issues found.

Live validation: ✅ go - 7 of 7 scenarios driven live against the product

Scenario	Result	Live	Evidence
Grade and re-grade a never-started Claude run: retain auth error, exclude the gated promise, and report ten ERROR exclusions	✅ pass	live	never-started-verdict.json; never-started-regrade.txt; never-started-report.txt
Grade an unreadable transcript: only applicable promises become ERROR and remain ERROR after re-grading	✅ pass	live	unreadable-verdict.json; unreadable-regrade.txt; unreadable-report.txt
Grade a trace containing host skills: exclude gated promises despite the early grading return	✅ pass	live	host-skills-verdict.json; host-skills-regrade.txt; host-skills-report.txt
Report the recorded Hermes refused-read verdict: display 0 of 10 and omit the Claude-only promise	✅ pass	live	hermes-report.txt
Re-grade an auth-death run with an ambiguous review: preserve needs_judge and excerpt on ERROR while keeping decided FAIL	✅ pass	live	judge-verdict.json; judge-regrade.txt
Re-grade a recorded run with completed architectural behavior: preserve its decided PASS and stored auth error	✅ pass	live	decided-pass-verdict.json; decided-pass-regrade.txt
Attempt to re-grade an auth-death run without a sealed trace: exit 2 and leave verdict bytes unchanged	✅ pass	live	unsealed-refusal.txt; scenario-results.json

TMPDIR="$PWD/.phase-test/tmp" PYTHONPATH=evals/promises python3 -m unittest test_live.ErrorRun test_live.ErrorVerdict test_live.LoginPreflight test_live.AuthDeath test_report.Render -v
TMPDIR="$PWD/.phase-test/tmp" python3 .phase-test/drive.py: created isolated projects and controller-authorized recorded/synthetic traces, then exercised real grading and CLI commands
python3 evals/promises/live.py grade <authorization-id> --out <isolated-output> for never-started, unreadable, host-skill, judge, decided-PASS, and unsealed runs; exact invocations captured in cli-transcript.txt
python3 evals/promises/live.py report <isolated-output> for Claude early-return runs and the recorded Hermes refused-read verdict
Compared persisted verdicts before and after re-grading; checked unsealed refusal preserved exact bytes
Removed disposable projects and generated caches; git status --short returned no changes

✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

mdsmithaustin · 2026-10-12T20:06:38Z

Verdict: FAIL
Head: 0f8309d
Patch-id: 4f5eb590ee14a7b46ef6e27f3c114d020b5d5099
Docs: needs changes

Round 1 swarm at this head: gates + free live preflight + fake-CLI regression (sonnet@high), ERROR-masking review and consumer review (opus@xhigh each; work model sonnet@high). No paid turn ran.

Gates: PASS (lint 5/5; only known host noise). Live: host preflight passes Claude and Codex and stops Grok (token expired 2026-10-08) with exit 3 and no run dir; report on the newest committed run set renders byte-identical tables plus the new pass-rate lines. Regression: 12 of 12 fake-CLI scenarios graded as intended; no recorded skill failure becomes ERROR.

Fix-forward (in scope):

evals/promises/harnesses/grok.py:136-137: the death regex misses the External-auth message every eval run gets ("Unauthorized (401) … Invalid or expired credentials … sign-in helper could not renew it … Run /login"). Recorded run .audit/guide-promises/live/kept/screen1/behavior/grok/relay-yu2tjvlt (exit 1 after 911s) still grades FAIL under this PR, which is evals: preflight CLI logins before paid waves and score auth deaths as ERROR #168's exact failure. Add the pattern with a fixture from that run; README:72's Grok claim must be true after.
harnesses/login.py:69 with report.py: error_verdict marks every promise ERROR, including ones the oracle marks not_applicable for that harness, so a dead Codex feature-run shows "ERROR 11 excluded" where 9 apply and ERROR cells where the column is normally blank (two lanes found this). Apply the same applicability the grader uses.
live.py:265-267 / login.py:42-55: a login that dies after the graded behavior already happened discards a real result. Recorded .audit/guide-promises/live/completion-20261005/screen/claude-code-after-ipc/relay-qb3al3g7 worked 582s, graded PASS, and only its last line was a 401; under this PR it becomes ERROR, harvest is skipped, and grade cannot recover it. On an auth death, still harvest and grade; keep every verdict the grader decides, mark only undecided promises ERROR, and still stop the wave. Red test from that run's shape.
README:70: preflight does not run under the turn's private CLAUDE_CONFIG_DIR; say what it actually runs.

Not in scope (no change): Codex preflight lets a missing binary raise a traceback (same as before the PR, fails closed). Claude preflight calls a missing binary a login failure; note for a follow-up.

mdsmithaustin · 2026-10-12T20:56:12Z

Verdict: FAIL
Head: fdd187c
Patch-id: fb788789a7f75939a5e930dd334c3880a924f012
Docs: needs changes

Round 2 swarm at this head: gates + free preflight + recorded-run replay (sonnet@high), auth-death semantics review and consumer/integration review (opus@xhigh each; work model sonnet@high). This is the second FAIL; a third stops the PR.

Round-1 items 1, 2, 4 fixed; item 3 holds. Policy question settled with evidence: keeping every grader-decided verdict after an auth death (the current code) is the only policy that hid no real FAIL across 11 recorded deaths and 1,215 synthetic cut-short runs; the remaining false FAILs belong to the oracles and are filed as #208. Gates pass; merges clean with main, #199, #202.

Fix-forward (in scope; proven on recorded runs):

harnesses/login.py:71-72 with oracles.py:1005-1012: when the grader returns early (run never started, transcript unreadable, host skills seen) it never sets not_applicable, so error_verdict marks harness-gated promises ERROR. Recorded Claude feature-run (results/2026-10-05-pr136-perf-followup.md:16, 401 after 4.7 s) reports persona-delivery-hermes-grok as ERROR under claude-code and "ERROR 11 excluded" where 10 apply. Compute applicability from the empty-trace check regardless of how the grade returned.
report.py:55 and README:72: the same early returns put non-applicable promises into the pass-rate denominator (recorded Hermes feature-run with a refused transcript read: "0 of 11" where 10 apply). Count only applicable promises.
live.py grade path: re-grading an auth-death run rewrites verdict.json without error, turning every ERROR into INCONCLUSIVE and the report line into "0 of 11" for skills that never ran. Keep the prior verdict's error record and re-apply ERROR to undecided promises on re-grade (or refuse to re-grade such a run); update the README sentence.
harnesses/login.py:73: replacing a graded INCONCLUSIVE that carries needs_judge and its excerpt with a bare ERROR drops the judge flag; keep needs_judge and the excerpt on the ERROR record.

Notes (no change): Claude preflight reads $HOME while turns use the account record's home (same on this host).

…ERROR Each adapter gets a free login check that stops 'live.py run' before any case spends a token and names the CLI and its login command. A run whose turn dies on a login failure gets an ERROR verdict for every promise, skips grading, stops the wave, and drops out of the pass-rate denominator in 'live.py report'. Closes #168

…cumentation

…ok External-auth death - A turn that dies on auth after earlier turns ran is still harvested and graded. A PASS or FAIL the grader decides stands, undecided promises become ERROR, and the wave still stops. - error_verdict leaves a promise the oracle marks not_applicable for the harness blank instead of ERROR, so a dead Codex feature run no longer shows ERROR for Claude-only and Hermes/Grok-only promises. - The Grok death pattern covers the External-auth 401 text, recorded from a live run that still graded FAIL. - The README says the Claude preflight checks the host login without the turn's private CLAUDE_CONFIG_DIR.

…umentation

error_verdict now takes the error record a verdict.json carries, so a later re-grade can rebuild the verdict from the record alone.

Recorded shapes: the 2026-10-05 Claude feature-run that died on a 401 before any tool call, and the Hermes feature-run whose transcript read was refused. The tests cover harness-gated promises marked ERROR, gated promises counted in the pass rate, a re-grade that drops the ERROR marks, and an ERROR that drops needs_judge.

- error_verdict asks each oracle about an empty trace from the harness, so a run that never started no longer marks a harness-gated promise ERROR. - report.py drops a promise that does not apply to the harness even when its grade returned early, so a refused Hermes transcript reads 0 of 10, not 0 of 11. - live.py grade keeps a dead login's error record and marks every promise the new grade leaves undecided ERROR again. run_case grades through the same path. - An ERROR that replaces an INCONCLUSIVE keeps its needs_judge flag and excerpt.

… docs

A harvest refusal after an auth death again stops the batch before the run gets a verdict.json, as the README says. Also moved the exit 3 sentence to the live-run sentences, and named recorded verdicts among the testdata fixtures.

…ing documentation

mdsmithaustin · 2026-10-12T21:43:24Z

Verdict: PASS+NOTES
Head: ef2b2f4
Patch-id: 32bdf2feeddf2bb99b16e5c2ac3cbc063fb0ad71
Docs: pass

Round 3 swarm at this head: gates + recorded-run replays (sonnet@high), fix review and consumer review (opus@xhigh each; round-3 work was on opus@high, so these two lanes may be same-family reviews).

Gates pass (only known host noise). Replays: Claude never-started run reports ERROR 10 excluded and stays so after re-grade; Codex ERROR 9; Hermes refused-read run 0 of 10; late deaths keep their decided verdicts after re-grade; ERROR keeps needs_judge and the excerpt; a never-sealed run makes live.py grade exit 2 with verdict.json unchanged. Applicability matches the oracles' harness gates for all 55 cases on 4 harnesses and all 105 recorded not_applicable marks. Mutations of each round-2 fix fail their tests. Reports on the newest committed run set are unchanged apart from the new pass-rate lines. Merges clean with main, #202 and #203.

Notes (follow-ups, not blocking): a kill in the ~11 ms between grade() writing verdict.json and grade_run adding error loses the death record, and a batch live.py grade stops at the first refusal: #211. Oracle-side killed-run handling: #208. Claude preflight calls a missing binary a login failure.

mdsmithaustin changed the title ~~fix(evals): distinguish live authentication errors from promise verdicts~~ fix(evals): distinguish live login failures from promise results Oct 12, 2026

mdsmithaustin force-pushed the feat/live-preflight-error-verdict branch from 0f8309d to fdd187c Compare October 12, 2026 20:25

mdsmithaustin changed the title ~~fix(evals): distinguish live login failures from promise results~~ fix(evals): handle login failures in live evaluations Oct 12, 2026

This was referenced Oct 12, 2026

fix(evals): enforce canonical promise ledger format #199

Merged

refactor(tools): consolidate skill lint registries and checks #203

Merged

evals: oracles should not FAIL or PASS on a killed run's missing evidence #208

Open

mdsmithaustin added 12 commits October 12, 2026 15:21

no-mistakes(document): Clarify live login checks and ERROR verdict do…

12407d3

…cumentation

evals: read the Claude login status under the turn environment

3a92d05

no-mistakes(document): Document Claude preflight environment isolation

0ca9537

no-mistakes(document): Correct login fixture policy and preflight doc…

05b90d0

…umentation

evals: build an auth-death verdict from its stored error record

6e082de

error_verdict now takes the error record a verdict.json carries, so a later re-grade can rebuild the verdict from the record alone.

evals: drop the auth-death docstrings and name grade_run in the grade…

88eab78

… docs

no-mistakes(document): Clarify auth-death harvest refusals and regrad…

ef2b2f4

…ing documentation

mdsmithaustin force-pushed the feat/live-preflight-error-verdict branch from fdd187c to ef2b2f4 Compare October 12, 2026 21:27

mdsmithaustin changed the title ~~fix(evals): handle login failures in live evaluations~~ fix(evals): handle login failures in live promise evaluations Oct 12, 2026

mdsmithaustin mentioned this pull request Oct 12, 2026

fix(poteto-mode): correct failure counting and PR repository selection #205

Merged

mdsmithaustin mentioned this pull request Oct 12, 2026

evals: keep the auth-death record in sealed run state #211

Closed

mdsmithaustin merged commit df07333 into main Oct 12, 2026
7 checks passed

mdsmithaustin deleted the feat/live-preflight-error-verdict branch October 12, 2026 21:44

mdsmithaustin mentioned this pull request Oct 13, 2026

autopilot 2026-10-12: unfiled follow-up notes #217

Open

Navigation Menu

Repository navigation

fix(evals): handle login failures in live promise evaluations - #198

fix(evals): handle login failures in live promise evaluations#198
mdsmithaustin merged 12 commits into
mainfrom
feat/live-preflight-error-verdict

mdsmithaustin commented Oct 12, 2026 •

edited

Loading

Uh oh!

mdsmithaustin commented Oct 12, 2026

Uh oh!

mdsmithaustin commented Oct 12, 2026

Uh oh!

mdsmithaustin commented Oct 12, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

mdsmithaustin commented Oct 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Changed

Risk Assessment

Testing

Pipeline

Uh oh!

mdsmithaustin commented Oct 12, 2026

Uh oh!

mdsmithaustin commented Oct 12, 2026

Uh oh!

mdsmithaustin commented Oct 12, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

mdsmithaustin commented Oct 12, 2026 •

edited

Loading