Skip to content

fix(codex): surface the backend reason for failed turns - #571

Closed
officialasishkumar wants to merge 1 commit into
steipete:mainfrom
officialasishkumar:fix/codex-failure-reason
Closed

officialasishkumar wants to merge 1 commit into
steipete:mainfrom
officialasishkumar:fix/codex-failure-reason

Conversation

@officialasishkumar

Copy link
Copy Markdown
Contributor

Summary

When Codex rejects a turn (for example The 'gpt-5.5' model is not supported when using Codex with a ChatGPT account.), it writes the reason to its JSONL stdout as a turn.failed event and exits non-zero. The Codex runner only parsed stdout after a successful exit. Users got the generic Command failed: codex exec … argv and incidental stderr such as Reading prompt from stdin..., but never the reason.

execCliWithInput already attaches stdout to exec errors; the Grok runner uses the same mechanism. The Codex runner now reads the turn.failed reason from that stdout and throws Codex CLI failed: <reason>, with the original exec error kept as cause. If a failed turn ever comes back with a zero exit, the same reason replaces the generic CLI returned empty output.

Only turn.failed is used. Codex's exec event processor fills it with the turn error, or else the last critical stream error. Top-level error events can be transient retry notices, and item-level error items are non-fatal warnings (such as the unrecognized-config notice from #564), so neither is reported. Timeouts, interrupts and other failures without a turn failure keep their current messages. A stream with only lifecycle events still fails as before.

Fixes #565

Validation

  • Red/green: two runner regressions in tests/llm.cli.regressions.test.ts cover a turn failure on a non-zero exit (the stream also contains an item-level warning and a retry error notice) and on a zero exit. Both fail on main and pass with this change. A third test checks that the exec error is kept when no turn.failed is present. Unit coverage for extractCodexTurnFailureFromJsonl is in tests/llm.cli-provider-output.coverage.test.ts.
  • pnpm check: formatting, localization, lint, typecheck and coverage all pass. 605 test files passed, 30 skipped; 3,898 tests passed, 49 skipped.
  • Built CLI with a stand-in CODEX_PATH script that emits the JSONL from the issue and exits 1, running summarize doc.txt --cli codex --force-summary --retries 0:
    • main: Command failed: …/codex exec --ephemeral … -m gpt-5.5 -c text.verbosity="medium" followed by Reading prompt from stdin...
    • this branch: Codex CLI failed: The 'gpt-5.5' model is not supported when using Codex with a ChatGPT account. (exit 1)

No changes to model selection, arguments or retry behavior.

When Codex rejects a turn (for example an unsupported model for a
ChatGPT login) it writes the reason to its JSONL stdout as a
`turn.failed` event and exits non-zero. The runner only parsed stdout
after a successful exit, so users saw the generic `Command failed:
codex exec ...` argv plus incidental stderr such as `Reading prompt from
stdin...`, with no hint about what actually went wrong.

The Codex runner now reads the `turn.failed` reason from the stdout that
execCliWithInput already attaches to exec errors and reports it as
`Codex CLI failed: <reason>`, keeping the original exec error as the
cause. The same reason replaces the generic "CLI returned empty output"
if a failed turn is ever reported with a zero exit.

Only `turn.failed` is used: Codex fills it with the turn error or the
last critical stream error, while top-level `error` events can be
transient retry notices and item-level errors are non-fatal warnings.
Failures without a turn failure (timeouts, interrupts, missing binary)
keep their existing messages.

Fixes steipete#565
@clawsweeper

clawsweeper Bot commented Nov 11, 2026 •

Copy link
Copy Markdown

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

ClawSweeper review complete

ClawSweeper finished reviewing this revision. The review result is being finalized.

View the workflow run.

@chatgpt-codex-connector

Copy link
Copy Markdown
Codex Review Summary: ❌ Failed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ❌ Failed 2026-11-11T12:03:34.378164Z
We couldn’t complete this review. Comment @codex review to try again.
20989b7 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

@clawsweeper clawsweeper Bot added clawsweeper:intake:bug Preliminary submission classification; full source review pending. clawsweeper:intake:needs-environment Preliminary submission classification; full source review pending. P2 Normal priority bug or improvement with limited blast radius. proof: sufficient Contributor real behavior proof is sufficient. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. and removed clawsweeper:intake:bug Preliminary submission classification; full source review pending. clawsweeper:intake:needs-environment Preliminary submission classification; full source review pending. labels Nov 11, 2026
@clawsweeper

clawsweeper Bot commented Nov 11, 2026

Copy link
Copy Markdown

Codex review: needs maintainer review before merge.

What this changes

This PR exposes terminal Codex turn-failure reasons in Summarize’s error messages.

Example: Codex rejects gpt-5.5 for a ChatGPT account and exits 1.

  • Before: Summarize displays Command failed and incidental stderr without the rejection reason.
  • After: Summarize displays Codex CLI failed: The 'gpt-5.5' model is not supported when using Codex with a ChatGPT account.

Review scores

Measure Result What it means
Overall readiness 🐚 platinum hermit (4/6) A focused, correct repair with relevant regressions and supplied CLI proof through the changed subprocess boundary.
Proof confidence 🐚 platinum hermit (4/6) Sufficient (live_output): The supplied built-CLI output exercises runCodexCli through Summarize with a real subprocess emitting the reported failure JSONL, showing the backend reason and exit 1 after the fix; the subprocess is a stand-in rather than an authenticated Codex session.
Patch quality 🦞 diamond lobster (5/6) No actionable review findings were identified.

Product

Kind: Bug fix · Worth it: Yes · Fix scope: Complete
User problem: A failed Codex summary hides the backend’s actionable rejection reason.
Reason: The linked report establishes a released diagnostic defect, and every production hunk serves its repair without changing configuration or model selection.

Merge readiness

✅ Ready for maintainer review

Keep open: this PR narrowly fixes a confirmed diagnostic bug still present in main, with no introduced correctness or security findings.

Priority: P2
Reviewed head: 20989b76dea1b9442213b6bdc08eedfea5b5d27a

Before merge

None.

Findings

None.

Agent review details

How this fits together

The CLI dispatches summary requests to the Codex runner, which executes Codex through the shared subprocess executor and converts its output into summary text or a user-facing error.

flowchart LR
A[Summarize CLI] --> B[CLI provider dispatcher]
B --> C[Codex runner]
C --> D[Shared subprocess executor]
D --> E[Codex process]
E --> F[JSONL stdout and exit status]
F --> C
C --> G[Summary or terminal error]
Loading

Technical review

Best possible solution:

Land the focused terminal-reason extraction using the existing executor stdout attachment and Codex parser owner.

Do we have a high-confidence way to reproduce the issue?

Current-main source confirms that rejected execution bypasses Codex JSONL parsing, and the linked report supplies a release reproduction; this review ran no target code.

Is this the best way to solve the issue?

The patch repairs the Codex-specific diagnostic at its existing owner, preserves the original cause, and avoids forwarding lifecycle events or nonfatal warnings.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning medium; reviewed against aaccb7c9c30e.

Provenance checked

  • Codex runner execution and output handling keeps the original intent (a50fb6e: The adapter split moved existing Codex behavior into a dedicated runner; it did not establish a requirement to hide terminal backend errors.)
  • Lifecycle-only JSONL rejection keeps the original intent (b971b39: Prevent lifecycle and usage events from being returned as successful summary content, as explained in the maintainer’s earlier issue closure.)

Testing

Proof path: shipped entry point.

Security

None.

Evidence

What I checked:

Likely related people:

  • Peter Steinberger: Raw commit a50fb6e adds src/llm/cli-runners/codex.ts:65 relative to its recorded parents. This identifies author metadata, not feature responsibility or a PR merger. (role: source-line author; confidence: high; commits: a50fb6e89b23; files: src/llm/cli-runners/codex.ts)
  • Vincent Peng: Raw commit e40aad4 adds src/llm/cli-runners/grok.ts:170 relative to its recorded parents. This identifies author metadata, not feature responsibility or a PR merger. (role: source-line author; confidence: medium; commits: e40aad452845; files: src/llm/cli-runners/grok.ts)

Preflight source

Source unknown. Source comparison at the observed PR snapshot only. Completion is self-reported; body, policy, tests, execution, and report bytes are not verified.

Labels

Label changes:

  • add P2: Failed Codex summaries lose actionable diagnostics, while process failure remains correctly signaled.
  • add proof: sufficient: Contributor real behavior proof is sufficient.
  • add rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🐚 platinum hermit and patch quality is 🦞 diamond lobster.
  • add status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR.

Label justifications:

  • P2: Failed Codex summaries lose actionable diagnostics, while process failure remains correctly signaled.
  • rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🐚 platinum hermit and patch quality is 🦞 diamond lobster.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR.
  • proof: sufficient: Contributor real behavior proof is sufficient.

Rating scale

6/6 🦀 challenger crab · 5/6 🦞 diamond lobster · 4/6 🐚 platinum hermit · 3/6 🦐 gold shrimp · 2/6 🦪 silver shellfish · 1/6 🧂 unranked krab. Overall follows the weaker of proof and patch quality; ✨ marks media proof (a screenshot, video, or linked artifact) that directly shows the changed behavior.

Workflow

ClawSweeper edits this one comment on every review. Comment @clawsweeper re-review for a fresh review only. Repair and merge execution lanes are retired.

Reviewed November 11, 2026, 7:17 AM ET / 12:17 UTC.

steipete added a commit that referenced this pull request Nov 14, 2026
Local audio/video extraction ignores `--json`, Markdown and transcription diagnostics misidentify their providers, and the Codex adapter pins an unavailable default while hiding backend failure reasons. Optional OpenRouter diagnostics can also wait beyond the request deadline, and Windows uninstall deletes launcher files even when Task Scheduler refuses deletion.

This consolidates and finishes #559, #560, #570, and #571, retaining their authors' fixes and adding truthful extraction provenance, supported Codex default selection, and regression coverage. It also corrects the YouTube fallback order and incorporates the small gateway documentation example from #568 through the existing OpenAI-compatible transport. Existing string-returning library converters remain compatible; explicitly requested Codex models remain explicit.

Validation:
- The new regression set fails on unmodified main across six test files, including denied task deletion, stalled HTTP response bodies, local media JSON, provenance, and Codex failure/default handling.
- Full `pnpm run build` and `pnpm run check` passed on Linux/Node 24: 605 test files passed, 3,909 tests passed, 49 skipped; coverage 94.52% lines and 85.67% branches.
- Independent Codex P0–P2 autoreview: scoped clean after fixing one stale test assertion.
- No version bump, release, deployment, or Chrome Web Store operation.

Addresses #561
Addresses #562
Addresses #563
Addresses #564
Addresses #565
Addresses #566
Addresses #567

Co-authored-by: Sebastien Tardif <[email protected]>
Co-authored-by: Vincent Peng <[email protected]>
Co-authored-by: Asish Kumar <[email protected]>
Co-authored-by: Nirupam Bhowmick <[email protected]>
@steipete

Copy link
Copy Markdown
Owner

Thanks for the focused contribution. Incorporated into #572 with co-author and changelog credit. The failed-turn regressions fail on main and now pass. Backend failure reasons retain their cause; the same patch also fixes the stale Codex default and unsupported injected verbosity option. Full build/check passed (3,909 tests), and #572 passed exact-head CI before landing. Closing this superseded branch.

@steipete steipete closed this Nov 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

P2 Normal priority bug or improvement with limited blast radius. proof: sufficient Contributor real behavior proof is sufficient. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(codex): failed turns omit the backend error reason

2 participants