Conversation
`gpt-5.4-mini` has been withdrawn. Every codex case in the prompt pilot
errored with
The 'gpt-5.4-mini' model is not supported when using Codex with a
ChatGPT account.
-- eight of sixteen cases red from a clean clone, for a reason that has
nothing to do with the skill under test.
Default to `gpt-5.6-sol`, the cheapest slug `codex` currently reports,
and say in the comment that the slug is dated rather than permanent, so
the next withdrawal is re-picked instead of re-diagnosed.
`CF_UX_CODEX_MODEL` is unchanged and still overrides.
Measured after the change: 8/8 codex cases pass.
Note for a separate change: skills/studio/scripts/studio/commands/agents.py
maps the cf model tiers onto gpt-5.4 and gpt-5.4-mini as well. That is the
product's model matrix, with a spec and snapshot tests behind it, and is
deliberately left alone here.
Signed-off-by: vasylcf <[email protected]>
|
Warning Review limit reachedNext included review available in 24 minutes. View limit detailsLimit details: You’ve used the included review currently available. This review ran on the open-source allowance, not this organization's plan, because the pull request author doesn't have an assigned seat. Waiting won't change this — ask an organization admin to assign them a seat, or add seats in Billing if every seat is already assigned, then retry. Review configuration: ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Advanced Run ID: 📒 Files selected for processing (4)
Comment |
Production agent model matrix still bakes in the withdrawn gpt-5.4-mini slug for Codex/Cursor+OpenAISeverity: Major Problem Reproduction, impact, suggested fix, verificationHow to reproduce
Expected behavior Actual behavior Impact Suggested correction How to verify |
Makefile still documents the withdrawn gpt-5.4-mini as the pilot's cheap defaultSeverity: Minor Problem Reproduction, impact, suggested fix, verificationHow to reproduce
Expected behavior Actual behavior Impact Suggested correction How to verify Original location: Makefile:332 -- inline anchoring could not be resolved after 1 attempt(s). |
| # SIGTERM/SIGINT, and via pid-liveness sweep on next invocation. | ||
| # | ||
| # Cheap-by-default models (Claude Haiku 4.5, gpt-5.4-mini) with low reasoning | ||
| # Cheap-by-default models (Claude Haiku 4.5, gpt-5.6-sol) with low reasoning |
There was a problem hiding this comment.
promptfooconfig.yaml's model-slug comment remains an unreferenced duplicate, unlike the Makefile's now-fixed equivalent
Severity: Minor
Problem
promptfooconfig.yaml line 7 hardcodes 'gpt-5.6-sol' as free-text prose, independent of codex_provider.py's DEFAULT_MODEL constant. This PR demonstrates awareness of exactly this failure mode by fixing the Makefile's identical duplication (removing the literal, explaining 'this comment is the copy that went stale last time'), but did not apply the same treatment to promptfooconfig.yaml, nor does the DEFAULT_MODEL runbook comment or CONTRIBUTING.md mention this yaml file as a location to update on re-pick.
Reproduction, impact, suggested fix, verification
How to reproduce
- Model gpt-5.6-sol is withdrawn in the future. 2. A maintainer follows codex_provider.py's runbook and updates DEFAULT_MODEL / CF_UX_CODEX_MODEL and CONTRIBUTING.md's table. 3. promptfooconfig.yaml:7 is not mentioned anywhere in the runbook, so it is not updated and now names a stale/withdrawn model.
Expected behavior
The yaml comment should either reference the source of truth (e.g. 'see codex_provider.py DEFAULT_MODEL') instead of duplicating the literal, or the update runbook should explicitly list this file as a location to update.
Actual behavior
The yaml comment independently duplicates the literal slug with no cross-reference, and no runbook mentions it.
codex_provider.py DEFAULT_MODEL updated on re-pick --> CONTRIBUTING.md table updated --> promptfooconfig.yaml:7 comment NOT updated (not in runbook) --> stale doc
Impact
Low-severity documentation drift; a contributor reading promptfooconfig.yaml's comment could be misled about the actual current default after a future re-pick, though the functional default lives correctly in codex_provider.py.
Suggested correction
Replace the literal slug in promptfooconfig.yaml:7 with a reference to codex_provider.py's DEFAULT_MODEL, or add this file to the re-pick runbook comment in codex_provider.py.
How to verify
After a future model re-pick, confirm promptfooconfig.yaml's comment either has no literal to go stale or was updated alongside the other two locations per an explicit runbook step.
There was a problem hiding this comment.
Re-verified against the current code -- this write-up has been updated.
Why
Problem (was): promptfooconfig.yaml line 7 ('Cheap-by-default models (Claude Haiku 4.5, gpt-5.6-sol) with low reasoning...') restates the model slug as free-text prose, independent of codex_provider.py's DEFAULT_MODEL constant. The new remediation guidance (codex_provider.py's comment and CONTRIBUTING.md's updated row) tells a future maintainer to re-pick the model and update CF_UX_CODEX_MODEL/CONTRIBUTING.md, but never mentions this yaml comment.
Problem (now): promptfooconfig.yaml line 7 hardcodes 'gpt-5.6-sol' as free-text prose, independent of codex_provider.py's DEFAULT_MODEL constant. This PR demonstrates awareness of exactly this failure mode by fixing the Makefile's identical duplication (removing the literal, explaining 'this comment is the copy that went stale last time'), but did not apply the same treatment to promptfooconfig.yaml, nor does the DEFAULT_MODEL runbook comment or CONTRIBUTING.md mention this yaml file as a location to update on re-pick.
…stale Review found Makefile:332 still naming gpt-5.4-mini as the pilot default, after CONTRIBUTING.md and promptfooconfig.yaml had been updated -- the third copy, missed exactly the way a fourth would be. Rather than sync it, remove the slug from it: the comment's subject is that the pilot spends real tokens on cheap models, not which ones. The values stay next to the code that uses them. Also note in codex_provider.py why its default is not shared with agents.py's _MODEL_MATRIX, since the two naming the same withdrawn model invites the conclusion that they should be one constant. Signed-off-by: vasylcf <[email protected]>
|
|
Both findings verified; one taken, one answered. Taken —
|
| #: this one is only the cheapest of that set as of 2026-09-18 (`codex` reports | ||
| #: gpt-5.6-sol, -terra, -luna, gpt-6-astra, gpt-5.5). Its predecessor here, | ||
| #: `gpt-5.4-mini`, had been withdrawn, and every codex case in the pilot errored | ||
| #: with `The 'gpt-5.4-mini' model is not supported when using Codex with a |
There was a problem hiding this comment.
No fail-fast validation of model availability
Non-blocking review challenge -- [engineering]
Rather than letting every codex-backed test case in the suite independently error out with the same 'model not supported' message when the pinned slug is withdrawn, why not have the provider validate DEFAULT_MODEL against the account's entitled set once at startup and raise a single, actionable error?
Why this is worth asking
The comment itself describes exactly this failure mode ('half the suite red for a reason that had nothing to do with the skill under test'), and the fix is purely reactive (update the slug, add a longer comment) rather than making the same class of failure cheap to diagnose next time it recurs.
Industry note
CLI/SDK wrappers commonly do an early capability/availability check (or catch the specific 'unsupported model' error and re-raise with a short, targeted message) so N parallel workers don't each surface the same root cause as N separate opaque failures.
|
↩️ Replying to #239 (comment) This concern still stands. WhyThe "pushback" is just an automated SonarCloud quality-gate bot comment (Quality Gate passed, 0 new issues) — it contains no argument or claim addressing the finding at all, vague or otherwise. It's boilerplate CI output, not a rebuttal. Separately, |
|
↩️ Replying to #239 (comment) Re-verified against the current code (fe5fbce) and this is still an issue. The write-up has also been updated to reflect the current code (see the edited comment above). Why
|
|
↩️ Replying to #239 (comment) This finding is already closed (fixed-by-new-commit) -- an automated re-check won't reopen it on its own, so flagging this for a human to look at directly rather than reversing it here. WhyThe posted "pushback" is not actually a rebuttal — it's an automated SonarCloud quality-gate comment with no relevance to the model-sync finding, making no checkable claim to evaluate. Separately, even on the merits, the diff and current file content show the finding is stale: Makefile lines 331-338 no longer say 'gpt-5.4-mini' at all — the comment was rewritten to deliberately stop repeating slugs and explains why ("The slugs themselves are deliberately not repeated here... this comment is the copy that went stale last time"). But since the pushback itself offers nothing substantive, there's nothing to agree with here. |
ainetx
left a comment
ainetx
left a comment
There was a problem hiding this comment.
Requesting changes: the pilot-side fix is fine, but it doesn't close out the actual production risk, and the PR's own new comment names that risk explicitly without addressing it.
- Production agent model matrix still bakes in the withdrawn gpt-5.4-mini slug —
codex_provider.pyis updated to a working default, butskills/studio/scripts/studio/commands/agents.py's_MODEL_MATRIX(the resolver that generates real end-user agent configs) still hard-codesgpt-5.4-minifor both('codex','openai')and('cursor','openai'), plus relatedgpt-5.4overrides — meaning any agent generated through this path picks a model that ChatGPT-account Codex sessions will reject at runtime. This needs the same fix applied here before merge. (comment)
2 participants
Problem
gpt-5.4-minihas been withdrawn. Every codex case in the prompt pilot errors out:From a clean clone that is 8 of 16 cases red, for a reason that has nothing to do with the skill under test — and the failure surfaces as a truncated JSON blob inside a promptfoo table cell, so it reads like a harness bug rather than a stale constant.
Change
Default
CF_UX_CODEX_MODELtogpt-5.6-sol, the cheapest slugcodexcurrently reports for this account type. The full entitled set today:gpt-5.6-sol,gpt-5.6-terra,gpt-5.6-luna,gpt-6-astra,gpt-5.5.The comment now states the slug is dated, not permanent, and says where to re-pick from, so the next withdrawal costs a lookup rather than a debugging session.
CF_UX_CODEX_MODELis untouched and still overrides.DEFAULT_EFFORT="low"stays valid —gpt-5.6-sollistslowamong its supported reasoning levels.Two docs naming the same dead default are corrected alongside:
CONTRIBUTING.mdand the header comment inpromptfooconfig.yaml.Result
Measured on this branch —
make test-promptswent from 0 passed / 16 errors to 11 passed / 0 failed / 5 errors, 1m30s, $0.72.All 8 codex cases now pass. The remaining 5 are claude-side and are genuine pilot findings, not environment: 4 × the
cfskill was never invoked, 1 ×cf-documenting-reviewwas invoked directly, bypassing thecfrouter. Those are being handled separately.Deliberately out of scope
skills/studio/scripts/studio/commands/agents.pymaps the cf model tiers ontogpt-5.4andgpt-5.4-minitoo, so the product's own model matrix names withdrawn models. That has a spec and snapshot tests behind it and deserves its own change — flagged here, not touched.