Conversation
gpt-5.4 and gpt-5.4-mini have been withdrawn. Measured against a
ChatGPT-plan Codex account:
gpt-5.4 400 The 'gpt-5.4' model is not supported when using
Codex with a ChatGPT account.
gpt-5.4-mini 400 (same)
gpt-5.5 OK
Those two are what the (codex, openai) and (cursor, openai) cells
resolved the cheap and balanced tiers to, and skills/studio/agents.toml
puts 43 of its 44 agents on those two tiers. Nothing validates a slug at
generate time, so the config wrote cleanly and every agent answered a
400 at the point of use.
Tiers are mapped from OpenAI's own descriptions of each model rather
than picked by taste, so the next re-pick is a reading:
cheap gpt-5.6-luna fast and affordable agentic coding model
balanced gpt-5.6-terra balanced agentic coding model for everyday work
expensive gpt-6-astra most capable model for complex, demanding work
The cheap overrides keep their meaning -- cheap on a codebase task bumps
one tier -- and so become gpt-5.6-terra.
Tests: the 18 assertions that resolve through the matrix are updated, and
the ones whose *names* carried a slug are renamed after the tier they
pin, so the next rotation is a value change and not a rename. Five
fixtures passing a model string through a renderer keep theirs; the value
is arbitrary there and they pass either way.
Deliberately untouched, and said so in the code:
- (copilot, openai) names models by display name from GitHub Copilot's
own catalogue (GPT-4.1 / GPT-5.4 / GPT-5.5). What Copilot offers is
not what a Codex account is entitled to, and nothing here has been
measured against it.
- Every Anthropic cell. claude-sonnet-4-6, claude-opus-4-7 and
claude-haiku-4-5 were each probed live and answer.
The spec pins the existence of each matrix cell, not the slugs in it, so
no CDSL step changes.
Signed-off-by: vasylcf <[email protected]>
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Advanced Run ID: 📒 Files selected for processing (5)
Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review. 📝 WalkthroughWalkthroughThe OpenAI model mappings for Codex and Cursor now use refreshed tier slugs. Tests update expected resolution, schema translation, manifest, template, fallback, and registration outputs. ChangesOpenAI model slug refresh
Priority: ➖ Normal Estimated code review effort: 2 (Simple) | ~10 minutes Change: Bug fix · Severity of issue fixed: Medium Suggested reviewers: Merge Risk: 🔵 Low · up to The generated configurations use new model IDs whose support has not been confirmed for both target backends. Confirming availability before rollout avoids affected agents failing at use time. 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
🧪 Generate unit tests (beta)
Comment |
|
code-ranker View diff report ↗python
baseline main @d4a6ca8 2026-09-18 07:12 UTC · updated 2026-09-18 09:56 UTC |
cf-ux Codex promptfoo provider still defaults to the withdrawn gpt-5.4-mini slugSeverity: Minor Problem Reproduction, impact, suggested fix, verificationHow to reproduce
Expected behavior Actual behavior Impact Suggested correction How to verify Original location: tests/prompts/cf-ux/providers/codex_provider.py:16 -- inline anchoring could not be resolved after 1 attempt(s). |
|
|
||
|
|
||
| # OpenAI slugs are dated, not permanent. As of 2026-09-18 a ChatGPT-plan Codex | ||
| # account is entitled to gpt-5.6-luna / -sol / -terra, gpt-6-astra and gpt-5.5; |
There was a problem hiding this comment.
New comment inventories 5 entitled OpenAI slugs but the matrix only ever uses 3
Severity: Minor
Problem
The new comment at agents.py:807-815 states the account is entitled to gpt-5.6-luna / -sol / -terra, gpt-6-astra, and gpt-5.5, but only luna, terra, and gpt-6-astra are actually referenced in the base/override dicts for (codex, openai) and (cursor, openai). 'gpt-5.6-sol' never appears anywhere else in the repo and 'gpt-5.5' appears nowhere else in code (only as an unrelated separate 'GPT-5.5' literal in the copilot/openai matrix, a different display-name namespace).
Reproduction, impact, suggested fix, verification
How to reproduce
- Read agents.py:807-815 comment. 2. Grep the repo for 'gpt-5.6-sol' -> zero matches. 3. Grep for lowercase 'gpt-5.5' -> only the comment line itself.
Expected behavior
Either a fourth tier/override using gpt-5.6-sol or gpt-5.5 exists somewhere in the matrix, or the comment explains why sol/5.5 are deliberately unused (e.g., reserved for a future tier, or not a good fit for any current role).
Actual behavior
The comment inventories 5 entitled slugs but the code only maps 3, with no explanation for the discrepancy.
comment: 5 entitled slugs listed --> matrix: only 3 mapped (luna/terra/astra) --> sol/5.5 silently unused, unexplained
Impact
A future maintainer trusting the comment as authoritative may believe sol/5.5 are already wired in, or may not realize a tier mapping might have been missed when gpt-5.5 (the prior expensive value) was replaced by gpt-6-astra.
Suggested correction
Either wire gpt-5.6-sol/gpt-5.5 into an appropriate tier or override, or amend the comment to state explicitly that only luna/terra/astra are used and why sol/5.5 are intentionally unmapped.
How to verify
Re-read the comment against the matrix after the fix and confirm every slug named in the comment is either used somewhere in the matrix or explicitly called out as unused.
| "cf:tier:expensive": "gpt-5.5", | ||
| "cf:tier:cheap": "gpt-5.6-luna", | ||
| "cf:tier:balanced": "gpt-5.6-terra", | ||
| "cf:tier:expensive": "gpt-6-astra", |
There was a problem hiding this comment.
cf:tier:expensive's new gpt-6-astra value has no exact-match test anywhere
Severity: Minor
Problem
This PR changed cf:tier:expensive from 'gpt-5.5' to 'gpt-6-astra' for both (codex, openai) and (cursor, openai). A grep of tests/ for 'gpt-6-astra' returns zero matches; every other changed literal (gpt-5.6-luna, gpt-5.6-terra) received multiple renamed, exact-match tests, but the expensive tier did not.
Reproduction, impact, suggested fix, verification
How to reproduce
- Grep tests/ for 'gpt-6-astra' -> no results. 2. Inspect TestResolveModelIdCrossProduct.test_every_combination_resolves (or equivalent) -> it only asserts the resolved value is a non-empty string for all tiers including expensive.
Expected behavior
A dedicated test asserting _resolve_model_id(...) == 'gpt-6-astra' for the (codex, openai) and (cursor, openai) expensive base tier, mirroring the coverage given to luna/terra.
Actual behavior
No test in the suite would fail if the expensive tier's literal were accidentally reverted to 'gpt-5.5' or typo'd, since only a non-empty-string check exercises it.
cf:tier:expensive changed gpt-5.5 -> gpt-6-astra --> no exact-match test added --> future revert/typo passes CI silently
Impact
Silent regression risk specifically for the expensive tier -- likely used for the most critical/complex agent work -- would ship undetected by CI.
Suggested correction
Add exact-match assertions for _resolve_model_id("codex"/"cursor", "openai", "cf:tier:expensive",...) == "gpt-6-astra" analogous to the existing cheap/balanced tests.
How to verify
Add the test, then revert the expensive literal locally and confirm the new test fails.
| ) | ||
|
|
||
| def test_cheap_override_agents_resolve_to_gpt_5_4_on_codex(self): | ||
| def test_cheap_override_agents_bump_a_tier_on_codex(self): |
There was a problem hiding this comment.
cursor/openai cheap-tier overrides never get an exact-value pin, unlike codex's
Severity: Minor
Problem
test_agents_existing_snapshot.py has test_cheap_override_agents_bump_a_tier_on_codex which calls _resolve_model_id('codex', 'openai',...) and asserts the exact literal 'gpt-5.6-terra' for the cheap-tier overrides. The equivalent cursor test, test_cheap_override_agents_emit_sonnet_on_cursor, calls _agent_template_cursor which defaults to the anthropic provider and only asserts 'claude-sonnet-4-6'. No test calls _resolve_model_id('cursor', 'openai', cheap-override-triple) and asserts a literal, even though the (cursor, openai) matrix cell holds byte-identical override values to (codex, openai).
Reproduction, impact, suggested fix, verification
How to reproduce
- Read test_cheap_override_agents_bump_a_tier_on_codex (codex/openai, exact literal). 2. Read test_cheap_override_agents_emit_sonnet_on_cursor (cursor, but anthropic provider only). 3. Confirm no test exercises cursor+openai for this override triple with an exact-value assertion.
Expected behavior
Given codex/openai and cursor/openai share identical override literals, cursor's openai-provider override path should get the same exact-value pinning codex's does.
Actual behavior
cursor/openai's cheap-tier override path is only implicitly covered by the generic non-empty-string cross-product smoke test, so an accidental divergence between the two matrix cells for cursor's openai path would not be caught.
codex/openai override -> pinned exact test
cursor/openai override (same values) -> only smoke-tested (non-empty string)
Impact
A future edit that updates codex/openai's overrides but misses cursor/openai's (or vice versa) would not be caught by any exact-match test for the cursor+openai combination.
Suggested correction
Add a test analogous to test_cheap_override_agents_bump_a_tier_on_codex that calls _resolve_model_id('cursor', 'openai',...) for _CHEAP_OVERRIDE_AGENTS and asserts 'gpt-5.6-terra'.
How to verify
Add the test, then intentionally desync cursor/openai's override value and confirm the new test catches it.
| # account is entitled to gpt-5.6-luna / -sol / -terra, gpt-6-astra and gpt-5.5; | ||
| # the previous gpt-5.4 and gpt-5.4-mini had been withdrawn, and every agent on | ||
| # the cheap and balanced tiers -- 43 of the 44 in skills/studio/agents.toml -- | ||
| # answered a 400 at the point of use, because nothing validates a slug at |
There was a problem hiding this comment.
PR diagnoses 'nothing validates a slug at generate time' as root cause but ships no validation
Severity: Major
Problem
The new comment block (agents.py:807-815) explicitly names the root cause of the 43/44-agent outage as 'nothing validates a slug at generate time.' The diff's actual fix is entirely a literal-value swap in _MODEL_MATRIX; _resolve_model_id (agents.py:977-1012) adds no allowlist check, warning, or probe against the resolved model id -- it only performs dict lookups and returns the literal or a passthrough raw id.
Reproduction, impact, suggested fix, verification
How to reproduce
- Read the new comment at agents.py:807-815 which names the root cause. 2. Read _resolve_model_id's full body (977-1012) -- confirm it contains only base/override lookups and a passthrough branch, no validation against any known-good/known-bad slug list.
Expected behavior
Either a validation/guard mechanism added at generate time (e.g., checking resolved ids against a known-good allowlist, warning on stale slugs, or a smoke-test probe), or an explicit acknowledgment (issue link/TODO) that this remains a known, accepted gap.
Actual behavior
Only the hardcoded literal strings were swapped; no validation or guard exists anywhere in the resolve/generate pipeline, so the exact same silent-400 failure mode can recur on the next OpenAI slug rotation.
OpenAI rotates slugs (as it does periodically) --> _MODEL_MATRIX literals become stale --> _resolve_model_id returns stale literal unchecked --> generate pipeline emits it --> next live use hits 400, same as this outage
Impact
Recurrence of a production-wide breakage (43/44 agents) is guaranteed on the next vendor slug rotation, since the diagnosed root cause is explicitly left unaddressed by this fix.
Suggested correction
Add a lightweight validation step (e.g., an allowlist of currently-known-good slugs checked at generate time, or a CI/smoke-test job that probes each resolved model id against the live API) rather than relying solely on manual literal updates.
How to verify
Simulate a withdrawn slug in the matrix and confirm the new validation path surfaces a warning/error at generate time instead of silently emitting it.
ainetx
left a comment
ainetx
left a comment
There was a problem hiding this comment.
Changes requested: the PR's own commentary names the root cause of the outage and the diff doesn't fix it, so this would ship the same failure mode again under a different set of slugs.
- PR diagnoses the root cause but ships no validation — The new comment at agents.py:807-815 explicitly states the cause of the 43/44-agent outage was that "nothing validates a slug at generate time," but the actual change is only a literal-value swap in
_MODEL_MATRIX._resolve_model_idstill just does dict lookups and returns a literal or passthrough id — no allowlist check, no warning, no probe against the resolved model. Once these new slugs are withdrawn the same class of outage recurs. (comment)
2 participants
Follow-up to the Major finding on #239: the production model matrix names withdrawn models, so the outage that PR fixed in the test pilot exists for real users.
Measured, not assumed
Against a ChatGPT-plan Codex account:
Blast radius
(codex, openai)and(cursor, openai)resolved:gpt-5.4-minigpt-5.4gpt-5.5skills/studio/agents.tomlputs 29 agents on balanced, 14 on cheap, 1 on expensive — so 43 of 44 shipped agents were broken for Codex and Cursor+OpenAI users.It fails at run time, not generate time:
cfs generate-agentswrites the string with no validation, so the config looks correct and each agent answers a 400 when it is used.The mapping
Taken from OpenAI's own description of each model rather than chosen by taste, so the next re-pick is a reading rather than a judgement call:
gpt-5.6-lunagpt-5.6-terragpt-6-astraThe cheap overrides keep their meaning — cheap on a codebase task bumps one tier — and so become
gpt-5.6-terra.Tests
18 assertions resolve through the matrix and are updated. Those whose names carried a slug (
test_cheap_agents_resolve_to_gpt_5_4_mini_on_codex) are renamed after the tier they pin (test_cheap_agents_resolve_to_the_cheap_model_on_codex), so the next rotation is a value change and not a rename.Five fixtures pass a model string straight through a renderer; the value is arbitrary there and they pass either way, so they keep theirs.
Stale prose is corrected too — one docstring described a Codex-specialized model that no longer exists.
Deliberately untouched
(copilot, openai)names models by display name from GitHub Copilot's own catalogue (GPT-4.1/GPT-5.4/GPT-5.5). What Copilot offers is not what a Codex account is entitled to, and I have measured nothing against it. Changing it blind would be worse than leaving it. Now says so in the code.claude-sonnet-4-6,claude-opus-4-7andclaude-haiku-4-5were each probed live and answer.Limits of the evidence
The error says "when using Codex with a ChatGPT account". That is the account type I tested. On an API-key account
gpt-5.4may still exist — untested. Behaviour stays correct either way; the models simply change.(cursor, openai)is unverified in the same way: Cursor is a separate runtime with its own catalogue. It carried the identical slugs, so it is updated identically, but that is consistency rather than measurement.Spec
The CDSL step
inst-matrix-codex-openaipins the existence of each matrix cell — "tier-to-model base + role/target overrides" — not the slugs inside it. No step changes.Verified
6268 passed, 4 skipped, 15 xfailed;make spec-coverageandmake validategreen.Summary by CodeRabbit