fix: only send reasoning.effort to models that support it

[shehabyasser-scale]

Why this is now load-bearing, not defensive

This started as cleanup for a 400 seen during a local gpt-4o swap. It is now a prerequisite, because gpt-4o is the only usable eval model on the configured Azure resource.

Probing the endpoint directly, only two deployments answer — gpt-5.3-codex and gpt-4o. Everything else, including the committed gpt-5.4-mini-2026-03-17 that four of the five benchmarks point at, returns 404 DeploymentNotFound (details in #58). So moving swe-atlas-qna to gpt-4o is the only way to get it running at all, and without this guard every one of its agent calls trades a 404 for:

400 unsupported_parameter: 'reasoning.effort' is not supported with this model

The change

Each of the five bench agents built an unconditional "reasoning": {"effort": ...} into its Responses request. Now it is sent only when the model supports it:

_model = self._api_model.lower()
if _model.startswith(("gpt-5", "o1", "o3", "o4")) or "codex" in _model:
    request["reasoning"] = {"effort": "high"}

The predicate runs on _api_model, which is model_name.removeprefix("openai/") (e.g. atlas_agent/agent.py:80), so the openai/ prefix carried in build.yaml never defeats the startswith.

Both branches matter and both are covered by the run below:

• gpt-4o → no reasoning key → the 400 cannot occur.
• gpt-5.3-codex (producer) → matches on "codex" → still receives effort. gpt-5.4-mini-2026-03-17 → matches on gpt-5. So no reasoning model silently loses its effort setting.

Verification

Live combined run in flight on swe-atlas-qna with a gpt-4o agent, on a branch carrying this plus #51/#52/#53. The check is session/artifacts/inference/requests/*.jsonl: zero records with status 400 / unsupported_parameter, against 322/322 upstream failures in the last run. Posting the counts here before merge.

Greptile Summary

This PR replaces a provider-name heuristic ("fireworks" not in model) with a capability-based predicate (_is_reasoning_model) that guards reasoning.effort (and the max_completion_tokens vs max_tokens parameter name) on all six bench agents. The fix is a prerequisite for running swe-atlas-qna against gpt-4o, which was returning HTTP 400 unsupported_parameter on every call due to the unconditional reasoning block.

• Four agents (browsecomp-plus, officeqa, atlas, tau3) define a shared _is_reasoning_model() helper; the two Responses-API agents (gaia, swe-bench-pro) inline the same predicate. Both are correct but create six maintenance points for the same logic.
• officeqa correctly splits the old combined guard into two independent checks so that gpt-4o gets parallel_tool_calls=False (which it supports) without reasoning_effort (which it rejects).

Confidence Score: 5/5

Safe to merge — the predicate is correct, both API shapes (Chat Completions and Responses) are handled, and the officeqa guard split is accurate.

The change is narrowly scoped: a boolean predicate replaces a provider-name string check in six agents, and the affected parameter (reasoning.effort) is simply omitted rather than set to a wrong value when the model does not support it. The logic has been verified against live runs described in the PR. The only structural gap is that two of the six agents inline the predicate instead of using the helper defined in the other four, which is a maintainability concern but not a correctness issue.

Files Needing Attention: gaia/agent.py and swe-bench-pro/agent.py inline the predicate rather than using a helper — worth unifying in a follow-up if new reasoning model families need to be added.

Important Files Changed

Filename	Overview
harness-engineering-bench/browsecomp-plus/baseline/target/src/browsecomp_plus_agent/agent.py	Adds _is_reasoning_model() helper and gates both reasoning_effort and max_completion_tokens/max_tokens on it. Logic is correct and consistent with other helper-based agents.
harness-engineering-bench/gaia/baseline/target/src/gaia_agent/agent.py	Guards reasoning.effort behind an inlined predicate (no helper defined). Logic is correct but inconsistent with the 4 agents that use _is_reasoning_model(); creates 6 locations to update when new reasoning models are added.
harness-engineering-bench/officeqa/baseline/target/src/officeqa_agent/agent.py	Correctly splits the old combined fireworks guard into two independent checks: reasoning_effort gated on _is_reasoning_model(), parallel_tool_calls still gated on provider. gpt-4o now correctly receives parallel_tool_calls=False without the unsupported reasoning_effort.
harness-engineering-bench/swe-atlas-qna/baseline/target/src/atlas_agent/agent.py	Adds _is_reasoning_model() helper and updates both the token limit key and reasoning_effort guard. The key fix motivating the whole PR — allows gpt-4o to serve as the eval model without hitting a 400.
harness-engineering-bench/swe-bench-pro/baseline/target/src/swebench_pro_agent/agent.py	Guards reasoning.effort behind an inlined predicate (no helper). Same logic as gaia; creates the 5th and 6th duplication points for the predicate across the codebase.
harness-engineering-bench/tau3/baseline/target/src/tau3_agent/agent.py	Adds _is_reasoning_model() helper, correctly moves the token limit key selection and reasoning_effort guard. Change is clean and correct.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Agent builds request dict] --> B{_is_reasoning_model?}
    B -- "Yes: gpt-5*, o1*, o3*, o4*, *codex*" --> C[token key = max_completion_tokens]
    B -- "No: e.g. gpt-4o" --> D[token key = max_tokens]
    C --> E[Add reasoning effort param]
    D --> F[Skip reasoning param]
    E --> G{Uses Responses API?}
    F --> G
    G -- "Yes: gaia / swe-bench-pro" --> H[reasoning effort string]
    G -- "No: browsecomp / officeqa / atlas / tau3" --> I[reasoning_effort string]
    H --> J[Send request]
    I --> J
    J --> K{officeqa only}
    K -- "fireworks in model" --> L[skip parallel_tool_calls]
    K -- "not fireworks" --> M[parallel_tool_calls = False]
    L --> N[API call]
    M --> N

Loading

_{Reviews (4): Last reviewed commit: "fix: pick the token-limit parameter by m..." | Re-trigger Greptile}

[shehabyasser-scale]

Validated live, with a same-model A/B

Both halves of the guard are now confirmed against real runs, using the same agent model on both sides so the only variable is this patch.

Without the guard — GAIA, 2026-07-24, gpt-4o agent

runs/gaia-full/jobs/2026-07-24__18-09-28, request-log census:

1274 × (evaluation,   gpt-4o, 200)
 514 × (evaluation,   gpt-4o, 400)   <- unsupported_parameter: reasoning.effort
  79 × (evaluation,   gpt-4o, 500)
 296 × (finalization, gpt-4o, 200)
 201 × (finalization, gpt-4o, 400)   <- same
  12 × (finalization, gpt-4o, 500)
 152 × (producer, gpt-5.3-codex, 200)

715 HTTP 400s — roughly 30% of every inference call that run made, thrown away on a parameter the model does not accept.

With the guard — swe-atlas-qna, 2026-07-25, gpt-4o agent

runs/atlas-all-fixes/jobs/2026-07-25__08-58-16, same census:

284 × (evaluation, gpt-4o,        200)
147 × (producer,   gpt-5.3-codex, 200)

Zero 400s. Zero non-200 responses of any kind. usage.json agrees: evaluation: 284 requests / 0 upstream_errors / 245,524 tokens.

The negative case is confirmed too

The predicate must not quietly strip effort from models that want it. Two independent confirmations:

• Live: the 2026-07-25 07:33 run was already running guarded candidate code at gpt-5.4-mini-2026-03-17, and all 322 of its evaluation requests still carried reasoning: {effort: medium} — startswith("gpt-5") matches after the prefix strip.
• In this run: the producer is gpt-5.3-codex, which matches on "codex", and its 147 requests all succeeded.

So: non-reasoning models lose the parameter, reasoning models keep it.

One thing worth flagging for review

The guard is five copy-pasted lines in five separate bench agents, with no shared helper and no unit test — the evidence above is all end-to-end. That is a fair reviewer objection. I did not refactor it into a helper because these files are the candidate programs under optimization: the optimizer edits them, and a shared import outside the candidate repo would change what the candidate can see and modify. Worth a deliberate decision rather than an accident, though.

This is also no longer optional. Per #58, gpt-4o and gpt-5.3-codex are the only two deployments that answer on the configured Azure resource, so every benchmark that gets un-blocked has to run a non-reasoning agent, and each one 400s on every call without this.

[shehabyasser-scale]

Restacked and narrowed from five agents to two.

b7a5b80 added a gate to officeqa, tau3, swe-atlas-qna and browsecomp-plus, so three of my five hunks conflicted with it. I have dropped those three rather than rewriting them, and kept only gaia and swe-bench-pro, which have no gate at all. gaia still sends "reasoning": {"effort": "medium"} unconditionally at gaia_agent/agent.py:188.

One thing to flag rather than change unilaterally, since it is your code and a judgement call:

if "fireworks" not in self._api_model:
    kwargs["reasoning_effort"] = "medium"

That gate is provider-shaped, not capability-shaped. It correctly spares Fireworks-served open models, but it still sends reasoning_effort to any non-Fireworks model that is not a reasoning model, Azure gpt-4o being the case I measured. The capability form is what my remaining hunks use:

_model = self._api_model.lower()
if _model.startswith(("gpt-5", "o1", "o3", "o4")) or "codex" in _model:

which also excludes fireworks_ai/*, so it is a strict superset of the current behaviour. If you want, the clean merge is to keep your provider gate for parallel_tool_calls (gpt-4o does support that, so the capability gate would wrongly drop it) and use the capability gate for reasoning_effort alone. I did not make that change here because it is your gate and either reading is defensible.

Live measurement behind the PR, unchanged: gpt-4o without a guard on gaia burned 715 HTTP 400s of 2376 calls (514 evaluation + 201 finalization); gpt-4o with the guard on swe-atlas-qna logged 284 x (evaluation, gpt-4o, 200) and 147 x (producer, gpt-5.3-codex, 200), zero non-200s.

Related and separate, worth its own look: gaia is the one benchmark now pointed at a Fireworks model while still on the Responses API with a hosted web_search tool, parallel_tool_calls: False, and a bare AsyncOpenAI() with no max_retries=8. That is the combination 7d57386's message says does not translate via litellm.

Full suite matches base: 11 pre-existing failures, none new.

[shehabyasser-scale]

Restoring the three hunks I dropped, because the gap turned out to be load-bearing rather than stylistic. I said above that the provider gate "will bite whoever points this at OpenAI or Azure". It bit today, and it is worse than I described: it makes three of the five benchmarks unrunnable on any non-Fireworks upstream.

Measured against the configured Azure endpoint, running the swe-atlas-qna seed agent over its full 50-case held-out split:

target model	gate	result
gpt-5.3-codex	provider	50/50 BadRequestError, 400 The requested operation is unsupported
gpt-4o	provider	50/50 BadRequestError, 400 Unrecognized request argument supplied: reasoning_effort
gpt-4o	capability	0 BadRequestError, agent inference succeeded

Two separate things conspire, and the second is the one worth knowing:

1. gpt-4o is not Fireworks, so the guard sends it reasoning_effort, and it 400s.
2. gpt-5.3-codex cannot be the escape hatch, because it has no /chat/completions surface on this resource at all. Probed directly: 400 for every parameter combination, including a bare request with only max_completion_tokens. It answers on /responses only. Since 1b5ee0c / 7d57386 ported these three agents to Chat Completions, no reasoning-capable model on this endpoint can drive them.

So the intersection of "model this endpoint serves on Chat Completions" and "model the provider gate does not break" is empty. The capability gate is the only thing that makes swe-atlas-qna, tau3 and officeqa runnable here.

The change is a strict superset of current behaviour: fireworks_ai/* names do not match gpt-5/o1/o3/o4/codex either, so nothing changes for the default configuration.

officeqa needed care. It gates two arguments on the one condition. Only reasoning_effort moves to the capability check; parallel_tool_calls stays provider-gated, because that one really is a Fireworks-vs-OpenAI distinction and gpt-4o supports it. Collapsing both would have silently changed gpt-4o's tool-calling behaviour.

Verified after the change: all three files compile, and the follow-on run reached the verifier stage with zero BadRequestError, so the remaining failure there is a separate issue (the in-container rubric judge needs upstream credentials, which task_services_use_upstream supplies under vero and --ve supplies under raw harbor).