fix(openai): latch the stream-only constraint and widen its detection

[M3gA-Mind]

Summary

Some OpenAI-compatible proxies refuse unary calls outright, answering a stream: false body with an HTTP 400 {"detail":"Stream must be set to true"}. ChatModel::invoke already falls back to the streaming path on that rejection (added in the requires_streaming change), but the fallback never converges, for two reasons.

1. The constraint is never remembered. requires_streaming was a plain bool that only with_requires_streaming could set — and nothing calls that builder, here or in the downstream host. It could not: ChatModel::invoke takes &self, so the auto-detected discovery had nowhere to be stored. Every unary call therefore re-paid a guaranteed-400 round trip before falling back: one wasted request per agent turn, indefinitely. This is the shape behind the linked issue's 511 events across 2 users — not a single failure, a per-turn one.

Made it an AtomicBool and latch it on first discovery, so only the first call pays and every later call goes straight to the streaming path. latch_stream_required logs on the transition only, so the discovery is visible exactly once per process rather than once per turn.

2. The detection was one case-sensitive literal. The trigger was err.message.contains("Stream must be set to true"), so any other phrasing still hard-failed the run. Worse, parse_error_body only promotes error.message / message to ProviderError::message, so a reason nested under any other key alongside a generic top-level message was invisible to the check.

Replaced it with is_stream_required_error, which:

• matches the wording family case-insensitively — stream must be set to true, 'stream' must be true, streaming is required, only streaming is supported, this deployment requires streaming, stream=true is required;
• searches the raw body as well as the decoded message, so a reason under detail / errors[].reason is still caught;
• accepts 422 alongside 400. The wording in the wild arrives in a FastAPI {"detail": …} envelope, and FastAPI's own field-validation status is 422 — a proxy that validates stream as a request field rather than raising an explicit HTTPException(400) lands there.

A stream mention is required in addition to the "must be true" phrasing, so unrelated 400s never get rerouted into the streaming path — degrade_for_400's tool_choice / response_format handlers and the /responses max_output_tokens retry keep their own bodies to themselves. The negative test matrix pins that.

API Or Behavior Changes

Public API

• OpenAiModel::requires_streaming() is now pub (was pub(super) + #[cfg(test)]), so a host can observe a constraint the transport learned at run time.
• OpenAiModel::with_requires_streaming(self, bool) — unchanged signature and semantics; it now seeds the latch instead of setting a plain field. Still useful to declare the constraint up front and skip even the one exploratory request.

Behavior

• A streaming-only endpoint now costs one rejected round trip per model instance instead of one per call. Previously-working behaviour is otherwise unchanged: the fallback still folds the SSE stream into a single ModelResponse, and the existing prompt-tool-call recovery on the streaming path is inherited unchanged.
• Rejections that only differ in wording (or nest the reason under a non-standard key), and rejections returned as 422, now trigger the fallback instead of failing the run.
• No change for providers that accept stream: false — nothing latches, and the detection requires an explicit streaming constraint in the error body.

Known gap, deliberately out of scope: the /responses path (responses_api_primary) returns from invoke before this check and sends stream: None, so a streaming-only Responses backend still surfaces the raw 400. Folding Responses-format SSE is separate work that the transport already defers ("True Responses SSE is a follow-up"); wiring this check there without it would only change which error is reported.

Tests

Added to src/harness/providers/openai/test.rs:

• stream_required_constraint_latches_after_discovery — the latch records the discovery and is idempotent.
• stream_required_latch_survives_through_a_shared_handle — visible through a shared Arc<OpenAiModel>, which is how hosts hold models and the whole reason this is an AtomicBool.
• is_stream_required_error_matches_the_known_proxy_wordings — 8 bodies including the exact one from the linked report, a lower-case variant, the alternative phrasings above, and one nested under a non-standard key alongside a generic top-level message; plus the 422 case.
• is_stream_required_error_ignores_unrelated_failures — wrong status (500), tool_choice, max_output_tokens, stream_options is not supported (mentions stream, states no requirement), and 'store' must be true (states a requirement, different field).

Commands run locally:

• cargo fmt --check
• cargo clippy --all-targets -- -D warnings — 0 warnings
• cargo clippy --all-targets --all-features -- -D warnings — 0 warnings
• cargo build --all-targets
• cargo build --all-targets --all-features
• cargo test — all suites green, 0 failed (3 ignored: the network-gated live_* tests)
• cargo test --all-features — all suites green, 0 failed

cargo test --lib alone is 1228 passed, 0 failed, and covers every test touched or added here.

Documentation

• Module doc in src/harness/providers/openai/mod.rs now describes the streaming-only rejection family, the fold, and the latch, next to the existing "local-server compatibility" paragraph on the tool_choice / response_format degradations.
• is_stream_required_error, STREAM_REQUIRED_STATUSES, latch_stream_required, and the stream_required field each carry doc comments explaining the rationale — in particular why interior mutability is required, so the next reader does not "simplify" it back to a bool and silently restore the per-call round trip.
• ChatModel::invoke's doc comment updated: the constraint may be declared or discovered, and a discovered one is latched.

Related

Fixes the provider-side root cause behind tinyhumansai/openhuman#5165 (tinyagents harness: openai HTTP 400 'Stream must be set to true'). Picking this up in the host is a submodule pin bump; no host-side code change is needed.

[greptile-apps]

M3gA-Mind has reached the 50-credit limit for trial accounts. To continue receiving code reviews, upgrade your plan.

[coderabbitai]

Warning

Review limit reached

You’ve reached a temporary PR review limit under our Fair Usage Limits Policy.

Your recent review volume is higher than typical usage, so adaptive limits are currently applied.

Next review available in: 32 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 235dd2e1-502f-4ab0-9d27-ee0ad26bc9be

📥 Commits

Reviewing files that changed from the base of the PR and between 801a77c and 2ae0d29.

📒 Files selected for processing (3)

• src/harness/providers/openai/mod.rs
• src/harness/providers/openai/test.rs
• src/harness/providers/openai/transport.rs

---

_{Comment @coderabbitai help to get the list of available commands.}

[sanil-23]

Approve. Well-reasoned root-cause fix, tight tests, thorough docs, CI green.

Solid:

• bool → AtomicBool is the correct fix — invoke(&self) behind Arc<dyn ChatModel> genuinely couldn't remember the discovery otherwise, so every turn re-paid a guaranteed 400. The doc comment warning against "simplifying" it back to a bool is a nice touch.
• Ordering::Relaxed is right: the latch is a standalone optimization flag with no happens-before to other state; a raced miss just pays one benign extra 400.
• Detection widening is properly guarded (status ∈ {400,422} AND a stream mention AND a phrase), and the negative matrix pins that tool_choice/max_output_tokens/stream_options/'store' must be true don't get hijacked.

Non-blocking nit (LOW): mentions_stream_required checks the two signals ("stream" + a phrase) independently anywhere in the body, no proximity — so a body co-mentioning both in unrelated contexts, e.g. {"error":{"message":"stream_options is not supported","detail":"temperature must be true"}}, would false-positive and latch streaming for the instance's lifetime. The negative tests cover each half in isolation but not the co-occurrence in one body. Consequence is mild (a wrong latch still yields correct streamed responses), so this is a robustness nit — worth either adding a co-occurrence case to the negative matrix or tightening to require proximity, but not a blocker.

/responses gap and the concurrency imprecision of "only the first call pays" are both correctly acknowledged/scoped. Ship it — fixes a live 511-event/2-user issue (openhuman#5165), host pickup is just a submodule pin bump.