Raise ai-gateway body-authz request-body cap to 32MiB (EAI-8489) - #832
Conversation
The body-aware extAuth on ai-gateway buffers the request body at maxRequestBytes=64KiB before authorizing header-less (model-in-body) requests. Envoy returns 413 for any larger body (skipping auth, and overriding failOpen), so standard OpenAI/Anthropic clients sending 200KB-2MB contexts are rejected gateway-wide, for every model. Parameterize as aiGateway.bodyAuthMaxRequestBytes (default 32MiB) so large-context and image-bearing inference turns are not rejected; the value sits under the 50Mi connection bufferLimit and is tunable per env. EAI-8489
Follow-up on the 32MiB commit on this branch. - 4MiB, not 32MiB. The ai-gateway-discovery authz pod that receives the buffered body is capped at limits.memory 256Mi, and an in-flight header-less request is held both by Envoy and by that handler, so 32MiB is more than the current deployment can absorb. 4MiB covers a ~200k-token turn (~1MB) with headroom. - Drive both charts from global.aiGateway.bodyAuthMaxRequestBytes. The gateway-scoped policy here is not the only cap on the header-less path: ai-gateway-discovery stamps per-model catch-all SecurityPolicies (sectionName route-not-found) that are rule-scoped and override this one per route. They carry their own ceiling, so the two must move together. - Render through int64 and fail on a non-positive value. A bare number in a values file is a float64, so the root wiring emitted 4.194304e+06 and int64 casts that to 0, which would have published maxRequestBytes: 0 instead of failing. The matching ai-gateway-discovery change lands in silogen/core.
|
Pushed a commit on top here, hope that is ok. Two things came out of digging into this. The gateway-scoped policy is not the one in the path. ai-gateway-discovery also stamps per-model catch-all SecurityPolicies (sectionName route-not-found) with their own hardcoded 65536, and being rule-scoped they win over the gateway one on every model route. On app-dev all three sit at 65536 today, so this PR on its own would not have lifted the 413. Companion: silogen/core#4520, both need to land. Dropped 32MiB to 4MiB. The authz pod receiving the buffered body is capped at 256Mi memory, and its handler was truncating bodies at 256KiB anyway (fixed in the core PR), so 32MiB was not reachable. 4MiB covers a 200k-token turn with headroom. Also: wiring the value in root values rendered it as 4.194304e+06, and int64 casts that to 0, which would have quietly published maxRequestBytes: 0. It renders through int64 now and the template fails on a non-positive value. Title and description still say 32MiB, left those for you. |
3 participants
Problem
The shared Envoy AI Gateway (
ai-gateway) authorizes header-less requests — model in the JSON body, nox-ai-eg-model/x-ai-eg-backendrouting header — via a body-aware extAuth call toai-gateway-discovery. That extAuth buffers the request body atmaxRequestBytes: 65536(64 KiB). Per the Envoy Gateway API, a body over that limit returns HTTP 413 and skips authorization entirely (this precedence holds even overfailOpen).So any large request on the header-less path is rejected before it reaches the model. This is gateway-wide — it hits every model on
ai-gateway(verified on Qwen3.8-27B and MiniMax-M2.5), not one model. Standard OpenAI/Anthropic clients (e.g. Claude Code) put the model in the body and routinely send 200 KB–2 MB requests (a ~200k-token coding turn is ~1 MB; image-bearing turns are larger), so they 413 consistently.Evidence (direct to the gateway, bypassing the translation proxy): ~67 KB body passes, 120 KB+ returns HTTP 413 (HTTP/2,
text/plain, Envoy listener shape — not the gateway JSON error). The live Envoy config on theai-gatewaypod showswith_request_body.max_request_bytes: 65536on the BUFFERED body filter.Change
Parameterize the limit as
aiGateway.bodyAuthMaxRequestBytes(default 32 MiB = 33554432) and reference it from theai-gateway-default-denySecurityPolicy. 32 MiB matches clients' own request ceiling and sits under the existingai-gatewayClientTrafficPolicyconnectionbufferLimit(50 Mi). ThebodyToExtAuthAPI exposes onlymaxRequestBytes(noallowPartialMessage), so raising the ceiling is the only lever in the current Envoy Gateway version.Rendered via
int64to avoid Helm emitting the value in scientific notation (which would fail the integer schema).Tradeoff: a larger buffer increases the memory the extAuth path can hold per in-flight header-less request. The value is now tunable per environment, so an overlay can lower it.
Test plan
helm templaterendersmaxRequestBytes: 33554432(integer, not3.3e+07) whenaiGateway.enabled=true; renders nothing when disabled.ai-gatewayenabled: a >64 KB header-less request that previously 413'd is authorized and reaches the model; small-body auth behavior unchanged.Fixes EAI-8489.