[EAI-8489]: Raise ai-gateway body-authz request-body cap to 4MiB - #834
Merged
Conversation
The body-aware extAuth on ai-gateway buffers the request body at maxRequestBytes=64KiB before authorizing header-less (model-in-body) requests. Envoy returns 413 for any larger body (skipping auth, and overriding failOpen), so standard OpenAI/Anthropic clients sending 200KB-2MB contexts are rejected gateway-wide, for every model. Parameterize as aiGateway.bodyAuthMaxRequestBytes (default 32MiB) so large-context and image-bearing inference turns are not rejected; the value sits under the 50Mi connection bufferLimit and is tunable per env. EAI-8489 (cherry picked from commit 758ee8b)
Follow-up on the 32MiB commit on this branch. - 4MiB, not 32MiB. The ai-gateway-discovery authz pod that receives the buffered body is capped at limits.memory 256Mi, and an in-flight header-less request is held both by Envoy and by that handler, so 32MiB is more than the current deployment can absorb. 4MiB covers a ~200k-token turn (~1MB) with headroom. - Drive both charts from global.aiGateway.bodyAuthMaxRequestBytes. The gateway-scoped policy here is not the only cap on the header-less path: ai-gateway-discovery stamps per-model catch-all SecurityPolicies (sectionName route-not-found) that are rule-scoped and override this one per route. They carry their own ceiling, so the two must move together. - Render through int64 and fail on a non-positive value. A bare number in a values file is a float64, so the root wiring emitted 4.194304e+06 and int64 casts that to 0, which would have published maxRequestBytes: 0 instead of failing. The matching ai-gateway-discovery change lands in silogen/core. (cherry picked from commit b4a3d54)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Backport of #832 (plus the follow-up commit on that branch) to
release/v2.4.x. Cherry-picked with-x, both commits preserved.bodyToExtAuth.maxRequestByteson theai-gateway-default-denySecurityPolicy asaiGateway.bodyAuthMaxRequestBytes, default 4MiB. The upstream 64KiB default returns HTTP 413 and skips authorization entirely (ahead offailOpen), so every large request on the header-less path was rejected before reaching a model.global.aiGateway.bodyAuthMaxRequestBytesinroot/values.yaml, so the gateway-scoped policy and the per-model catch-all policies stamped byai-gateway-discoverycannot drift. The catch-all policies are rule-scoped and override the gateway-scoped one per route, so a mismatch is silently decided by whichever is lower.int64and fails the template on a non-positive value. A bare number in a values file is a float64, so an unguarded render emits4.194304e+06, whichint64casts to0and would publishmaxRequestBytes: 0.Needs the matching
ai-gateway-discoverychange, silogen/core#4520, to be effective on any cluster with models deployed.https://jira.amd.com/browse/EAI-8489
How to test
Send a header-less inference request larger than 64KiB (model in the JSON body, no
x-ai-eg-modelheader) with a valid API key. It returned 413 before and should now be served. Verified on app-dev againstMiniMaxAI/MiniMax-M2.5: 40KB and 200KB return 200, 5MB returns 413 at the new ceiling, and no key returns 401.