Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
91 changes: 91 additions & 0 deletions docs/output-schema-validation.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,91 @@
# Explicit output schema validation

Set `OUTPUT_SCHEMA_VALIDATION_ENABLED=true` to enable the explicit managed request
option `multillm_output_validation`. The default is `false`; an empty setting also
disables validation. Malformed settings disable validation and log one warning
without the setting value. When disabled, the option is neither interpreted nor
removed, and requests and responses retain their existing behavior.

Send the option on managed Chat Completions, Messages or Responses requests:

```json
{
"model": "mimo:mimo-v2.5",
"messages": [{"role": "user", "content": "Return a JSON object with an integer count."}],
"stream": false,
"multillm_output_validation": {
"mode": "strict",
"schema": {
"type": "object",
"properties": {"count": {"type": "integer"}},
"required": ["count"],
"additionalProperties": false
}
}
}
```

The gateway removes only this option before dispatch. It checks the generated text
as strict JSON against the explicit schema, preserving the original successful
response bytes and headers. For Chat Completions, each choice is checked. For
Messages and Responses, text blocks are joined in their original order. There is
no protocol translation solely for validation. The option does not instruct a
provider to generate JSON; use the provider's supported response format or prompt
alongside it. Existing response format and tool options are preserved.

With `OUTPUT_SCHEMA_VALIDATION_ENABLED=false`, the same request follows the
existing path, including the option. With validation enabled but the option
absent, the request and response follow the existing path. Raw passthrough and
Worker-native routes keep their existing behavior. Worker requests forwarded to
the managed Flask endpoints use the Flask validator.

Streaming requests with the option return HTTP 400
`output_validation_requires_nonstream` before provider dispatch. Malformed options
return HTTP 400 `output_validation_invalid_option`; invalid, oversized or unsupported
schemas return HTTP 400 `output_validation_invalid_schema`. Only `mode: "strict"`
and a schema are accepted in the option. Object and boolean schemas are supported.

Malformed generated JSON, schema mismatches, oversized responses and exhausted
validation limits return HTTP 502 `output_schema_violation`. For example:

```json
{
"error": {
"code": "output_schema_violation",
"message": "Explicit output validation failed",
"reason": "schema_mismatch",
"paths": ["/*"]
}
}
```

Error paths retain array indexes and replace all object keys with `*`, so generated
property names cannot disclose response content. Errors omit values, schema text,
validator messages and provider bodies. Reasons include `invalid_json`,
`schema_mismatch`, `invalid_response`, `unexpected_stream`, `body_limit` and
`work_limit`. Upstream HTTP errors retain their existing status, body and headers.
An unexpected successful stream is closed without consuming its content.

The response envelope is limited to 1 MiB. Schemas are limited to 64 KiB, depth 32
and 4,096 nodes. Generated JSON is limited to depth 32 and 4,096 nodes. The product
of schema and instance node counts is limited to 100,000. Validation shares a
10,000-keyword budget and 100 ms elapsed budget across choices, with at most 64
nested keyword evaluations and 100 error paths. Time is checked between keyword
evaluations; it is not a preemptive process deadline. `uniqueItems` supports arrays
of at most 128 items to bound pairwise comparisons.

Validation uses the installed `jsonschema` package with an empty reference
registry: no network or filesystem schema retrieval is allowed. Local references
are supported; unresolved references fail validation. Draft 4, 6, 7, 2019-09 and
2020-12 are supported, with 2020-12 as the default. Nested dialect and reference
scope changes and regex keywords (`pattern` and `patternProperties`) are rejected because they
cannot reliably observe the work budget. Format annotations do not enable format
assertions. Duplicate object keys, non-finite numbers and surrounding prose are
invalid JSON. This is bounded schema validation, not a universal schema engine
for native Worker traffic.

Validation never starts a second generation. Tool repair remains separately
opt-in and retains its existing retry policy. A provider may have already charged
for a response that fails validation; a validation error does not grant permission
to retry it. The validator adds no persistent storage or content logs. Existing
retention, request accounting and provider logging policies still apply.
53 changes: 53 additions & 0 deletions routes/tool_repair.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,10 @@
from services.protocol_translation import CHAT, translate_request
from services.tool_repair_runtime import HEADER, SKIP_REASK, log_report, repair_completion, repair_mode, summary_header
from services.tool_repair_stream import ToolCallBuffer, repair_sse
from services.output_schema_validation import (
MAX_BODY_BYTES, OutputValidationError, output_contents, prepare_validation, validate_contents, violation,
)
from services.free_json_contract import JsonOutputError, parse_json_output

MAX_RESPONSE_BYTES = 8 * 1024 * 1024

Expand Down Expand Up @@ -175,3 +179,52 @@ def reask(body):
return translate_downstream_response(repaired, source=CHAT, target=source,
stream=bool(payload.get("stream")), request_payload=payload)
return wrapped


def _validate_managed_response(upstream, contract, protocol):
response = as_flask_response(upstream)
if response.status_code >= 400:
return response
try:
if response.mimetype == "text/event-stream":
raise violation("unexpected_stream")
chunks, size = [], 0
for chunk in response.iter_encoded():
size += len(chunk)
if size > MAX_BODY_BYTES:
raise violation("body_limit")
chunks.append(chunk)
body = b"".join(chunks)
try:
envelope = parse_json_output(body.decode("utf-8"))
except (JsonOutputError, UnicodeError):
raise violation("invalid_response") from None
validate_contents(output_contents(envelope, protocol), contract)
response.set_data(body)
return response
except OutputValidationError:
response.close()
raise
except Exception:
response.close()
raise violation("invalid_response") from None


def with_managed_output_validation(dispatch, *, protocol=None):
"""Wrap the outer managed dispatcher, before routing and upstream handoff."""
@wraps(dispatch)
def wrapped(app, auth, metrics, proxy, payload, *args, **kwargs):
try:
prepared, contract = prepare_validation(payload, app.config)
except OutputValidationError as error:
return Response(json.dumps(error.body()), status=error.status, content_type="application/json")
response = dispatch(app, auth, metrics, proxy, prepared, *args, **kwargs)
if contract is None:
return response
endpoint = kwargs.get("endpoint") or (args[0] if args else None)
selected = protocol or ENDPOINT_PROTOCOLS.get(endpoint, "chat")
try:
return _validate_managed_response(response, contract, selected)
except OutputValidationError as error:
return Response(json.dumps(error.body()), status=error.status, content_type="application/json")
return wrapped
26 changes: 19 additions & 7 deletions services/free_json_contract.py
Original file line number Diff line number Diff line change
Expand Up @@ -59,17 +59,14 @@ def _check_bounds(value):
pending.extend((child, depth + 1) for child in children)


def _check_references(schema):
def iter_schema_nodes(schema):
"""Visit schema objects, excluding literal values in enum, const and defaults."""
pending = [schema]
while pending:
item = pending.pop()
if not isinstance(item, dict):
continue
for name in ("$ref", "$dynamicRef", "$recursiveRef"):
if name in item and (
not isinstance(item[name], str) or not item[name].startswith("#")
):
raise ValueError("Only local schema references are supported")
yield item
for key, value in item.items():
if key in _SCHEMA_MAPS and isinstance(value, dict):
pending.extend(value.values())
Expand All @@ -81,6 +78,15 @@ def _check_references(schema):
pending.extend(v for v in value.values() if isinstance(v, dict))


def _check_references(schema):
for item in iter_schema_nodes(schema):
for name in ("$ref", "$dynamicRef", "$recursiveRef"):
if name in item and (
not isinstance(item[name], str) or not item[name].startswith("#")
):
raise ValueError("Only local schema references are supported")


def _schema_validator(response_format):
wrapper = response_format.get("json_schema")
if (
Expand Down Expand Up @@ -158,7 +164,8 @@ def _finite_float(value):
return number


def check_json_output(content, response_format):
def parse_json_output(content):
"""Parse strict JSON once, rejecting duplicate keys and non-finite numbers."""
if not isinstance(content, str):
raise JsonOutputError("invalid_json")
try:
Expand All @@ -170,6 +177,11 @@ def check_json_output(content, response_format):
)
except (ValueError, RecursionError) as error:
raise JsonOutputError("invalid_json") from error
return parsed


def check_json_output(content, response_format):
parsed = parse_json_output(content)
if response_format["type"] == "json_object" and not isinstance(parsed, dict):
raise JsonOutputError("schema_mismatch")
if response_format["type"] == "json_schema":
Expand Down
Loading
Loading