Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion app.py
Original file line number Diff line number Diff line change
Expand Up @@ -147,7 +147,7 @@ def create_app() -> Flask:
register_gateway_mcp_routes(app, csrf)
register_documentation_routes(app, AuthService, ProxyService)
register_usage_routes(app, csrf)
register_gateway_extensions(app, callbacks=gateway_callbacks())
register_gateway_extensions(app, callbacks=gateway_callbacks(csrf=csrf))
# Restore the dashboard's recent requests from the durable ledger in the background.
usage_ledger.start(MetricsService.get_instance())

Expand Down
93 changes: 93 additions & 0 deletions docs/responses-state.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
# Hosted Responses continuation

`HOSTED_RESPONSES_ENABLED` enables principal-owned continuation on the managed
`POST /v1/responses` route. It defaults to off; an empty or malformed value is
off. A malformed value produces one warning without the supplied value. When
off, existing provider requests, response bytes, headers and storage behavior
are preserved, and retrieval and deletion retain their existing routing behavior.

Enable the flag in the Worker and Flask Container, select the existing D1
intelligence storage backend, and apply the additive `hosted_responses` migration.
The authority requires `INTELLIGENCE_DB` and the existing `multillm_media` R2
bucket. Missing bindings or a missing table produce a JSON 503 before provider
dispatch for a hosted request. Native Worker generation routes do not serve
hosted Responses; managed requests are forwarded to Flask.

## Requests

This body opts into gateway storage:

```json
{"model":"provider:model","input":"Summarize this text","store":true,"gateway_state":true}
```

A classified completed response returns an ID beginning with the reserved
gateway prefix `gwresp_`, followed by 32 hexadecimal characters. The gateway
removes `gateway_state` before dispatch and disables provider-side storage for
the opted-in generation. The returned response has `store: true`.

Continue using that ID and a new input, including tool results:

```json
{"model":"provider:model","previous_response_id":"gwresp_0123456789abcdef0123456789abcdef","input":[{"type":"function_call_output","call_id":"call_1","output":"result"}],"store":true,"gateway_state":true}
```

The gateway rebuilds the original input, completed output items and new input
in that order. It preserves item order and tool call/result boundaries. Each
stored child is self-contained and has its own expiry. A gateway parent can
also be used without `gateway_state: true`; the generation then uses the
retained context and does not create another hosted record.

Without hosted opt-in or a gateway parent, requests keep the existing behavior.
For example, `{"store":true,"previous_response_id":"resp_provider",...}`
continues through the provider unchanged. Combining hosted opt-in with a
provider-owned parent returns 400: supply full input or use a gateway parent,
because the gateway cannot retain unseen provider history. Hosted opt-in
requires the literal boolean `store: true`.

## Retrieval and deletion

Authenticated `GET /v1/responses/<id>` returns the stored completed response.
Authenticated `DELETE /v1/responses/<id>` removes its metadata and R2 body and
returns `{"id":"gwresp_...","object":"response.deleted","deleted":true}`.
Both operations accept only gateway IDs owned by the authenticated principal.
An absent, expired, malformed or foreign ID returns JSON 404. A foreign ID does
not reveal content or authorize deletion. Responses use `Cache-Control: no-store`.

Deletion is explicit. TTL expiry is the only automatic deletion condition;
expired records are removed when accessed. Failed deletions retain an
inaccessible metadata marker so a subsequent explicit DELETE can finish.
Missing or corrupt bodies return 503 without regeneration or automatic repair.

## Completion, limits and cost

State is published only after completed terminal status and settled existing
usage accounting, with measured or explicitly unknown usage. Failed,
incomplete, cancelled, deadline-expired and ambiguous outcomes cannot be read
or continued as completed state. Existing conservative accounting decisions
remain authoritative, including uncertain streaming handoffs under a deadline.
An unverified completed outcome returns `response_outcome_unknown` without
submitting the generation again. Storage acknowledgement lost after dispatch
also does not grant permission to repeat a provider call.

Streaming stores after a terminal `response.completed` event, clean EOF and
accounting settlement. The terminal event is held until storage succeeds.
An interrupted stream has no completed hosted state; errors are returned as
SSE error events once streaming has begun. There is no SSE reattachment.
Enabled managed idempotency can replay a completed nonstreaming response with
the same gateway ID; an unknown earlier outcome remains blocked. Streaming
idempotency replay is unsupported.

Each stored input plus response is limited to 1 MiB of UTF-8 JSON, expires
24 hours after creation, and belongs to a chain of at most 16 generations.
Oversized input and excessive depth return 400 before dispatch. An oversized
generated state returns 502 after dispatch and still incurs provider usage.
Storage errors return 503 and never trigger a provider retry. D1 stores owner,
status, provider/model, parent, policy revision, expiry, body pointer, byte
length and SHA-256; R2 bodies use the `responses-state/` prefix. The bounded
private hop is `POST http://intelligence.internal/v1/managed-state/responses`.

Zero-content retention returns 400 `retention_conflict` before dispatch for
hosted opt-in and gateway continuation. It also forbids retrieval; explicit
deletion remains available. Hosted storage introduces D1/R2 storage operations
in addition to existing provider costs. No prompt or response content is logged.
17 changes: 17 additions & 0 deletions intelligence-migrations/0028_responses_state.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
-- Only a completed row can expose its immutable response body.
CREATE TABLE IF NOT EXISTS hosted_responses (
id TEXT PRIMARY KEY,
owner TEXT NOT NULL,
status TEXT NOT NULL CHECK (status IN ('writing', 'failed', 'completed', 'deleting')),
provider TEXT NOT NULL,
model TEXT NOT NULL,
parent_id TEXT,
policy_revision TEXT NOT NULL,
depth INTEGER NOT NULL CHECK (depth BETWEEN 1 AND 16),
created_at INTEGER NOT NULL,
expires_at INTEGER NOT NULL,
body_key TEXT NOT NULL,
body_bytes INTEGER NOT NULL CHECK (body_bytes BETWEEN 1 AND 1048576),
body_sha256 TEXT NOT NULL
);
CREATE INDEX IF NOT EXISTS hosted_responses_owner_expiry ON hosted_responses (owner, expires_at);
2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
"lint": "ruff check . --select E9,F63,F7,F82",
"typecheck": "mypy --ignore-missing-imports --follow-imports=skip --disable-error-code import-untyped --disable-error-code var-annotated --disable-error-code arg-type --disable-error-code assignment --disable-error-code operator app.py routes services providers streaming",
"test": "python -m pytest -q",
"test:worker": "node --test tests/test_cloudflare_worker.mjs tests/test_cascades_worker.mjs tests/test_cascades_dashboard.mjs tests/test_image_quality_worker.mjs tests/test_secret_scan_worker.mjs tests/test_secret_firewall_worker.mjs tests/test_secret_firewall_coverage_worker.mjs tests/test_roleplay_compaction_budget_worker.mjs tests/test_roleplay_compaction_model_worker.mjs tests/test_roleplay_stream_validation_worker.mjs tests/test_aihubmix_worker.mjs tests/test_gpt_image_moderation_worker.mjs tests/test_dashboard_scripts.mjs tests/test_operations_dashboard.mjs tests/test_opencode_reasoning_worker.mjs tests/test_opencode_thinking_worker.mjs tests/test_nanogpt_reasoning_worker.mjs tests/test_nanogpt_speed_routing_worker.mjs tests/test_sse_heartbeat_worker.mjs tests/test_roleplay_worker.mjs tests/test_roleplay_config_worker.mjs tests/test_roleplay_glm_failover_worker.mjs tests/test_roleplay_intelligence_worker.mjs tests/test_roleplay_memory_worker.mjs tests/test_roleplay_compatibility_worker.mjs tests/test_roleplay_history_reconciliation_worker.mjs tests/test_roleplay_output_contract_analysis.mjs tests/test_roleplay_output_contract_cleanup_worker.mjs tests/test_roleplay_output_contract_nonstream_worker.mjs tests/test_roleplay_output_contract_worker.mjs tests/test_roleplay_performance_worker.mjs tests/test_roleplay_quality_routing_worker.mjs tests/test_roleplay_reasoning_worker.mjs tests/test_roleplay_session_storage_worker.mjs tests/test_roleplay_transport_failure_worker.mjs tests/test_roleplay_unlimited_worker.mjs tests/test_roleplay_workbench_worker.mjs tests/test_roleplay_candidate_context_worker.mjs tests/test_roleplay_operator_worker.mjs tests/test_workbench_stream.mjs tests/test_intelligence_worker.mjs tests/test_intelligence_d1_worker.mjs tests/test_intelligence_auth_worker.mjs tests/test_intelligence_outbound_worker.mjs tests/test_intelligence_operator.mjs tests/test_control_users_worker.mjs tests/test_control_audit_worker.mjs tests/test_access_sso_worker.mjs tests/test_d1_schema_worker.mjs tests/test_shadow_eval_worker.mjs tests/test_auto_routes_worker.mjs tests/test_ai_outbound_worker.mjs tests/test_knowledge_*.mjs tests/test_control_state_worker.mjs tests/test_route_health_worker.mjs tests/test_gateway_mcp_worker.mjs tests/test_usage_ledger_worker.mjs tests/test_media_outbound_worker.mjs tests/test_media_files_worker.mjs tests/test_media_jobs_worker.mjs tests/test_tool_repair_worker.mjs tests/test_prometheus_worker.mjs tests/test_evaluation_noise_floor_worker.mjs tests/test_prompt_templates_worker.mjs tests/test_config_snapshots_worker.mjs tests/test_managed_idempotency_worker.mjs tests/test_wave3_hub_worker.mjs tests/test_wave3_transport_worker.mjs tests/test_admission_leases_worker.mjs tests/test_edge_request_metrics_worker.mjs tests/test_request_cancellation_worker.mjs tests/test_mcp_contract_drift_worker.mjs tests/test_zero_content_retention_worker.mjs tests/test_prompt_cache_cost_worker.mjs tests/test_config_revision_sync_worker.mjs tests/test_gateway_alerts_worker.mjs tests/test_session_tiers_worker.mjs tests/test_usage_reservations_worker.mjs tests/test_shared_generation_cache_worker.mjs tests/test_deferred_tools_worker.mjs tests/test_generation_deadline_worker.mjs tests/test_wave4_worker.mjs tests/test_gateway_batches_worker.mjs tests/test_semantic_generation_cache_worker.mjs tests/test_canary_traffic_worker.mjs tests/test_context_pages_worker.mjs",
"test:worker": "node --test tests/test_cloudflare_worker.mjs tests/test_cascades_worker.mjs tests/test_cascades_dashboard.mjs tests/test_image_quality_worker.mjs tests/test_secret_scan_worker.mjs tests/test_secret_firewall_worker.mjs tests/test_secret_firewall_coverage_worker.mjs tests/test_roleplay_compaction_budget_worker.mjs tests/test_roleplay_compaction_model_worker.mjs tests/test_roleplay_stream_validation_worker.mjs tests/test_aihubmix_worker.mjs tests/test_gpt_image_moderation_worker.mjs tests/test_dashboard_scripts.mjs tests/test_operations_dashboard.mjs tests/test_opencode_reasoning_worker.mjs tests/test_opencode_thinking_worker.mjs tests/test_nanogpt_reasoning_worker.mjs tests/test_nanogpt_speed_routing_worker.mjs tests/test_sse_heartbeat_worker.mjs tests/test_roleplay_worker.mjs tests/test_roleplay_config_worker.mjs tests/test_roleplay_glm_failover_worker.mjs tests/test_roleplay_intelligence_worker.mjs tests/test_roleplay_memory_worker.mjs tests/test_roleplay_compatibility_worker.mjs tests/test_roleplay_history_reconciliation_worker.mjs tests/test_roleplay_output_contract_analysis.mjs tests/test_roleplay_output_contract_cleanup_worker.mjs tests/test_roleplay_output_contract_nonstream_worker.mjs tests/test_roleplay_output_contract_worker.mjs tests/test_roleplay_performance_worker.mjs tests/test_roleplay_quality_routing_worker.mjs tests/test_roleplay_reasoning_worker.mjs tests/test_roleplay_session_storage_worker.mjs tests/test_roleplay_transport_failure_worker.mjs tests/test_roleplay_unlimited_worker.mjs tests/test_roleplay_workbench_worker.mjs tests/test_roleplay_candidate_context_worker.mjs tests/test_roleplay_operator_worker.mjs tests/test_workbench_stream.mjs tests/test_intelligence_worker.mjs tests/test_intelligence_d1_worker.mjs tests/test_intelligence_auth_worker.mjs tests/test_intelligence_outbound_worker.mjs tests/test_intelligence_operator.mjs tests/test_control_users_worker.mjs tests/test_control_audit_worker.mjs tests/test_access_sso_worker.mjs tests/test_d1_schema_worker.mjs tests/test_shadow_eval_worker.mjs tests/test_auto_routes_worker.mjs tests/test_ai_outbound_worker.mjs tests/test_knowledge_*.mjs tests/test_control_state_worker.mjs tests/test_route_health_worker.mjs tests/test_gateway_mcp_worker.mjs tests/test_usage_ledger_worker.mjs tests/test_media_outbound_worker.mjs tests/test_media_files_worker.mjs tests/test_media_jobs_worker.mjs tests/test_tool_repair_worker.mjs tests/test_prometheus_worker.mjs tests/test_evaluation_noise_floor_worker.mjs tests/test_prompt_templates_worker.mjs tests/test_config_snapshots_worker.mjs tests/test_managed_idempotency_worker.mjs tests/test_wave3_hub_worker.mjs tests/test_wave3_transport_worker.mjs tests/test_admission_leases_worker.mjs tests/test_edge_request_metrics_worker.mjs tests/test_request_cancellation_worker.mjs tests/test_mcp_contract_drift_worker.mjs tests/test_zero_content_retention_worker.mjs tests/test_prompt_cache_cost_worker.mjs tests/test_config_revision_sync_worker.mjs tests/test_gateway_alerts_worker.mjs tests/test_session_tiers_worker.mjs tests/test_usage_reservations_worker.mjs tests/test_shared_generation_cache_worker.mjs tests/test_deferred_tools_worker.mjs tests/test_generation_deadline_worker.mjs tests/test_wave4_worker.mjs tests/test_responses_state_worker.mjs tests/test_gateway_batches_worker.mjs tests/test_semantic_generation_cache_worker.mjs tests/test_canary_traffic_worker.mjs tests/test_context_pages_worker.mjs",
"db:migrate": "python scripts/validate_sqlite_schema.py --apply",
"db:validate": "python scripts/validate_sqlite_schema.py",
"security:secrets": "python scripts/check_static_secrets.py",
Expand Down
Loading
Loading