Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
67 changes: 67 additions & 0 deletions docs/request-cancellation.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
# Disconnect-aware upstream cancellation

Cleanup is always enabled. There is no environment flag or disabled mode.
Successful bytes, status codes and headers are preserved. No response content,
credentials or request bodies are added to cancellation outcomes or logs.

For example, closing a streaming client before its first event, after a partial
event, or while waiting for the next event closes the upstream owner once.
Reading the whole response keeps its original bytes. Aborting after a completed
response does not change that completed outcome.

## Python transport

`bind_cancellation(response, on_outcome=callback)` attaches a
`CancellationContext` to `response.multillm_cancellation` and shares its
idempotent close operation with existing owners. `iter_stream_content` and
`iter_stream_lines` return closable iterators immediately, before generator
execution. `GeneratorExit`, read exceptions, normal exhaustion and redundant
close calls release the upstream once. A socket error propagates; it does not
restart reads through a different iterator.

Unified dispatch binds each returned response without changing request bodies
or existing origin-fallback policy. Flask body ownership closes the underlying
iterable even when no body iteration occurred. The stream preflight (`preflight_chat_stream`)
closes that same owner on rejection; validated prefix replay keeps the same
owner.

WSGI detects disconnects during writes or iteration. It cannot reliably detect
a disconnect while a synchronous provider setup or socket read is blocked.
Existing bounded connect/read timeouts remain in force. No per-request watcher
thread is created, and cancellation cannot promise the provider stopped billing.

## Worker transport

`UpstreamCancellation` owns an AbortController, a bounded header timer, the
parent abort listener and an upstream reader. Parent abort propagates to fetch
through its signal. The listener remains active after headers until the body
finishes or closes. Cancelling a downstream body also aborts fetch and cancels
the upstream reader once. Cleanup does not wait for a provider's cancellation
acknowledgement. `readBoundedUpstreamBytes` owns cancellation and reader release
for bounded nonstream reads, including abort and size rejection.

Roleplay uses this owner for generation and compaction. A client abort, including
one racing an HTTP rejection, stops candidate and origin fallback. Existing
non-cancellation fallback decisions remain unchanged. Terminal HTTP error bodies
retain their owner until consumption or cancellation instead of detaching at
headers. Header timers end at headers; compaction retains its total timer.

## Outcomes and settlement

Python callbacks receive `CancellationOutcome`; Worker callbacks receive an
object with `reason`, `ambiguous`, `usageState`, `usage` and `replayPermission`.
Python also reports an [`UpstreamOutcome`](upstream-outcomes.md). After upstream handoff,
cancellation or interruption is ambiguous, usage is unknown (`None`/`null`),
and replay permission is false. Nothing here asserts zero tokens or zero cost.
Callbacks run once and contain no generated content. Cleanup errors are logged
by type only and do not mask the original result.

These are transient transport outcomes, not durable accounting records.

## Coverage

Unified dispatch, the stream preflight and the roleplay transport use these
owners. Other native Worker provider fetches and the provider-specific streaming
paths in `services/proxy_service.py` keep their existing connect and read
timeouts. Whether a provider stops billing after a disconnect depends on that
provider.
2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
"lint": "ruff check . --select E9,F63,F7,F82",
"typecheck": "mypy --ignore-missing-imports --follow-imports=skip --disable-error-code import-untyped --disable-error-code var-annotated --disable-error-code arg-type --disable-error-code assignment --disable-error-code operator app.py routes services providers streaming",
"test": "python -m pytest -q",
"test:worker": "node --test tests/test_cloudflare_worker.mjs tests/test_cascades_worker.mjs tests/test_cascades_dashboard.mjs tests/test_image_quality_worker.mjs tests/test_secret_scan_worker.mjs tests/test_secret_firewall_worker.mjs tests/test_secret_firewall_coverage_worker.mjs tests/test_roleplay_compaction_budget_worker.mjs tests/test_roleplay_compaction_model_worker.mjs tests/test_roleplay_stream_validation_worker.mjs tests/test_aihubmix_worker.mjs tests/test_gpt_image_moderation_worker.mjs tests/test_dashboard_scripts.mjs tests/test_operations_dashboard.mjs tests/test_opencode_reasoning_worker.mjs tests/test_opencode_thinking_worker.mjs tests/test_nanogpt_reasoning_worker.mjs tests/test_nanogpt_speed_routing_worker.mjs tests/test_sse_heartbeat_worker.mjs tests/test_roleplay_worker.mjs tests/test_roleplay_config_worker.mjs tests/test_roleplay_glm_failover_worker.mjs tests/test_roleplay_intelligence_worker.mjs tests/test_roleplay_memory_worker.mjs tests/test_roleplay_compatibility_worker.mjs tests/test_roleplay_history_reconciliation_worker.mjs tests/test_roleplay_output_contract_analysis.mjs tests/test_roleplay_output_contract_cleanup_worker.mjs tests/test_roleplay_output_contract_nonstream_worker.mjs tests/test_roleplay_output_contract_worker.mjs tests/test_roleplay_performance_worker.mjs tests/test_roleplay_quality_routing_worker.mjs tests/test_roleplay_reasoning_worker.mjs tests/test_roleplay_session_storage_worker.mjs tests/test_roleplay_transport_failure_worker.mjs tests/test_roleplay_unlimited_worker.mjs tests/test_roleplay_workbench_worker.mjs tests/test_roleplay_candidate_context_worker.mjs tests/test_roleplay_operator_worker.mjs tests/test_workbench_stream.mjs tests/test_intelligence_worker.mjs tests/test_intelligence_d1_worker.mjs tests/test_intelligence_auth_worker.mjs tests/test_intelligence_outbound_worker.mjs tests/test_intelligence_operator.mjs tests/test_control_users_worker.mjs tests/test_control_audit_worker.mjs tests/test_access_sso_worker.mjs tests/test_d1_schema_worker.mjs tests/test_shadow_eval_worker.mjs tests/test_auto_routes_worker.mjs tests/test_ai_outbound_worker.mjs tests/test_knowledge_*.mjs tests/test_control_state_worker.mjs tests/test_route_health_worker.mjs tests/test_gateway_mcp_worker.mjs tests/test_usage_ledger_worker.mjs tests/test_media_outbound_worker.mjs tests/test_media_files_worker.mjs tests/test_media_jobs_worker.mjs tests/test_tool_repair_worker.mjs tests/test_prometheus_worker.mjs tests/test_evaluation_noise_floor_worker.mjs tests/test_prompt_templates_worker.mjs tests/test_config_snapshots_worker.mjs",
"test:worker": "node --test tests/test_cloudflare_worker.mjs tests/test_cascades_worker.mjs tests/test_cascades_dashboard.mjs tests/test_image_quality_worker.mjs tests/test_secret_scan_worker.mjs tests/test_secret_firewall_worker.mjs tests/test_secret_firewall_coverage_worker.mjs tests/test_roleplay_compaction_budget_worker.mjs tests/test_roleplay_compaction_model_worker.mjs tests/test_roleplay_stream_validation_worker.mjs tests/test_aihubmix_worker.mjs tests/test_gpt_image_moderation_worker.mjs tests/test_dashboard_scripts.mjs tests/test_operations_dashboard.mjs tests/test_opencode_reasoning_worker.mjs tests/test_opencode_thinking_worker.mjs tests/test_nanogpt_reasoning_worker.mjs tests/test_nanogpt_speed_routing_worker.mjs tests/test_sse_heartbeat_worker.mjs tests/test_roleplay_worker.mjs tests/test_roleplay_config_worker.mjs tests/test_roleplay_glm_failover_worker.mjs tests/test_roleplay_intelligence_worker.mjs tests/test_roleplay_memory_worker.mjs tests/test_roleplay_compatibility_worker.mjs tests/test_roleplay_history_reconciliation_worker.mjs tests/test_roleplay_output_contract_analysis.mjs tests/test_roleplay_output_contract_cleanup_worker.mjs tests/test_roleplay_output_contract_nonstream_worker.mjs tests/test_roleplay_output_contract_worker.mjs tests/test_roleplay_performance_worker.mjs tests/test_roleplay_quality_routing_worker.mjs tests/test_roleplay_reasoning_worker.mjs tests/test_roleplay_session_storage_worker.mjs tests/test_roleplay_transport_failure_worker.mjs tests/test_roleplay_unlimited_worker.mjs tests/test_roleplay_workbench_worker.mjs tests/test_roleplay_candidate_context_worker.mjs tests/test_roleplay_operator_worker.mjs tests/test_workbench_stream.mjs tests/test_intelligence_worker.mjs tests/test_intelligence_d1_worker.mjs tests/test_intelligence_auth_worker.mjs tests/test_intelligence_outbound_worker.mjs tests/test_intelligence_operator.mjs tests/test_control_users_worker.mjs tests/test_control_audit_worker.mjs tests/test_access_sso_worker.mjs tests/test_d1_schema_worker.mjs tests/test_shadow_eval_worker.mjs tests/test_auto_routes_worker.mjs tests/test_ai_outbound_worker.mjs tests/test_knowledge_*.mjs tests/test_control_state_worker.mjs tests/test_route_health_worker.mjs tests/test_gateway_mcp_worker.mjs tests/test_usage_ledger_worker.mjs tests/test_media_outbound_worker.mjs tests/test_media_files_worker.mjs tests/test_media_jobs_worker.mjs tests/test_tool_repair_worker.mjs tests/test_prometheus_worker.mjs tests/test_evaluation_noise_floor_worker.mjs tests/test_prompt_templates_worker.mjs tests/test_config_snapshots_worker.mjs tests/test_request_cancellation_worker.mjs",
"db:migrate": "python scripts/validate_sqlite_schema.py --apply",
"db:validate": "python scripts/validate_sqlite_schema.py",
"security:secrets": "python scripts/check_static_secrets.py",
Expand Down
14 changes: 10 additions & 4 deletions routes/unified_transport.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@
)
from providers.image_relays import image_relay_backup_base_url
from route_helpers import copy_raw_provider_response_headers
from services.request_cancellation import bind_cancellation

IMAGE_GENERATION_TIMEOUT = (10, 600)

Expand All @@ -29,15 +30,20 @@ def send_unified_provider_request(
request_headers: Mapping[str, Any],
):
"""Send one unified request with bounded, replay-safe origin fallback."""
def send_owned_request(kwargs):
response = proxy_service_cls.make_request(**kwargs)
bind_cancellation(response)
return response

secondary_origin = secondary_origin or image_relay_backup_base_url(provider)
if secondary_origin is None and provider != "aihubmix":
return proxy_service_cls.make_request(**request_kwargs)
return send_owned_request(request_kwargs)
if secondary_origin is None:
raise APIError("AIHubMix backup origin is not configured", status_code=500)

return request_with_origin_fallback(
lambda origin: proxy_service_cls.make_request(
**{
lambda origin: send_owned_request(
{
**request_kwargs,
"url": build_aihubmix_url(origin, upstream_path),
}
Expand Down Expand Up @@ -158,4 +164,4 @@ def normalized_aihubmix_image_response(
headers=copy_raw_provider_response_headers(response.headers),
)
finally:
response.close()
bind_cancellation(response).complete()
144 changes: 144 additions & 0 deletions services/request_cancellation.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,144 @@
"""One cleanup owner and content-free outcome for an upstream handoff."""

from __future__ import annotations

import logging
from collections.abc import Callable, Iterator
from dataclasses import dataclass
from typing import Any, Generic, TypeVar

from flask import Response

from services.upstream_outcome import UpstreamOutcome, classify_upstream_outcome

logger = logging.getLogger(__name__)
T = TypeVar("T")


@dataclass(frozen=True)
class CancellationOutcome:
reason: str
ambiguous: bool
upstream: UpstreamOutcome
usage_state: str = "unknown"
usage: None = None


class CancellationContext:
"""Own close without spawning a disconnect watcher or authorizing replay."""

def __init__(self, close: Callable[[], Any], *, on_outcome=None):
self._close = close
self._on_outcome = on_outcome
self.handed_off = False
self.closed = False
self.outcome: CancellationOutcome | None = None

def handoff(self) -> None:
self.handed_off = True

def _finish(self, reason: str, upstream: UpstreamOutcome) -> None:
if self.closed:
return
self.closed = True
self.outcome = CancellationOutcome(
reason, self.handed_off and reason != "complete", upstream,
)
try:
self._close()
except Exception as error:
logger.warning("Upstream cleanup failed type=%s", type(error).__name__)
if self._on_outcome is not None:
try:
self._on_outcome(self.outcome)
except Exception as error:
logger.warning("Cancellation observer failed type=%s", type(error).__name__)

def complete(self) -> None:
self._finish("complete", classify_upstream_outcome(200))
Comment on lines +57 to +58

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Classify completion using the actual response status

Every exhausted response is classified as HTTP 200 here, regardless of its real status. For example, fully consuming an owned 401, 429, or 503 response produces an UpstreamOutcome reporting provider success and accepted credentials, contradicting the repository's status classification and potentially corrupting health or credential accounting once this callback is consumed. Capture the response status when binding and classify completion with that value.

Useful? React with 👍 / 👎.


def cancel(self) -> None:
self._finish("cancelled", classify_upstream_outcome(cancelled=True))

def fail(self) -> None:
self._finish(
"interrupted", classify_upstream_outcome(transport_failure="interrupted"),
)

def close(self) -> None:
self.cancel()


class CancellationIterator(Iterator[T], Generic[T]):
"""A closable lazy iterator, including before its generator has started."""

def __init__(self, factory: Callable[[], Iterator[T]], context: CancellationContext,
*, close_source: Callable[[], Any] | None = None):
self._factory = factory
self.context = context
self._source: Iterator[T] | None = None
self._close_source = close_source
self._closed = False

def __iter__(self) -> CancellationIterator[T]:
return self

def __next__(self) -> T:
if self._closed or self.context.closed:
raise StopIteration
try:
if self._source is None:
self._source = iter(self._factory())
return next(self._source)
except StopIteration:
self.context.complete()
self.close()
raise
except GeneratorExit:
self.close()
raise
except BaseException:
self.context.fail()
self.close()
raise

def throw(self, error):
# GeneratorExit from a downstream owner must never resume the socket.
self.close()
raise error

def close(self) -> None:
if self._closed:
return
self._closed = True
close = self._close_source or getattr(self._source, "close", None)
try:
if close is not None:
close()
except Exception as error:
logger.warning("Stream cleanup failed type=%s", type(error).__name__)
finally:
self.context.close()


def bind_cancellation(response, *, on_outcome=None) -> CancellationContext:
"""Share close ownership with existing transport, WSGI and preflight owners.

Flask bodies must themselves own hidden provider resources. A transport
creating a lazy generator should bind its upstream before building that body.
"""
context = getattr(response, "multillm_cancellation", None)
if isinstance(context, CancellationContext):
return context
Comment on lines +130 to +132

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Attach observers when rebinding an owned response

If a response already has a cancellation context, this returns it without applying the newly supplied on_outcome. Unified dispatch now binds every response without an observer, so a later accounting or settlement layer calling the documented bind_cancellation(response, on_outcome=callback) API will silently receive the existing context and its callback will never run. Preserve or register the new observer when reusing a context, or arrange for the initial unified binding to receive it.

Useful? React with 👍 / 👎.

# Test doubles and some adapters have no close(); ownership still applies.
close = getattr(response, "close", None)
context = CancellationContext(close if callable(close) else (lambda: None), on_outcome=on_outcome)
context.handoff()
response.multillm_cancellation = context
response.close = context.close
if isinstance(response, Response):
source = response.response
response.response = CancellationIterator(
lambda: iter(source), context, close_source=getattr(source, "close", None),
Comment on lines +138 to +142

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Mark fully consumed raw responses as complete

For unified requests returning a requests.Response, this replaces close() but only wraps Flask Response bodies. The raw passthrough path in stream_upstream_response() therefore consumes the complete body and invokes this overridden close() from its finally, which records cancelled with ambiguous=true instead of completion. Normal raw chat, image, and protocol responses will consequently expose an uncertain cancellation outcome even when every byte was delivered; the raw-body iterator needs to signal complete() on EOF before cleanup.

Useful? React with 👍 / 👎.

)
return context
41 changes: 18 additions & 23 deletions services/upstream_transport.py
Original file line number Diff line number Diff line change
@@ -1,28 +1,23 @@
"""Resource-safe helpers for consuming provider HTTP responses."""

import logging
from typing import Generator
from collections.abc import Iterator

import requests

from streaming.sse import iter_sse_data

logger = logging.getLogger(__name__)

from services.request_cancellation import CancellationIterator, bind_cancellation

def close_retry_response(response: requests.Response) -> None:
"""Release a retryable response without masking the original result."""
try:
response.close()
except Exception as error:
logger.warning(
"Retry response cleanup failed type=%s",
type(error).__name__,
)
bind_cancellation(response).close()


def iter_stream_content(response: requests.Response) -> Generator[bytes, None, None]:
def iter_stream_content(response: requests.Response) -> Iterator[bytes]:
"""Yield decoded upstream bytes as soon as the socket exposes them."""
return CancellationIterator(lambda: _stream_content(response), bind_cancellation(response))


def _stream_content(response: requests.Response) -> Iterator[bytes]:
raw_response = getattr(response, "raw", None)
raw_read1 = getattr(raw_response, "read1", None)
if callable(raw_read1):
Expand All @@ -35,21 +30,21 @@ def iter_stream_content(response: requests.Response) -> Generator[bytes, None, N
yield from response.iter_content(chunk_size=128)


def iter_stream_lines(response: requests.Response) -> Generator[str, None, None]:
def iter_stream_lines(response: requests.Response) -> Iterator[str]:
"""Own the stream before the first read, including preflight rejection."""
return CancellationIterator(lambda: _stream_lines(response), bind_cancellation(response))


def _stream_lines(response: requests.Response) -> Iterator[str]:
"""Parse SSE responses incrementally, with a line-based fallback."""
content_type = response.headers.get("content-type", "").lower()
if content_type.startswith("text/event-stream") and hasattr(
response,
"iter_content",
):
try:
for data_payload in iter_sse_data(iter_stream_content(response)):
yield f"data: {data_payload}"
return
except Exception as error:
logger.warning(
"SSE parser failed; falling back to line iteration type=%s",
type(error).__name__,
)
# A failed socket read is an interrupted handoff, not a fresh read path.
for data_payload in iter_sse_data(_stream_content(response)):
yield f"data: {data_payload}"
return

yield from response.iter_lines(decode_unicode=True)
1 change: 1 addition & 0 deletions tests/fixtures/secret_firewall_coverage.json
Original file line number Diff line number Diff line change
Expand Up @@ -345,6 +345,7 @@
"worker/shadow-eval-schedule.mjs",
"worker/sse-heartbeat.mjs",
"worker/status-page.mjs",
"worker/upstream-cancellation.mjs",
"worker/usage-ledger-d1.mjs"
],
"edge_route_literals": {
Expand Down
1 change: 1 addition & 0 deletions tests/helpers/load_cloudflare_worker.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -392,6 +392,7 @@ export async function roleplayModuleUrl() {
);
const transportDataUrl = dataModuleUrl(
transportSource
.replace('from "../upstream-cancellation.mjs";', `from "${new URL("../../worker/upstream-cancellation.mjs", import.meta.url)}";`)
.replace('from "../secret-firewall.mjs";', `from "${new URL("../../worker/secret-firewall.mjs", import.meta.url)}";`)
.replace('from "./compaction-budget.mjs";', `from "${compactionBudgetUrl}";`)
.replace(
Expand Down
Loading
Loading