Repository navigation
fix(transport): close upstream work once on client disconnect #31
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,67 @@ | ||
| # Disconnect-aware upstream cancellation | ||
|
|
||
| Cleanup is always enabled. There is no environment flag or disabled mode. | ||
| Successful bytes, status codes and headers are preserved. No response content, | ||
| credentials or request bodies are added to cancellation outcomes or logs. | ||
|
|
||
| For example, closing a streaming client before its first event, after a partial | ||
| event, or while waiting for the next event closes the upstream owner once. | ||
| Reading the whole response keeps its original bytes. Aborting after a completed | ||
| response does not change that completed outcome. | ||
|
|
||
| ## Python transport | ||
|
|
||
| `bind_cancellation(response, on_outcome=callback)` attaches a | ||
| `CancellationContext` to `response.multillm_cancellation` and shares its | ||
| idempotent close operation with existing owners. `iter_stream_content` and | ||
| `iter_stream_lines` return closable iterators immediately, before generator | ||
| execution. `GeneratorExit`, read exceptions, normal exhaustion and redundant | ||
| close calls release the upstream once. A socket error propagates; it does not | ||
| restart reads through a different iterator. | ||
|
|
||
| Unified dispatch binds each returned response without changing request bodies | ||
| or existing origin-fallback policy. Flask body ownership closes the underlying | ||
| iterable even when no body iteration occurred. The stream preflight (`preflight_chat_stream`) | ||
| closes that same owner on rejection; validated prefix replay keeps the same | ||
| owner. | ||
|
|
||
| WSGI detects disconnects during writes or iteration. It cannot reliably detect | ||
| a disconnect while a synchronous provider setup or socket read is blocked. | ||
| Existing bounded connect/read timeouts remain in force. No per-request watcher | ||
| thread is created, and cancellation cannot promise the provider stopped billing. | ||
|
|
||
| ## Worker transport | ||
|
|
||
| `UpstreamCancellation` owns an AbortController, a bounded header timer, the | ||
| parent abort listener and an upstream reader. Parent abort propagates to fetch | ||
| through its signal. The listener remains active after headers until the body | ||
| finishes or closes. Cancelling a downstream body also aborts fetch and cancels | ||
| the upstream reader once. Cleanup does not wait for a provider's cancellation | ||
| acknowledgement. `readBoundedUpstreamBytes` owns cancellation and reader release | ||
| for bounded nonstream reads, including abort and size rejection. | ||
|
|
||
| Roleplay uses this owner for generation and compaction. A client abort, including | ||
| one racing an HTTP rejection, stops candidate and origin fallback. Existing | ||
| non-cancellation fallback decisions remain unchanged. Terminal HTTP error bodies | ||
| retain their owner until consumption or cancellation instead of detaching at | ||
| headers. Header timers end at headers; compaction retains its total timer. | ||
|
|
||
| ## Outcomes and settlement | ||
|
|
||
| Python callbacks receive `CancellationOutcome`; Worker callbacks receive an | ||
| object with `reason`, `ambiguous`, `usageState`, `usage` and `replayPermission`. | ||
| Python also reports an [`UpstreamOutcome`](upstream-outcomes.md). After upstream handoff, | ||
| cancellation or interruption is ambiguous, usage is unknown (`None`/`null`), | ||
| and replay permission is false. Nothing here asserts zero tokens or zero cost. | ||
| Callbacks run once and contain no generated content. Cleanup errors are logged | ||
| by type only and do not mask the original result. | ||
|
|
||
| These are transient transport outcomes, not durable accounting records. | ||
|
|
||
| ## Coverage | ||
|
|
||
| Unified dispatch, the stream preflight and the roleplay transport use these | ||
| owners. Other native Worker provider fetches and the provider-specific streaming | ||
| paths in `services/proxy_service.py` keep their existing connect and read | ||
| timeouts. Whether a provider stops billing after a disconnect depends on that | ||
| provider. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,144 @@ | ||
| """One cleanup owner and content-free outcome for an upstream handoff.""" | ||
|
|
||
| from __future__ import annotations | ||
|
|
||
| import logging | ||
| from collections.abc import Callable, Iterator | ||
| from dataclasses import dataclass | ||
| from typing import Any, Generic, TypeVar | ||
|
|
||
| from flask import Response | ||
|
|
||
| from services.upstream_outcome import UpstreamOutcome, classify_upstream_outcome | ||
|
|
||
| logger = logging.getLogger(__name__) | ||
| T = TypeVar("T") | ||
|
|
||
|
|
||
| @dataclass(frozen=True) | ||
| class CancellationOutcome: | ||
| reason: str | ||
| ambiguous: bool | ||
| upstream: UpstreamOutcome | ||
| usage_state: str = "unknown" | ||
| usage: None = None | ||
|
|
||
|
|
||
| class CancellationContext: | ||
| """Own close without spawning a disconnect watcher or authorizing replay.""" | ||
|
|
||
| def __init__(self, close: Callable[[], Any], *, on_outcome=None): | ||
| self._close = close | ||
| self._on_outcome = on_outcome | ||
| self.handed_off = False | ||
| self.closed = False | ||
| self.outcome: CancellationOutcome | None = None | ||
|
|
||
| def handoff(self) -> None: | ||
| self.handed_off = True | ||
|
|
||
| def _finish(self, reason: str, upstream: UpstreamOutcome) -> None: | ||
| if self.closed: | ||
| return | ||
| self.closed = True | ||
| self.outcome = CancellationOutcome( | ||
| reason, self.handed_off and reason != "complete", upstream, | ||
| ) | ||
| try: | ||
| self._close() | ||
| except Exception as error: | ||
| logger.warning("Upstream cleanup failed type=%s", type(error).__name__) | ||
| if self._on_outcome is not None: | ||
| try: | ||
| self._on_outcome(self.outcome) | ||
| except Exception as error: | ||
| logger.warning("Cancellation observer failed type=%s", type(error).__name__) | ||
|
|
||
| def complete(self) -> None: | ||
| self._finish("complete", classify_upstream_outcome(200)) | ||
|
|
||
| def cancel(self) -> None: | ||
| self._finish("cancelled", classify_upstream_outcome(cancelled=True)) | ||
|
|
||
| def fail(self) -> None: | ||
| self._finish( | ||
| "interrupted", classify_upstream_outcome(transport_failure="interrupted"), | ||
| ) | ||
|
|
||
| def close(self) -> None: | ||
| self.cancel() | ||
|
|
||
|
|
||
| class CancellationIterator(Iterator[T], Generic[T]): | ||
| """A closable lazy iterator, including before its generator has started.""" | ||
|
|
||
| def __init__(self, factory: Callable[[], Iterator[T]], context: CancellationContext, | ||
| *, close_source: Callable[[], Any] | None = None): | ||
| self._factory = factory | ||
| self.context = context | ||
| self._source: Iterator[T] | None = None | ||
| self._close_source = close_source | ||
| self._closed = False | ||
|
|
||
| def __iter__(self) -> CancellationIterator[T]: | ||
| return self | ||
|
|
||
| def __next__(self) -> T: | ||
| if self._closed or self.context.closed: | ||
| raise StopIteration | ||
| try: | ||
| if self._source is None: | ||
| self._source = iter(self._factory()) | ||
| return next(self._source) | ||
| except StopIteration: | ||
| self.context.complete() | ||
| self.close() | ||
| raise | ||
| except GeneratorExit: | ||
| self.close() | ||
| raise | ||
| except BaseException: | ||
| self.context.fail() | ||
| self.close() | ||
| raise | ||
|
|
||
| def throw(self, error): | ||
| # GeneratorExit from a downstream owner must never resume the socket. | ||
| self.close() | ||
| raise error | ||
|
|
||
| def close(self) -> None: | ||
| if self._closed: | ||
| return | ||
| self._closed = True | ||
| close = self._close_source or getattr(self._source, "close", None) | ||
| try: | ||
| if close is not None: | ||
| close() | ||
| except Exception as error: | ||
| logger.warning("Stream cleanup failed type=%s", type(error).__name__) | ||
| finally: | ||
| self.context.close() | ||
|
|
||
|
|
||
| def bind_cancellation(response, *, on_outcome=None) -> CancellationContext: | ||
| """Share close ownership with existing transport, WSGI and preflight owners. | ||
|
|
||
| Flask bodies must themselves own hidden provider resources. A transport | ||
| creating a lazy generator should bind its upstream before building that body. | ||
| """ | ||
| context = getattr(response, "multillm_cancellation", None) | ||
| if isinstance(context, CancellationContext): | ||
| return context | ||
|
Comment on lines
+130
to
+132
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
If a response already has a cancellation context, this returns it without applying the newly supplied Useful? React with 👍 / 👎. |
||
| # Test doubles and some adapters have no close(); ownership still applies. | ||
| close = getattr(response, "close", None) | ||
| context = CancellationContext(close if callable(close) else (lambda: None), on_outcome=on_outcome) | ||
| context.handoff() | ||
| response.multillm_cancellation = context | ||
| response.close = context.close | ||
| if isinstance(response, Response): | ||
| source = response.response | ||
| response.response = CancellationIterator( | ||
| lambda: iter(source), context, close_source=getattr(source, "close", None), | ||
|
Comment on lines
+138
to
+142
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
For unified requests returning a Useful? React with 👍 / 👎. |
||
| ) | ||
| return context | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Every exhausted response is classified as HTTP 200 here, regardless of its real status. For example, fully consuming an owned 401, 429, or 503 response produces an
UpstreamOutcomereporting provider success and accepted credentials, contradicting the repository's status classification and potentially corrupting health or credential accounting once this callback is consumed. Capture the response status when binding and classify completion with that value.Useful? React with 👍 / 👎.