Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
60 changes: 60 additions & 0 deletions docs/prompt-cache-affinity.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
# Outcome-bound prompt cache affinity

Prompt cache affinity is disabled by default. Enable it for authenticated managed
automatic text routes with:

```sh
PROMPT_CACHE_AFFINITY_ENABLED=true
PROMPT_CACHE_AFFINITY_TTL_SECONDS=900
```

Disable it with `PROMPT_CACHE_AFFINITY_ENABLED=false`. An empty value uses the
default. TTL defaults to 900 seconds and accepts integers from 1 to 86400.
Malformed settings disable affinity and produce one warning naming the setting
without its value. Disabling affinity preserves request and response bytes,
headers, credential order, and retry behavior.

The managed Chat Completions route and automatic aliases translated through
Responses or Messages observe the credential that actually served an attempt.
A successful complete response binds that provider/model and credential. Stream
headers and early content do not bind: the observer waits for terminal protocol
evidence and normal exhaustion. Failed, incomplete, interrupted, and cancelled
responses evict only the matching binding. Affinity adds no retry permission.
Explicit models, raw proxy paths, and native roleplay keep their existing behavior.
This integration covers stored ordered automatic aliases. The intelligence
gateway uses a separate transport and does not consume this request scope.

Credential preference only considers keys that the pool already allows. Model
cooldown, credential rest, grants, budgets, context capability, and fallback
eligibility remain authoritative. Ordered automatic routes retain their configured
provider order. A tier-aware scheduler can call `AffinityScope.order_candidates`
with its already eligible ordered models and approved tier map. Preference can
only move a successful candidate within the first eligible tier. Missing tier
evidence retains the original order. This interface does not authorize candidates
or retries, and it never crosses a fallback tier during tool loops.

Bindings are isolated by authenticated tenant/principal, session, role lane,
automatic route, reusable opening prefix, tool schema, and policy revision.
Changing route configuration or the supplied policy revision invalidates an old
binding. Credential removal and cooldown invalidate its key preference. Explicit
session IDs do not replace prefix checks; subsequent turns and tool results keep
the opening identity. The final observer captures an opaque credential fingerprint.
No prompt, response, principal, session ID, or credential plaintext is persisted
in the affinity map or logged by this feature.

Storage is a process-local LRU capped at 10000 entries. TTL is monotonic and is
not extended by reads. Expired entries are removed on access and insertion. A
restart loses bindings; multiple Containers have independent maps. Cross-instance
affinity is best effort, and there is no D1 migration or distributed consistency
claim. The bounded stream parser ignores oversized lines and cannot bind a JSON
body larger than 1 MiB without other completion evidence. Unknown completion
evidence declines to bind; it never fabricates a success body or changes the
original error returned to the client.

The observer consumes provider cache-read usage through the existing prompt cache
usage buckets. Counts remain measured when the provider reports them. Read cost
and rebuilding the same tokens at ordinary input prices are explicitly named
estimates derived from configured catalog prices; absent counts or prices remain
unknown. These observations do not prove savings on every request, do not change
budget accounting, and never update routing policy automatically. Provider cache
retention and billing still follow that provider's contract.
13 changes: 10 additions & 3 deletions routes/unified.py
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,8 @@
from services.rate_limit_service import RateLimitService
from services.reasoning_policy import apply_glm_5_reasoning_policy
from services.transport_policy import RAW_PASSTHROUGH_PROVIDERS
from services.prompt_cache_affinity import affinity_auto_request, affinity_candidate
from services.managed_dispatch import execute_managed_attempt, observe_managed_response

logger = logging.getLogger(__name__)

Expand Down Expand Up @@ -342,7 +344,7 @@ def send_request(token: str):
request_kwargs["force_raw_passthrough"] = True
if request_timeout is not None:
request_kwargs["timeout_override"] = bounded_timeout(request_timeout)
return proxy_service_cls.make_request(**request_kwargs)
return execute_managed_attempt(lambda: proxy_service_cls.make_request(**request_kwargs), f"{provider}:{provider_model}", token)

return _request_with_provider_token_rotation(
app,
Expand Down Expand Up @@ -413,13 +415,15 @@ def _dispatch_bridged_chat(
target=CHAT,
stream=bool(payload.get("stream")),
)
observe_managed_response(downstream, response, stream=bool(payload.get("stream")))
return _add_credential_attempt_headers(
_add_prompt_cache_headers(downstream, cache_decision),
provider,
credential_attempts,
)


@affinity_candidate
@with_chat_tool_repair
def _dispatch_unified_chat_candidate(
app,
Expand Down Expand Up @@ -556,14 +560,14 @@ def send_request(token: str):
}
if request_timeout is not None:
request_kwargs["timeout_override"] = bounded_timeout(request_timeout)
return send_configured_unified_provider_request(
return execute_managed_attempt(lambda: send_configured_unified_provider_request(
proxy_service_cls,
request_kwargs,
provider=provider,
runtime_config=app.config,
upstream_path=upstream_path,
request_headers=headers_source,
)
), f"{provider}:{provider_model}", token)

response, credential_attempts = _request_with_provider_token_rotation(
app,
Expand Down Expand Up @@ -599,6 +603,7 @@ def send_request(token: str):
)

if isinstance(response, Response):
observe_managed_response(response, response, stream=bool(payload.get("stream")))
return _add_credential_attempt_headers(
_add_prompt_cache_headers(
_add_adaptive_context_headers(response, adaptive_result),
Expand All @@ -617,6 +622,7 @@ def send_request(token: str):
headers=copy_raw_provider_response_headers(response.headers),
)
mark_transport_failure(downstream_response, response)
observe_managed_response(downstream_response, response, stream=bool(payload.get("stream")))
return _add_credential_attempt_headers(
_add_prompt_cache_headers(
_add_adaptive_context_headers(
Expand All @@ -643,6 +649,7 @@ def send_request(token: str):


@sample_chat_dispatch
@affinity_auto_request
def dispatch_unified_chat_completion(
app,
auth_service_cls,
Expand Down
11 changes: 9 additions & 2 deletions services/credential_pool.py
Original file line number Diff line number Diff line change
Expand Up @@ -117,7 +117,7 @@ def select(
if config.enabled and (model or quota_bucket):
with cls._lock:
legacy = {key: cls._resting.get((provider, key), 0) for key in keys}
return model_cooldown.select(
selected = model_cooldown.select(
provider,
keys,
model=model,
Expand All @@ -126,7 +126,14 @@ def select(
now=now,
legacy_rest_until=legacy,
)
return next(iter(cls.available(provider, keys, now=now) or keys), None)
from services.prompt_cache_affinity import prefer_credential
preferred = prefer_credential(provider, model, cls.available(
provider, keys, model=model, quota_bucket=quota_bucket, now=now,
))
return preferred or selected
available = cls.available(provider, keys, now=now)
from services.prompt_cache_affinity import prefer_credential
return prefer_credential(provider, model, available) or next(iter(available or keys), None)

@classmethod
def record(
Expand Down
174 changes: 174 additions & 0 deletions services/managed_dispatch.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
"""Managed attempt scheduling, separate from raw dispatch and normalization."""

import math
import json
import time
from typing import Callable

Expand All @@ -10,11 +11,184 @@
from services.retry_advice import parse_retry_advice, retry_advice_settings
from services.upstream_outcome import UpstreamOutcome
from services.upstream_transport import close_retry_response
from services.upstream_outcome import classify_upstream_outcome

Attempt = Callable[[int], requests.Response]
Outcome = Callable[[], UpstreamOutcome]


def execute_managed_attempt(send, model, credential):
"""Capture the actual dispatch credential; headers alone never bind affinity."""
from services.prompt_cache_affinity import current_scope
scope = current_scope()
if scope is None:
return send()
try:
response = send()
except BaseException:
scope.observer(model, credential)(classify_upstream_outcome(transport_failure="interrupted"))
raise
return capture_managed_attempt(response, model, credential)


def capture_managed_attempt(response, model, credential):
from services.prompt_cache_affinity import current_scope
scope = current_scope()
if scope is not None:
observer = scope.observer(model, credential)
if not 200 <= response.status_code < 300:
observer(classify_upstream_outcome(response.status_code))
else:
response.multillm_affinity_observer = observer
return response


def _completed_body(body):
if not isinstance(body, dict) or body.get("error"):
return False
if body.get("type") in {"error", "response.failed", "response.incomplete"}:
return False
if body.get("status") in {"failed", "incomplete", "cancelled", "in_progress"}:
return False
choices = body.get("choices")
if isinstance(choices, list) and choices:
return all(isinstance(choice, dict) and choice.get("finish_reason") is not None for choice in choices)
return body.get("status") == "completed" or (
body.get("type") == "message" and body.get("stop_reason") is not None)


class ManagedOutcomeIterator:
"""Observe final bytes without changing them or granting another dispatch."""

def __init__(self, source, callback, *, event_stream, context=None):
from services.prompt_cache_cost import CacheStreamObserver
self.source = iter(source)
self.callback = callback
self.event_stream = event_stream
self.context = context
self.usage = CacheStreamObserver(65536)
self.buffer = b""
self.oversized = False
self.complete = False
self.failed = False
self.closed = False

def __iter__(self):
return self

def _event(self, line):
if line.startswith(b"event:"):
self.failed |= line[6:].strip() in {b"error", b"response.failed", b"response.incomplete"}
if not line.startswith(b"data:"):
return
data = line[5:].strip()
if data == b"[DONE]":
self.complete = True
return
try:
body = json.loads(data)
except (ValueError, RecursionError):
return
if not isinstance(body, dict):
return
self.failed |= bool(body.get("error")) or body.get("type") in {
"error", "response.failed", "response.incomplete"}
self.complete |= _completed_body(body) or body.get("type") in {"message_stop", "response.completed"}

def _feed(self, chunk):
self.usage.feed(chunk)
if not self.event_stream:
if len(self.buffer) + len(chunk) <= 1024 * 1024 and not self.oversized:
self.buffer += chunk
else:
self.buffer, self.oversized = b"", True
return
for index, part in enumerate(chunk.split(b"\n")):
if index:
if not self.oversized:
self._event(self.buffer.strip())
self.buffer, self.oversized = b"", False
if len(self.buffer) + len(part) <= 65536 and not self.oversized:
self.buffer += part
else:
self.buffer, self.oversized = b"", True

def _finish(self, outcome):
if self.closed:
return
self.closed = True
owned = getattr(self.context, "outcome", None)
if owned is not None and owned.reason != "complete":
outcome = owned.upstream
self.callback(outcome, self.usage.finish())

def _exhausted(self):
if self.event_stream and not self.oversized:
self._event(self.buffer.strip())
if not self.event_stream and not self.oversized:
try:
body = json.loads(self.buffer)
self.complete = _completed_body(body)
from services.prompt_cache_cost import CacheObservation
self.usage.observation = CacheObservation.from_body(body)
except (ValueError, RecursionError):
self.complete = False
outcome = classify_upstream_outcome(200) if self.complete and not self.failed else classify_upstream_outcome(
transport_failure="interrupted")
self._finish(outcome)

def __next__(self):
if self.closed:
raise StopIteration
try:
chunk = next(self.source)
except StopIteration:
self._exhausted()
raise
except BaseException:
self._finish(classify_upstream_outcome(transport_failure="interrupted"))
raise
self._feed(chunk.encode("utf-8") if isinstance(chunk, str) else chunk)
return chunk

def close(self):
# The existing Flask owner closes the body while reporting upstream EOF.
owned = getattr(self.context, "outcome", None)
if owned is not None and owned.reason == "complete":
self._exhausted()
else:
self._finish(classify_upstream_outcome(cancelled=True))
close = getattr(self.source, "close", None)
if close is not None:
close()


def observe_managed_response(downstream, upstream, *, stream=False):
"""Deliver classified final outcomes through the managed-dispatch callback."""
callback = getattr(upstream, "multillm_affinity_observer", None)
if callback is None:
return downstream
if not 200 <= downstream.status_code < 300:
callback(classify_upstream_outcome(downstream.status_code))
return downstream
event_stream = stream or downstream.mimetype == "text/event-stream"
if event_stream or downstream.is_streamed:
observer = ManagedOutcomeIterator(downstream.response, callback, event_stream=event_stream,
context=getattr(upstream, "multillm_cancellation", None))
downstream.response = observer
downstream.call_on_close(observer.close)
else:
from services.prompt_cache_cost import CacheObservation
try:
body = json.loads(downstream.get_data())
except (ValueError, RecursionError):
body = None
classified = classify_upstream_outcome(200) if _completed_body(body) else classify_upstream_outcome(
transport_failure="interrupted")
callback(classified, CacheObservation.from_body(body))
return downstream


def retry_managed_attempt(
attempt: Attempt,
outcome: Outcome,
Expand Down
19 changes: 19 additions & 0 deletions services/opencode_session.py
Original file line number Diff line number Diff line change
Expand Up @@ -83,3 +83,22 @@ def with_opencode_request_session(
session = f"multillm_request_{uuid.uuid4()}"
result["X-Opencode-Session"] = session
return result


def reusable_prefix_digest(payload) -> str | None:
"""Hash the opening and tool schema separately from a caller's session ID."""
if not isinstance(payload, dict):
return None
opening_payload = {key: payload[key] for key in ("messages", "input", "system", "instructions") if key in payload}
anchor = _conversation_anchor(opening_payload)
if anchor is None:
return None
try:
encoded = json.dumps([anchor, payload.get("tools"), payload.get("tool_choice")],
sort_keys=True, ensure_ascii=False, separators=(",", ":"),
allow_nan=False).encode("utf-8")
except (ValueError, TypeError, UnicodeError, RecursionError):
return None
if len(encoded) > MAX_SESSION_BODY_BYTES:
return None
return hashlib.sha256(encoded).hexdigest()
Loading
Loading