Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
67 changes: 67 additions & 0 deletions docs/retry-advice.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
# Bounded upstream Retry-After advice

`UPSTREAM_RETRY_AFTER_ADVICE_ENABLED` defaults to false. Unset or empty settings
use their defaults. Disabled traffic keeps its existing retries, waits, headers,
response bodies and free-pool cooldown timing. Native Worker and raw Flask paths
continue to forward upstream bytes without parsing or scheduling retries.

Enable local advice with:

```text
UPSTREAM_RETRY_AFTER_ADVICE_ENABLED=true
UPSTREAM_RETRY_AFTER_MAX_SECONDS=3600
```

The maximum defaults to 3600 seconds and accepts whole seconds from 1 to 604800,
matching the existing free-pool cooldown ceiling. An invalid flag or enabled
maximum disables advice and logs the setting name once per process, without its
value. The maximum is ignored while advice is disabled.

`services.retry_advice.parse_retry_advice` returns immutable local advice with
`source`, `observed_at`, `retry_at`, `delay_seconds` and `scope`. Times are Unix
seconds. It recognizes these formats, using case-insensitive header names:

- `Retry-After`: nonnegative decimal seconds or an HTTP-date with a timezone.
- `X-RateLimit-Reset`: an absolute Unix timestamp in seconds.
- `X-RateLimit-Reset-Requests` and `X-RateLimit-Reset-Tokens`: duration strings,
such as `1h2m3.4s`, when the matching remaining counter is finite and exhausted.
- `Anthropic-RateLimit-Requests-Reset` and `Anthropic-RateLimit-Tokens-Reset`:
ISO timestamps with a timezone, when the matching remaining counter is exhausted.

Malformed, nonfinite, zero-delay and stale hints are ignored. Individual values
are limited to 512 characters. When several active hints are valid, the latest
reset wins so an exhausted dimension is not bypassed. Only local advice is
clamped; the upstream header is never replaced. With observation time 100,
`Retry-After: 20` yields retry time 120. With maximum 5 it yields local retry time
105, while the header remains `20`.

The [upstream outcome](upstream-outcomes.md) classifier keeps 429 throttling neutral for provider health.
Exhausted remaining counters label advice `quota_exhausted`; other 429 advice
uses `rate_limit`; provider failures use `provider`. Authentication and caller
errors do not supply retry advice. Successful exhausted responses can supply
cooldown advice without changing their health classification.

Managed retries still require the existing method/idempotency/transport policy
and attempt limit. Parsing never grants replay permission. An ambiguous POST or
stream does not gain a retry from headers. While enabled, legacy timeout-body
retries additionally require a safe method or an idempotency key. Connect-timeout
retries remain allowed by the existing policy because nothing reached upstream.
Existing managed response normalization remains in the proxy service.

For an allowed retry, the scheduler waits for the greater of existing backoff
and bounded advice. Invalid or absent advice uses existing backoff. An optional
monotonic `deadline` on `_make_base_request`, or the existing Flask
`g.cascade_deadline`, must cover the entire wait. A refused wait returns the
original upstream result. The scheduler rechecks the deadline after sleeping and
does not dispatch a late retry. Without a shared deadline, existing per-attempt
timeouts remain unchanged; the feature does not create a request-wide deadline
or reserve time for another attempt. Free-pool `retry_seconds` uses the same
bounded parser while enabled, including a bounded fallback cooldown.

There are no new routes, response headers, storage tables or dependencies. Advice
reads headers only and retains no credentials, prompts or response content.
Existing cooldown storage and its monotonic extension rules remain in use;
enabling a lower maximum does not shorten a previously stored cooldown. Advice
is neither proof of provider capacity nor authorization for additional cost.
Retries can consume additional quota only when already authorized by transport
policy. The Worker forwards both settings to the Container.
6 changes: 6 additions & 0 deletions services/free_quota_service.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@
from email.utils import parsedate_to_datetime

from services import free_quota_d1
from services.retry_advice import parse_retry_advice, retry_advice_settings
from services.sqlite_store import connect, storage_path

MAX_COOLDOWN = 7 * 24 * 3600
Expand All @@ -30,6 +31,11 @@ def reset_seconds(value: str) -> float | None:


def retry_seconds(headers, now: float, default: int = 60) -> int:
settings = retry_advice_settings()
if settings.enabled:
advice = parse_retry_advice(headers, now=now, max_seconds=settings.max_seconds)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Pass the actual upstream status into advice parsing

When advice is enabled, this call omits status_code, so parse_retry_advice classifies every free-route outcome as its default 429. The callers in routes/free_routes.py also use retry_seconds for 401/403/404 and other failures; consequently, a Retry-After on an authentication or caller error is accepted and can block a provider or model for up to the configured maximum, even though the parser explicitly rejects advice for those statuses. Add a status parameter and pass the available upstream status so these responses use the fallback cooldown instead.

Useful? React with 👍 / 👎.

delay = advice.delay_seconds if advice is not None else default
return min(settings.max_seconds, max(1, math.ceil(delay)))
normalized = {k.lower(): v for k, v in headers.items()}
value = normalized.get("retry-after", "")
delay = None
Expand Down
65 changes: 65 additions & 0 deletions services/managed_dispatch.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
"""Managed attempt scheduling, separate from raw dispatch and normalization."""

import math
import time
from typing import Callable

import requests
from flask import g, has_request_context

from services.retry_advice import parse_retry_advice, retry_advice_settings
from services.upstream_outcome import UpstreamOutcome
from services.upstream_transport import close_retry_response

Attempt = Callable[[int], requests.Response]
Outcome = Callable[[], UpstreamOutcome]


def retry_managed_attempt(
attempt: Attempt,
outcome: Outcome,
*,
retry_count: int,
max_retries: int,
retry_delay: float,
response: requests.Response | None = None,
deadline: float | None = None,
sleep: Callable[[float], None] = time.sleep,
) -> requests.Response | None:
"""Dispatch again only under the existing transport's replay permission.

A supplied deadline is monotonic. The existing cascade deadline is used when
available; absent one, per-attempt timeouts remain the transport's policy.
None means that the original response/error must finish normally.
"""
classified = outcome()
if not classified.replay_permission or retry_count >= max_retries:
return None
settings = retry_advice_settings()
delay = retry_delay * (retry_count + 1)
if settings.enabled:
if response is not None:
advice = parse_retry_advice(
response.headers,
now=time.time(),
status_code=response.status_code,
max_seconds=settings.max_seconds,
)
if advice is not None:
delay = max(delay, advice.delay_seconds)
if deadline is None and has_request_context():
deadline = getattr(g, "cascade_deadline", None)
if deadline is not None:
if not math.isfinite(deadline) or delay >= deadline - time.monotonic():
return None
# Keep the original response usable if scheduling consumes the budget.
sleep(delay)
if time.monotonic() >= deadline:
return None
if response is not None:
close_retry_response(response)
return attempt(retry_count + 1)
if response is not None:
close_retry_response(response)
sleep(delay)
return attempt(retry_count + 1)
87 changes: 46 additions & 41 deletions services/proxy_service.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
import json
import logging
import requests
from functools import partial
from typing import Optional, Dict, Any, Tuple, List, Generator
from concurrent.futures import ThreadPoolExecutor
from error_handlers import APIError
Expand All @@ -14,6 +15,8 @@
from services.credential_pool import CredentialPool
from services.resilience_service import ResilienceService
from services.upstream_outcome import UpstreamOutcome, classify_upstream_outcome
from services.managed_dispatch import retry_managed_attempt
from services.retry_advice import retry_advice_settings
from services.transport_policy import RAW_PASSTHROUGH_PROVIDERS
from services.secret_firewall import protect_body
from providers.cline_pass import cline_completion_payload
Expand Down Expand Up @@ -1250,6 +1253,7 @@ def _make_base_request(
timeout_override: Optional[Tuple[int, int]] = None,
force_raw_passthrough: bool = False,
is_streaming: Optional[bool] = None,
deadline: Optional[float] = None,
) -> requests.Response:
"""
Make a base request with retries and error handling
Expand All @@ -1262,6 +1266,16 @@ def _make_base_request(
force_raw_passthrough
or api_provider in RAW_PASSTHROUGH_PROVIDERS
)
next_request = partial(
cls._make_base_request, method=method, url=url, headers=headers,
params=params, data=data, api_provider=api_provider, use_cache=use_cache,
timeout_override=timeout_override, force_raw_passthrough=force_raw_passthrough,
deadline=deadline,
)

def next_attempt(count: int) -> requests.Response:
return next_request(retry_count=count, is_streaming=is_streaming)

try:
if not raw_passthrough and retry_count == 0:
circuit_response = cls._circuit_open_response(api_provider)
Expand Down Expand Up @@ -1325,21 +1339,14 @@ def _make_base_request(
retry_count + 1,
MAX_RETRIES,
)
close_retry_response(response)
time.sleep(RETRY_DELAY * (retry_count + 1))
return cls._make_base_request(
method=method,
url=url,
headers=headers,
params=params,
data=data,
api_provider=api_provider,
use_cache=use_cache,
retry_count=retry_count + 1,
timeout_override=timeout_override,
force_raw_passthrough=force_raw_passthrough,
is_streaming=is_streaming,
retried = retry_managed_attempt(
next_attempt,
lambda: classify_upstream_outcome(response.status_code, replay_permission=True),
retry_count=retry_count, max_retries=MAX_RETRIES,
retry_delay=RETRY_DELAY, response=response, deadline=deadline, sleep=time.sleep,
)
if retried is not None:
return retried

if not is_streaming:
try:
Expand Down Expand Up @@ -1384,21 +1391,23 @@ def _make_base_request(
retry_count + 1,
MAX_RETRIES,
)
close_retry_response(response)
time.sleep(RETRY_DELAY * (retry_count + 1))
return cls._make_base_request(
method=method,
url=url,
headers=headers,
params=params,
data=data,
api_provider=api_provider,
use_cache=use_cache,
retry_count=retry_count + 1,
timeout_override=timeout_override,
force_raw_passthrough=force_raw_passthrough,
is_streaming=is_streaming,
replay_allowed = (
not retry_advice_settings().enabled
or (not is_streaming and (
method.upper() in cls.SAFE_RETRY_METHODS
or cls._has_idempotency_key(headers)
))
)
retried = retry_managed_attempt(
next_attempt,
lambda: classify_upstream_outcome(
response.status_code, replay_permission=replay_allowed,
),
retry_count=retry_count, max_retries=MAX_RETRIES,
retry_delay=RETRY_DELAY, response=response, deadline=deadline, sleep=time.sleep,
)
if retried is not None:
return retried

if api_provider == "cline-pass" and response.status_code < 400:
normalized_payload = cline_completion_payload(normalized_payload)
Expand Down Expand Up @@ -1461,20 +1470,16 @@ def _make_base_request(
and retry_count < MAX_RETRIES
):
logger.info(f"Retrying request (attempt {retry_count + 1}/{MAX_RETRIES})")
time.sleep(RETRY_DELAY * (retry_count + 1))
return cls._make_base_request(
method=method,
url=url,
headers=headers,
params=params,
data=data,
api_provider=api_provider,
use_cache=use_cache,
retry_count=retry_count + 1,
timeout_override=timeout_override,
force_raw_passthrough=force_raw_passthrough,
is_streaming=is_streaming,
retried = retry_managed_attempt(
next_attempt,
lambda error=e: classify_upstream_outcome(
transport_failure=cls._transport_failure_kind(error), replay_permission=True,
),
retry_count=retry_count, max_retries=MAX_RETRIES,
retry_delay=RETRY_DELAY, deadline=deadline, sleep=time.sleep,
)
if retried is not None:
return retried
if not raw_passthrough:
cls._record_circuit_result(
api_provider, 503,
Expand Down
Loading
Loading