Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions cloudflare-worker.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ import { handleAiOutbound } from "./worker/ai-outbound.mjs";
import { handleMediaOutbound } from "./worker/media-outbound.mjs";
import { serveSignedMediaFile, signedMediaFileId } from "./worker/media-files.mjs";
import { MediaJobWorkflow } from "./worker/media-workflow.mjs";
import { AdmissionCoordinator } from "./worker/admission-do.mjs";
import { handleKnowledgeEdgeRequest, isKnowledgeEdgePath } from "./worker/knowledge-edge.mjs";
import { withAccessIdentity } from "./worker/access-sso.mjs";
import { runScheduledShadowEval } from "./worker/shadow-eval-schedule.mjs";
Expand Down Expand Up @@ -41,6 +42,7 @@ import { withJanitorGlmReasoningNormalization } from "./worker/janitor-reasoning


export { RoleplaySession };
export { AdmissionCoordinator };
export { MediaJobWorkflow };

import { CORS_ALLOWED_METHODS, CORS_DEFAULT_HEADERS, CORS_EXPOSE_HEADERS } from "./worker/cors-policy.mjs";
Expand Down
118 changes: 118 additions & 0 deletions docs/admission-leases.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,118 @@
# Shared keyed concurrency admission

Admission is off by default. `ADMISSION_ENABLED=false` (or empty) leaves dispatch,
response bytes, headers and storage unchanged. No lease call or Durable Object access
occurs. Invalid flags or limits disable admission and produce one fixed warning per
process/isolate without the invalid value. `ADMISSION_LIMITS_JSON={}` (or empty) is
unlimited even when the flag is true. There is no default heavy-model gate.

## Limits and identity

Enable with `ADMISSION_ENABLED=true` and, for example:

```json
{"principal": 4, "model_groups": {"heavy": 1, "fast": 3}}
```

`principal` limits all active requests for one authenticated key identity across its
model groups. Each `model_groups` entry limits that same principal within the named
**authorized** group. Missing or zero limits are unlimited. Limits are integers from
0 through 10,000; at most 256 group entries are accepted. Unknown fields disable the
configuration. These are concurrency limits, not RPM, token or upstream quota limits.
Both applicable limits are checked atomically before dispatch. Lowering a limit does
not revoke existing work; new acquisitions wait for capacity.

The authentication/routing registrar supplies an `AdmissionIdentity` (Python) or an
explicit object (Worker) with `principal_hash`, `model_group`, `request_id`, and
`deadline_ms`. The principal is a stable opaque SHA-256 identity for the authenticated
key, shared between runtimes. Do not supply raw keys, usernames, caller headers or
unverified body fields. Resolve/authorize the group before admission. The request ID
must identify one generation, including both runtimes when forwarding. The deadline
is a Unix timestamp in milliseconds, after now and at most 24 hours away; the registrar
must supply the actual eventual generation deadline, not a new deadline on renewal.

## Authority and private operations

`ADMISSION_COORDINATOR` uses `AdmissionCoordinator` with SQLite Durable Object storage.
The binding and `v3_admission` migration are additive; existing bindings/migrations
stay unchanged. The class is inert while disabled: construction, fetch and alarm make
no storage writes and schedule no alarms. Limits for a principal share one of 64
stable shards, selected from its hash, so native Worker and Flask callers compete for
the same capacity. Each shard holds at most 10,000 outstanding leases, independent of
configured per-principal limits. This safety bound also returns 429.

The fixed Container egress endpoint is `POST http://intelligence.internal/v1/admission`.
Container egress isolation is its authentication boundary, as with the other private
intelligence operations. It must remain absent from the public Worker router. Native
Worker hooks call the bound authority directly, after authentication. Durable Object
fetch accepts only `POST http://admission.internal/v1/admission`; it is not a public
HTTP authority. A public origin, query, fragment, alternate method, unknown field or
client-supplied limit/key is rejected. The strict version-1 body contains the identity
fields and `operation` (`acquire`, `renew`, `release`); renew/release also require
`lease_id`. Lease IDs cannot release or renew another principal, group, request or
deadline. Repeated acquisition of a live request returns its existing lease without
extending it; changing the group/deadline is rejected. Release is idempotent.

Standalone Flask has no global local semaphore. It needs an explicit private
`ADMISSION_AUTHORITY_URL`, for example `http://127.0.0.1:8081/v1/admission`, pointing to
a trusted private adapter implementing these same shared authority operations. That
adapter is an integration requirement, not provided as a public Flask route here.
Only private IPs, localhost or `.internal` hostnames with `/v1/admission` are accepted;
credentials, redirects, query strings and fragments are forbidden. The adapter must
be isolated/authenticated at the network or service boundary. Missing authority while
limited and enabled fails with 503. Container D1 mode uses the fixed private egress
URL by default. Explicit URLs take precedence. Requests bypass environment proxies,
use one POST with no retries, 2-second connect/3-second inactivity timeouts, and bound
response decoding to 4 KiB and a 5-second decoding budget. The Requests timeout is
not a hard overall wall-clock deadline during a stalled setup. Worker authority calls
have a 5-second wait bound; a lost reply is never permission to replay generation.

## Lease lifecycle

A lease lives for 30 seconds, capped by the generation deadline. Renew every 10 seconds.
Python has one lazy daemon renewal scheduler per process, shared by clients, with
bounded local lease bookkeeping. It performs private operations sequentially, so very
large active sets or a slow authority can cause leases to expire; it never admits on
that basis. Worker leases own a renewal timer and an expiry timer. Renewal cannot
extend the generation deadline or revive an expired lease. Lost renewal stops further
output and calls the injected cancellation collaborator. Wire `on_lost` (Python) or
`onLost` (Worker) to the actual upstream cancellation owner. A blocking dispatch cannot
be forcibly interrupted by a lease helper without that collaborator.

`dispatch_with_admission(identity, dispatch, client, on_lost=...)` wraps a normalized
Flask Response; `runWithAdmission(identity, env, dispatch, options)` wraps native Worker
dispatch. Worker options accept the request `signal`, `onLost`, and an injectable
millisecond clock. Use one lifecycle owner per generation. Forwarded work must be
admitted by Flask only; the Worker must not acquire an additional lease for forwarding.
A pre-dispatch/preflight rejection, response construction failure, normal completion,
stream error, disconnect or deadline releases once. Streaming wrappers pass chunks
unchanged and do not buffer the full response. Closing a Flask stream before its first
iteration still closes its source and releases its lease. WSGI detects disconnects
only on writes/iteration/close; cancellation during blocking setup needs the transport
collaborator. Release failure is not replayed and does not rewrite provider output;
the authority reclaims the slot at lease expiry. A process crash has the same bound.

Expired leases stop counting immediately and are removed on the next authority
operation using an expiry index. No cleanup alarms are scheduled. An idle shard may
retain expired metadata until its next operation. Only opaque principal identity,
authorized group, request ID, lease ID, deadline and expiry are stored; no request,
prompt, response, tool content or credential is stored or logged. SQLite records and
private calls incur normal platform storage/request costs. Admission does not claim
zero generation cost, successful output or replay permission after a partial result.

## Errors and integration

Capacity denial is a real 429 `admission_denied` with integer `Retry-After` seconds
until the earliest blocking lease expiry (minimum 1). Unavailable authority/storage or
unconfirmed acquisition while enabled is 503 `admission_unavailable`. Lease identity
mismatch is 403; expired renewal is 409 on the private API, translated to 503 by the
generation lifecycle. No success body is synthesized. Error responses are no-store.
Raw provider headers are preserved; only gateway denial adds Retry-After.

Native Worker generations acquire their lease at the edge. Requests forwarded to the
Container acquire only in Flask, after authentication and before dispatch, so one
request never holds two leases. No public route is added; `/v1/admission` exists only
on the private intelligence transport. Set `ADMISSION_ENABLED`, `ADMISSION_LIMITS_JSON`
and, for a standalone Flask deployment, `ADMISSION_AUTHORITY_URL` on both the Worker and
the Container. No D1 tables are added. `wrangler deploy` applies the `v3_admission`
Durable Object migration; the class stays inert until the flag is enabled.
84 changes: 84 additions & 0 deletions middleware/admission.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
"""Explicit post-authentication admission hook for the Flask registrar."""
import json

from flask import Response

from services.admission_leases import AdmissionError


def admission_error_response(error):
headers = {"Cache-Control": "no-store"}
if error.status == 429 and error.retry_after is not None:
headers["Retry-After"] = str(error.retry_after)
return Response(json.dumps({"error": {"code": error.code, "message": "Concurrency admission failed."}}),
status=error.status, headers=headers, content_type="application/json")


class _LeasedIterable:
def __init__(self, source, lease):
self.source, self.lease = source, lease
self.iterator = None
self.closed = False

def __iter__(self):
return self

def __next__(self):
if self.closed:
raise StopIteration
try:
self.lease.check()
if self.iterator is None:
self.iterator = iter(self.source)
chunk = next(self.iterator)
self.lease.check()
return chunk
except BaseException:
self.close()
raise

def close(self):
if self.closed:
return
self.closed = True
try:
close = getattr(self.source, "close", None)
if close:
close()
finally:
self.lease.release()


def dispatch_with_admission(identity, dispatch, client, *, on_lost=None):
"""Identity is supplied after auth/routing; dispatch returns a normalized Response.

The registrar calls this before dispatch/preflight and supplies transport cancellation
through on_lost. Flask/WSGI detects disconnects on iteration or response close.
"""
try:
lease = client.acquire(identity, on_lost=on_lost)
except AdmissionError as error:
return admission_error_response(error)
if lease is None:
return dispatch()
response = None
try:
lease.check()
response = dispatch()
lease.check()
if not isinstance(response, Response):
raise TypeError("Admission dispatch must return a normalized Response")
if response.is_streamed:
response.response = _LeasedIterable(response.response, lease)
else:
lease.release()
return response
except BaseException as error:
try:
if response is not None and hasattr(response, "close"):
response.close()
finally:
lease.release()
if isinstance(error, AdmissionError):
return admission_error_response(error)
raise
2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
"lint": "ruff check . --select E9,F63,F7,F82",
"typecheck": "mypy --ignore-missing-imports --follow-imports=skip --disable-error-code import-untyped --disable-error-code var-annotated --disable-error-code arg-type --disable-error-code assignment --disable-error-code operator app.py routes services providers streaming",
"test": "python -m pytest -q",
"test:worker": "node --test tests/test_cloudflare_worker.mjs tests/test_cascades_worker.mjs tests/test_cascades_dashboard.mjs tests/test_image_quality_worker.mjs tests/test_secret_scan_worker.mjs tests/test_secret_firewall_worker.mjs tests/test_secret_firewall_coverage_worker.mjs tests/test_roleplay_compaction_budget_worker.mjs tests/test_roleplay_compaction_model_worker.mjs tests/test_roleplay_stream_validation_worker.mjs tests/test_aihubmix_worker.mjs tests/test_gpt_image_moderation_worker.mjs tests/test_dashboard_scripts.mjs tests/test_operations_dashboard.mjs tests/test_opencode_reasoning_worker.mjs tests/test_opencode_thinking_worker.mjs tests/test_nanogpt_reasoning_worker.mjs tests/test_nanogpt_speed_routing_worker.mjs tests/test_sse_heartbeat_worker.mjs tests/test_roleplay_worker.mjs tests/test_roleplay_config_worker.mjs tests/test_roleplay_glm_failover_worker.mjs tests/test_roleplay_intelligence_worker.mjs tests/test_roleplay_memory_worker.mjs tests/test_roleplay_compatibility_worker.mjs tests/test_roleplay_history_reconciliation_worker.mjs tests/test_roleplay_output_contract_analysis.mjs tests/test_roleplay_output_contract_cleanup_worker.mjs tests/test_roleplay_output_contract_nonstream_worker.mjs tests/test_roleplay_output_contract_worker.mjs tests/test_roleplay_performance_worker.mjs tests/test_roleplay_quality_routing_worker.mjs tests/test_roleplay_reasoning_worker.mjs tests/test_roleplay_session_storage_worker.mjs tests/test_roleplay_transport_failure_worker.mjs tests/test_roleplay_unlimited_worker.mjs tests/test_roleplay_workbench_worker.mjs tests/test_roleplay_candidate_context_worker.mjs tests/test_roleplay_operator_worker.mjs tests/test_workbench_stream.mjs tests/test_intelligence_worker.mjs tests/test_intelligence_d1_worker.mjs tests/test_intelligence_auth_worker.mjs tests/test_intelligence_outbound_worker.mjs tests/test_intelligence_operator.mjs tests/test_control_users_worker.mjs tests/test_control_audit_worker.mjs tests/test_access_sso_worker.mjs tests/test_d1_schema_worker.mjs tests/test_shadow_eval_worker.mjs tests/test_auto_routes_worker.mjs tests/test_ai_outbound_worker.mjs tests/test_knowledge_*.mjs tests/test_control_state_worker.mjs tests/test_route_health_worker.mjs tests/test_gateway_mcp_worker.mjs tests/test_usage_ledger_worker.mjs tests/test_media_outbound_worker.mjs tests/test_media_files_worker.mjs tests/test_media_jobs_worker.mjs tests/test_tool_repair_worker.mjs tests/test_prometheus_worker.mjs tests/test_evaluation_noise_floor_worker.mjs tests/test_prompt_templates_worker.mjs tests/test_config_snapshots_worker.mjs",
"test:worker": "node --test tests/test_cloudflare_worker.mjs tests/test_cascades_worker.mjs tests/test_cascades_dashboard.mjs tests/test_image_quality_worker.mjs tests/test_secret_scan_worker.mjs tests/test_secret_firewall_worker.mjs tests/test_secret_firewall_coverage_worker.mjs tests/test_roleplay_compaction_budget_worker.mjs tests/test_roleplay_compaction_model_worker.mjs tests/test_roleplay_stream_validation_worker.mjs tests/test_aihubmix_worker.mjs tests/test_gpt_image_moderation_worker.mjs tests/test_dashboard_scripts.mjs tests/test_operations_dashboard.mjs tests/test_opencode_reasoning_worker.mjs tests/test_opencode_thinking_worker.mjs tests/test_nanogpt_reasoning_worker.mjs tests/test_nanogpt_speed_routing_worker.mjs tests/test_sse_heartbeat_worker.mjs tests/test_roleplay_worker.mjs tests/test_roleplay_config_worker.mjs tests/test_roleplay_glm_failover_worker.mjs tests/test_roleplay_intelligence_worker.mjs tests/test_roleplay_memory_worker.mjs tests/test_roleplay_compatibility_worker.mjs tests/test_roleplay_history_reconciliation_worker.mjs tests/test_roleplay_output_contract_analysis.mjs tests/test_roleplay_output_contract_cleanup_worker.mjs tests/test_roleplay_output_contract_nonstream_worker.mjs tests/test_roleplay_output_contract_worker.mjs tests/test_roleplay_performance_worker.mjs tests/test_roleplay_quality_routing_worker.mjs tests/test_roleplay_reasoning_worker.mjs tests/test_roleplay_session_storage_worker.mjs tests/test_roleplay_transport_failure_worker.mjs tests/test_roleplay_unlimited_worker.mjs tests/test_roleplay_workbench_worker.mjs tests/test_roleplay_candidate_context_worker.mjs tests/test_roleplay_operator_worker.mjs tests/test_workbench_stream.mjs tests/test_intelligence_worker.mjs tests/test_intelligence_d1_worker.mjs tests/test_intelligence_auth_worker.mjs tests/test_intelligence_outbound_worker.mjs tests/test_intelligence_operator.mjs tests/test_control_users_worker.mjs tests/test_control_audit_worker.mjs tests/test_access_sso_worker.mjs tests/test_d1_schema_worker.mjs tests/test_shadow_eval_worker.mjs tests/test_auto_routes_worker.mjs tests/test_ai_outbound_worker.mjs tests/test_knowledge_*.mjs tests/test_control_state_worker.mjs tests/test_route_health_worker.mjs tests/test_gateway_mcp_worker.mjs tests/test_usage_ledger_worker.mjs tests/test_media_outbound_worker.mjs tests/test_media_files_worker.mjs tests/test_media_jobs_worker.mjs tests/test_tool_repair_worker.mjs tests/test_prometheus_worker.mjs tests/test_evaluation_noise_floor_worker.mjs tests/test_prompt_templates_worker.mjs tests/test_config_snapshots_worker.mjs tests/test_admission_leases_worker.mjs",
"db:migrate": "python scripts/validate_sqlite_schema.py --apply",
"db:validate": "python scripts/validate_sqlite_schema.py",
"security:secrets": "python scripts/check_static_secrets.py",
Expand Down
Loading
Loading