Route across logins and keys with CPA's strategies, retries and cooldowns

free/README.md

@@ -7,10 +7,10 @@ It covers **Claude, ChatGPT (Codex) and Kimi subscriptions** and **API keys** fo
 - CPA's dashboard, downloaded at build time from the latest panel release (pin one with `PANEL_VERSION`, and its checksum with `PANEL_SHA256`).
 - Claude and ChatGPT (Codex) OAuth logins from the dashboard's **OAuth Login** page (the same PKCE flows as CPA, Claude Code and the Codex CLI), with tokens refreshed automatically.
 - Kimi logins (kimi.com or kimi.ai) through Kimi's device sign-in, as in CPA. Claude Code and both OpenAI formats reach Kimi untranslated: ask for a Kimi model such as `kimi-k3` or `kimi-k2.8`. Requests name this gateway, as CPA's name CLIProxyAPI.
-- Several logins per provider, used in turn. Upload, download, disable, edit and delete them under **Auth Files**, in CPA's auth-file format, so files move between the two installs.
+- Several logins per provider, used in turn, with a failing or rate-limited Claude or ChatGPT login skipped for the next. Upload, download, disable, edit and delete them under **Auth Files**, in CPA's auth-file format, so files move between the two installs.
 - CPA's YAML config, edited in the **Config Panel**. Client API keys live in `access.api-keys`, as in CPA.
 - `POST /v1/messages` and `POST /v1/messages/count_tokens`, streamed straight through to Anthropic.
-- API-key providers added under **AI Providers** (CPA's `api-keys` config): Claude, Gemini, Vertex AI, Codex, xAI and OpenAI-compatible endpoints such as OpenRouter. Model aliases, `prefix/model` routing, priorities, key weights and excluded models work as in CPA, and a request moves on to the next key when one is rate limited or failing.
+- API-key providers added under **AI Providers** (CPA's `api-keys` config): Claude, Gemini, Vertex AI, Codex, xAI and OpenAI-compatible endpoints such as OpenRouter. Model aliases, `prefix/model` routing, priorities, key weights (with the `weighted-round-robin` strategy) and excluded models work as in CPA, and a request moves on to the next key when one is rate limited or failing.
 - Codex CLI through a ChatGPT login, on `/backend-api/codex/responses` or `/v1/responses`. Other OpenAI Responses clients work too: their requests are adapted to what the ChatGPT endpoint takes, and a non-streaming request gets the final response as JSON.
 - Claude Code on an OpenAI-compatible key (OpenRouter and the like): ask for one of the provider's model aliases and the request goes out as Chat Completions, with the answer streamed back as Anthropic events, following CPA's Claude-to-OpenAI translator.
 - Vertex AI service accounts imported under **OAuth Login** (a Google Cloud key file and a region, as in CPA). The Durable Object signs in to Google with the key and refreshes the token hourly. Gemini clients use them as they are, and Claude Code through the Gemini translator below.
@@ -23,6 +23,7 @@ It covers **Claude, ChatGPT (Codex) and Kimi subscriptions** and **API keys** fo
 - Each client talks in its own format: Anthropic (`/v1/messages`), OpenAI Chat (`/v1/chat/completions`), OpenAI Responses (`/v1/responses`) and Gemini (`/v1beta/models/{model}:generateContent`). A request goes to a provider that takes the same format, apart from the translations above.
 - Model lists as in CPA: `GET /v1/models` (OpenAI's shape, or Anthropic's for Claude Code and the Anthropic SDKs), `GET /v1beta/models` and `GET /v1beta/models/{model}` list every model across the logins and API keys, and so does the dashboard's **Available Models**. A login or key with no model list of its own lists CPA's catalog for its provider, and a ChatGPT login the catalog for its plan. The catalog is refreshed from CPA's model catalog every three hours, with a built-in copy as fallback; it only decides what is listed, never where a request goes.
 - Usage and logs as in CPA: each login and API key shows its success and failure counts and the last 200 minutes in 10-minute buckets, and with **Logging to file** on, the **Logs** page shows a line per API request (the last 2,000). Lines hold the time, status, latency, client IP, path, provider and model, never request bodies or keys. Records are sent to the Durable Object in batches a few seconds apart.
+- CPA's routing across logins and keys, set in the `routing` config section: the `round-robin` (default), `weighted-round-robin` and `fill-first` strategies within the highest priority, failover to the next login or key, extra rounds with `retry.request-retry`, `max-retry-credentials` and `max-retry-interval`, and CPA's cooldowns: a rate-limited credential rests until the provider's stated reset (Retry-After, Anthropic's rejected-window reset, ChatGPT's usage-limit reset, Gemini's retry delay) or on CPA's 1 s to 30 min backoff, a refused or unpaid one for 30 minutes, a missing model for 12 hours, and a transient error for 60 seconds (`cooldown.transient-error-cooldown-seconds`). `cooldown.disable-cooling`, and per-key or per-login `disable-cooling`, `request-retry`, `priority`, `weight` and `prefix`, work as in CPA. **Auth Files** shows each login's cooldowns, and the reset button clears them. Each isolate shares its cooldowns through the Durable Object, so other isolates see them within a minute. A request makes at most 10 upstream attempts and waits at most 2 minutes in all, to stay inside the free plan's limits. With a single Claude login and nothing else, Claude Code requests still stream straight through. Kimi and Vertex logins still take turns in the Durable Object, without cooldowns, for now.
 - Everything is stored in one SQLite-backed Durable Object, which the free plan includes.
 
 Claude Code and Codex CLI request bodies are never parsed, so a Worker spends well under 1 ms of its 10 ms CPU budget even on a 600 KB Claude Code request (see the CPU test in `test/app.test.ts`). Requests that need a model rewritten or a body adapted are parsed once. Translated requests cost the most, about 4 to 6 ms for a 600 KB conversation; very long conversations translated can run past the free plan's 10 ms, which the Container install doesn't have.

free/src/account.ts

@@ -2,7 +2,8 @@ import { OAuthError, authorizeUrl, exchangeCode, parseCallback, randomToken, ref
 import * as codex from "./codex-oauth.ts";
 import { Config } from "./config.ts";
 import { CatalogCache } from "./models.ts";
-import { upstreamCredentials, type RoutingTable, type UpstreamCredential } from "./providers.ts";
+import { flagOf, retryOf, upstreamCredentials, weightOf, type LoginEntry, type RoutingTable, type UpstreamCredential } from "./providers.ts";
+import { routingSettings, type CooldownEvent, type RoutingSettings } from "./routing.ts";
 import { storedKey, type UsageRecord } from "./usage.ts";
 import * as kimi from "./kimi-oauth.ts";
 import { claudeCredential, claudeFileData, type CredentialFile, type LoginSession, type Vault } from "./vault.ts";
@@ -30,14 +31,16 @@ const PROVIDER_OF_FILE: Record<string, LoginProvider> = { "kimi-ai": "kimi" };
 // state below (config, token refreshes, login sessions) is never raced by another copy.
 export interface Backend {
   keyValid(hash: string): Promise<boolean>;
-  token(): Promise<TokenResult>;
-  codexToken(): Promise<TokenResult>;
+  // A login's token by its file name, or, with no name, the next login's in turn.
+  token(name?: string): Promise<TokenResult>;
+  codexToken(name?: string): Promise<TokenResult>;
   loginToken(provider: ServedLogin): Promise<TokenResult>;
   invalidate(accessToken: string): Promise<void>;
   attemptAllowed(client: string): Promise<boolean>;
   manage(request: Request): Promise<Response>;
   routing(): Promise<RoutingTable>;
   record(records: UsageRecord[]): Promise<void>;
+  cool(events: CooldownEvent[]): Promise<void>;
 }
 
 // A Kimi device login waiting for approval, kept in its session row.
@@ -61,6 +64,29 @@ interface Login {
   readonly headers?: Record<string, string>;
 }
 
+function textOf(value: unknown): string {
+  return typeof value === "string" ? value.trim() : "";
+}
+
+function numberOf(value: unknown): number {
+  const parsed = typeof value === "string" && value.trim() !== "" ? Number(value) : value;
+  return typeof parsed === "number" && Number.isFinite(parsed) ? parsed : 0;
+}
+
+// CPA reads a login's routing fields from the top level of its auth file.
+function loginEntry(file: CredentialFile): LoginEntry {
+  const data = file.data;
+  return {
+    name: file.name,
+    provider: file.provider === "codex" ? "codex" : "claude",
+    priority: numberOf(data.priority),
+    weight: weightOf(data.weight),
+    prefix: textOf(data.prefix),
+    disableCooling: flagOf(data.disable_cooling ?? data["disable-cooling"]),
+    requestRetry: retryOf(data.request_retry ?? data["request-retry"]),
+  };
+}
+
 function loginOf(file: CredentialFile): Login | null {
   if (file.provider === "codex") return codex.codexCredential(file.data);
   if (file.provider === "kimi" || file.provider === "kimi-ai") {
@@ -83,6 +109,7 @@ export async function hashKey(key: string): Promise<string> {
 export class Account {
   private config: Config | null = null;
   private upstreams: UpstreamCredential[] | null = null;
+  private settings: RoutingSettings | null = null;
   private keyHashes: Set<string> | null = null;
   private readonly refreshing = new Map<string, Promise<TokenResult>>();
   private readonly lastForcedRefresh = new Map<string, number>();
@@ -107,15 +134,46 @@ export class Account {
     this.config = config;
     this.keyHashes = null;
     this.upstreams = null;
+    this.settings = null;
+    // API-key ids follow their place in the config, so their cooldowns don't outlive an edit.
+    this.vault.clearCooldowns("keys");
   }
 
-  // What the Worker needs to pick an upstream for a request: the configured API-key providers
-  // and which logins are on.
+  // What the Worker needs to pick an upstream for a request: the configured API-key providers,
+  // the enabled logins (and the ChatGPT plans, which decide the models they list), the routing settings and the cooldowns every isolate has reported.
   async routing(): Promise<RoutingTable> {
-    this.upstreams ??= upstreamCredentials(this.currentConfig().read([]));
-    const codexFiles = this.loginFiles("codex");
-    const codexPlans = codexFiles.map((file) => (typeof file.data.plan_type === "string" ? file.data.plan_type : ""));
-    return { credentials: this.upstreams, claudeOAuth: this.loginFiles("claude").length > 0, codexOAuth: codexFiles.length > 0, codexPlans, vertexOAuth: this.loginFiles("vertex").length > 0, kimiOAuth: this.loginFiles("kimi").length > 0 };
+    const tree = this.currentConfig().read([]);
+    this.upstreams ??= upstreamCredentials(tree);
+    this.settings ??= routingSettings(tree);
+    // Claude and ChatGPT logins are routed one by one; Kimi and Vertex still take turns in the Durable Object.
+    const files = (["claude", "codex"] as const).flatMap((provider) => this.loginFiles(provider));
+    const logins = files.map(loginEntry);
+    const codexPlans = files.filter((file) => file.provider === "codex").map((file) => (typeof file.data.plan_type === "string" ? file.data.plan_type : ""));
+    return {
+      credentials: this.upstreams,
+      claudeOAuth: logins.some((login) => login.provider === "claude"),
+      codexOAuth: logins.some((login) => login.provider === "codex"),
+      codexPlans,
+      vertexOAuth: this.loginFiles("vertex").length > 0,
+      kimiOAuth: this.loginFiles("kimi").length > 0,
+      logins,
+      settings: this.settings,
+      cooldowns: this.vault.cooldowns(this.nowSeconds()),
+    };
+  }
+
+  // Cooldowns and successes reported by the Worker's isolates.
+  async cool(events: CooldownEvent[]): Promise<void> {
+    for (const event of events.slice(0, 200)) {
+      if (typeof event?.id !== "string" || event.id === "" || event.id.length > 512) continue;
+      if ("clear" in event) {
+        this.vault.clearCooldowns(event.id);
+        continue;
+      }
+      const until = Math.min(Number(event.until), this.nowSeconds() + 24 * 60 * 60);
+      if (typeof event.model !== "string" || event.model.length > 256 || !Number.isFinite(until)) continue;
+      this.vault.saveCooldown({ id: event.id, model: event.model, until, reason: String(event.reason).slice(0, 64), status: Number(event.status) || 0, level: Math.max(0, Math.min(32, Number(event.level) || 0)) });
+    }
   }
 
   async keyValid(hash: string): Promise<boolean> {
@@ -134,18 +192,24 @@ export class Account {
     return this.loginFiles("claude");
   }
 
-  token(): Promise<TokenResult> {
-    return this.loginToken("claude");
+  token(name?: string): Promise<TokenResult> {
+    return this.loginToken("claude", name);
   }
 
-  codexToken(): Promise<TokenResult> {
-    return this.loginToken("codex");
+  codexToken(name?: string): Promise<TokenResult> {
+    return this.loginToken("codex", name);
   }
 
-  // Picks the provider's next enabled login in turn (round-robin), refreshing it when near expiry.
-  async loginToken(provider: LoginProvider): Promise<TokenResult> {
+  // The named login's token, or the provider's next enabled login in turn (round-robin),
+  // refreshed when near expiry.
+  async loginToken(provider: LoginProvider, name?: string): Promise<TokenResult> {
     const files = this.loginFiles(provider);
     if (files.length === 0) return { ok: false, status: 503, message: `No ${LABEL[provider]} account is connected yet: add one under OAuth login in the dashboard.` };
+    if (name !== undefined) {
+      const file = files.find((candidate) => candidate.name === name);
+      if (file === undefined) return { ok: false, status: 503, message: `That ${LABEL[provider]} login is disabled or was removed.` };
+      return this.tokenFor(file);
+    }
     const turn = this.next.get(provider) ?? 0;
     this.next.set(provider, turn + 1);
     return this.tokenFor(files[turn % files.length] as CredentialFile);