fix(gateway): degrade when a provider is down

docs/ARCHITECTURE.md

@@ -250,6 +250,9 @@ prerendered pages, the fonts, the images, `robots.txt` and the files for languag
 other request goes to `src/worker/entry.ts`, the entry of the Worker. It exports the Durable Object
 class and looks at the path:
 
+- `/health` is answered by `src/worker/api.ts` before anything else. It reads the version
+  binding and returns. It does not touch KV, a hub or a provider, so a third-party outage cannot
+  fail it.
 - A path that starts with `/api/` is answered by `src/worker/api.ts` and never reaches Nitro. The
   prefix is exact and case-sensitive.
 - Everything else is passed to the Worker that Nitro builds, unchanged. There, the middleware
@@ -264,6 +267,7 @@ the route rules, and the response of a WebSocket upgrade does not survive that.
 
 | Endpoint                         | Method    | Answer                                               |
 | -------------------------------- | --------- | ---------------------------------------------------- |
+| `/health`                        | GET, HEAD | `{ ok: true, version }` — the running deployment id, or null |
 | `/api/irc?channel=<login>`       | GET with a WebSocket upgrade | The chat of one channel, in the form of Twitch IRC |
 | `/api/data/<prefix>/<path>`      | GET; POST for 7TV paints | Provider data as JSON, with the header `x-petal-cache` |
 | `/api/status`                    | GET       | Counts as JSON                                       |
@@ -532,6 +536,7 @@ The three windows follow each other. While an answer is fresh it is served witho
 provider. In the window after that, lists that every overlay shares are served at once and refreshed
 in the background. Channel data has no such window: a reload of the overlay must show an emote that
 was added a minute ago. In the last window the stored answer is only served when the provider fails.
+A copy still held after that window is served the same way when the provider fails, instead of a 502.
 
 A 404 of a provider means that the channel has no account there. It is stored for the time in the
 table, and served for up to an hour when the provider fails. Lists that always exist have no such
@@ -606,8 +611,9 @@ twice. A replayed line is shown like any other.
 
 **Provider data.** Every REST request goes to the gateway first. After a network error, a timeout of
 8 seconds, an answer without the header `x-petal-cache`, or a status that is neither a success nor
-404, such as 429 or 503, the same request is sent to the provider directly. A 404 of the gateway is
-the provider's answer and is not retried.
+404, such as 429 or 503, the same request is sent to the provider directly, with the same 8 second
+limit. A 404 of the gateway is the provider's answer and is not retried. A provider 404 is cached
+by the gateway and is not an error: it means the channel has no account there.
 
 **Forced direct mode.** `?direct=1` on the overlay URL switches the relay and the gateway off for
 that overlay. The same holds for a page that was not loaded over http(s), such as a local file.
@@ -630,7 +636,7 @@ log or to Analytics Engine, and they never appear in an error message or in `/ap
 | Overlay connection attachment | Channel, anonymous nick, phase of the handshake, time of acceptance |
 | KV                       | Answers of the providers: lists of emotes, badges and paints, under keys made of the route id and the id of a channel, an emote set or a paint |
 | Analytics Engine         | Counters: the event (`irc-connect`, `irc-refused`, `data`, `hub-tick`, `hub-paused`, `hub-resumed`), the reason of a refusal or a pause, the route id, cache status and layer, the status code, the shard number, per alarm of a hub, how many overlays, channels, connections to Twitch, relayed lines, accepted and refused connections and failed connections to Twitch it counted and, per pause, the count that exceeded a threshold and the threshold. No channel, no address, no name, no text |
-| Workers Logs             | What Cloudflare writes per request and per hub event, kept 7 days: among it the URL, which names the channel in `/chat/<channel>` and `/api/irc?channel=` and carries the options of an overlay link, and the client address. The code adds error messages, which name a path, a route id or a shard number and never the query, and one line when a hub pauses itself, with the reason, the shard, a count and its threshold |
+| Workers Logs             | What Cloudflare writes per request and per hub event, kept 7 days: among it the URL, which names the channel in `/chat/<channel>` and `/api/irc?channel=` and carries the options of an overlay link, and the client address. Alarm invocations are logged by the platform as the scheduled time (`Date.toString()`); the hub's `alarm()` does not log. The code adds error messages, which name a path, a route id or a shard number and never the query; one JSON line `upstream_unavailable` per provider per minute with the provider, the route id and a reason (`timeout`, `network`, `status`, `response`, `abandoned`, `unknown`) and never a channel; and one line when a hub pauses itself, with the reason, the shard, a count and its threshold |
 | Rate limiter             | Counts of requests per client address and endpoint, over one minute, in the memory of Cloudflare's rate limiter |
 | `/api/status`            | Counts per hub, the pause of a hub and the version of the deployment   |
 | Browser, local storage   | The theme choice under `theme`, once the button in the header was used. Not on overlay pages |

docs/DEPLOYMENT.md

@@ -15,7 +15,8 @@ The Worker and everything that belongs to it carry the product's name: the Worke
 | `/setup` (redirects to `/#setup`), unknown paths (404) | The Worker     | One Worker request |
 | `/api/irc` (chat of one channel, WebSocket) | The Worker passes the connection to a `ChatHub` Durable Object, which reads the channel from Twitch | One Worker request and one Durable Object request per connection, plus the time the hub spends in memory |
 | `/api/data/...` (lists of emotes, badges and name paints) | The Worker's data gateway: from memory or KV, otherwise from the provider | One Worker request, usually one KV read, one KV write per refresh |
-| `/api/status` (health)                    | The Worker, which asks every hub | One Worker request and one Durable Object request per hub |
+| `/health`                                 | The Worker, from the version binding only | One Worker request |
+| `/api/status` (relay status)              | The Worker, which asks every hub | One Worker request and one Durable Object request per hub |
 | Emote and badge images, live updates of 7TV, BetterTTV and FrankerFaceZ | The providers, directly from the browser | Never reaches Cloudflare |
 
 An overlay that loads costs one Worker request for the page, one for the chat connection and about a dozen for provider data. While it runs, a chat line costs nothing: messages from a hub to an overlay are not billed, and Cloudflare answers the overlay's keep-alive without running code. What is billed while overlays are connected is the time the hubs spend in memory.
@@ -116,6 +117,7 @@ Open `https://petal.shiftbloom.studio` and check these in a browser; command lin
 | `/chat/<channel>?size=3&names=0` | The overlay with large text and without user names |
 | `/chat/<channel>?demo=1` | The overlay with sample messages, without a chat connection |
 | `/no-such-page`      | 404 page                                  |
+| `/health`            | JSON `{ ok: true, version }` with the deployment id; no hub and no provider is asked |
 | `/api/status`        | JSON; after the overlay of a channel was opened, one hub reports a client and a joined channel |
 | `/api/data/bttv/3/cached/emotes/global` | JSON with the response header `x-petal-cache` |
 

src/lib/chat/gateway.ts

@@ -14,8 +14,7 @@ export const GATEWAY_PREFIXES: Readonly<Record<string, string>> = {
     "https://api.ivr.fi/": "/api/data/ivr/",
 };
 
-/** The gateway itself gives a provider six seconds; longer than that and it is stuck. */
-const GATEWAY_TIMEOUT_MS = 8000;
+export const PROVIDER_FETCH_TIMEOUT_MS = 8_000;
 
 /** On every answer of the gateway, and on nothing else that may answer under its path. */
 const GATEWAY_HEADER = "x-petal-cache";
@@ -33,30 +32,72 @@ export function gatewayUrl(url: string): string | undefined {
     return undefined;
 }
 
+const NOT_FOUND = 404;
+
 async function readJson<T>(response: Response, url: string): Promise<T | undefined> {
-    if (response.status === 404) return undefined;
+    if (response.status === NOT_FOUND) return undefined;
     if (!response.ok) throw new Error(`${url} answered ${response.status}`);
     return (await response.json()) as T;
 }
 
-async function fetchFromGateway<T>(url: string, request?: JsonRequest): Promise<T | undefined> {
-    const deadline = new AbortController();
-    const timer = setTimeout(() => deadline.abort(), GATEWAY_TIMEOUT_MS);
+interface Deadline {
+    signal: AbortSignal;
+    elapsed: Promise<never>;
+    stop: () => void;
+}
+
+function deadlineAfter(milliseconds: number): Deadline {
+    const controller = new AbortController();
+    let stop = () => {};
+    const elapsed = new Promise<never>((_, reject) => {
+        const timer = setTimeout(() => {
+            controller.abort();
+            reject(controller.signal.reason);
+        }, milliseconds);
+        stop = () => clearTimeout(timer);
+    });
+    return { signal: controller.signal, elapsed, stop };
+}
+
+function providerStatus(response: Response): boolean {
+    return response.ok || response.status === NOT_FOUND;
+}
+
+function requireGatewayResponse(response: Response, url: string): void {
+    if (providerStatus(response) && !response.headers.has(GATEWAY_HEADER)) {
+        throw new Error(`${url} is not the gateway`);
+    }
+}
+
+function preventUnhandledRejection<T>(work: Promise<T>): Promise<T> {
+    work.catch(() => undefined);
+    return work;
+}
+
+async function readWithinTimeout<T>(
+    url: string,
+    request: JsonRequest | undefined,
+    read: (response: Response) => Promise<T>,
+): Promise<T> {
+    const deadline = deadlineAfter(PROVIDER_FETCH_TIMEOUT_MS);
     try {
-        const response = await fetch(url, { ...request, signal: deadline.signal });
-        // A 404 from the gateway is the provider's answer. A 404 from whatever else serves
-        // this origin, such as a deployment without the gateway, says nothing about the data.
-        const answer = response.ok || response.status === 404;
-        if (answer && !response.headers.has(GATEWAY_HEADER)) {
-            throw new Error(`${url} is not the gateway`);
-        }
-        // Awaited here, so the deadline covers the body as well.
-        return await readJson<T>(response, url);
+        const response = await Promise.race([
+            preventUnhandledRejection(fetch(url, { ...request, signal: deadline.signal })),
+            deadline.elapsed,
+        ]);
+        return await Promise.race([preventUnhandledRejection(read(response)), deadline.elapsed]);
     } finally {
-        clearTimeout(timer);
+        deadline.stop();
     }
 }
 
+async function fetchFromGateway<T>(url: string, request?: JsonRequest): Promise<T | undefined> {
+    return readWithinTimeout(url, request, async (response) => {
+        requireGatewayResponse(response, url);
+        return readJson<T>(response, url);
+    });
+}
+
 /**
  * Fetches provider data as JSON, undefined when the provider answers 404. The gateway is asked
  * first; whenever it fails to give the provider's answer, the provider is asked directly, which
@@ -71,5 +112,5 @@ export async function fetchJson<T>(url: string, request?: JsonRequest): Promise<
             console.warn("[gateway] no answer, asking the provider directly", error);
         }
     }
-    return readJson<T>(await fetch(url, request), url);
+    return readWithinTimeout(url, request, (response) => readJson<T>(response, url));
 }

src/worker/api.ts

@@ -3,6 +3,7 @@ import { record, recordGateway } from "./analytics";
 import type { Env } from "./env";
 import { handleData } from "./gateway/index";
 import type { HubStatus } from "./hub";
+import { logPlatformError, PlatformError, PlatformEvent } from "./log";
 import { withinLimit } from "./rate-limit";
 import { normaliseChannel, shardOf } from "./relay/shard";
 
@@ -18,7 +19,7 @@ const MAX_SHARDS = 64;
 const HUB_LOCATION: DurableObjectLocationHint = "weur";
 
 /** An overlay gives the relay 15 seconds; answering sooner sends it to Twitch sooner. */
-const HUB_TIMEOUT_MS = 5000;
+export const HUB_TIMEOUT_MS = 5000;
 const STATUS_TIMEOUT_MS = 3000;
 
 const DATA_PREFIX = "/api/data/";
@@ -163,7 +164,7 @@ async function chat(request: Request, env: Env, url: URL): Promise<Response> {
     try {
         response = await within(HUB_TIMEOUT_MS, hub(env, shard).fetch(request));
     } catch (error) {
-        console.error("hub did not take the connection", shard, error);
+        logPlatformError(new PlatformError(PlatformEvent.HubUnavailable, error, { shard }));
         return relayUnavailable(env, shard);
     }
     if (response.status === 101) {
@@ -198,7 +199,7 @@ async function shardStatus(env: Env, shard: number): Promise<ShardStatus> {
         );
         return { ...status, shard, available: true };
     } catch (error) {
-        console.error("hub did not report its status", shard, error);
+        logPlatformError(new PlatformError(PlatformEvent.HubStatusUnavailable, error, { shard }));
         return { shard, available: false };
     }
 }
@@ -223,6 +224,26 @@ async function status(request: Request, env: Env): Promise<Response> {
     return json(body);
 }
 
+const HEALTH_METHODS = new Set(["GET", "HEAD"]);
+const HEALTH_ALLOW = "GET, HEAD";
+
+export const HEALTH_PATH = "/health";
+
+export function health(request: Request, env: Pick<Env, "CF_VERSION_METADATA">): Response {
+    if (!HEALTH_METHODS.has(request.method)) {
+        return refuse("method_not_allowed", 405, { allow: HEALTH_ALLOW });
+    }
+    if (request.method === "HEAD") {
+        return new Response(null, { status: 200, headers: JSON_HEADERS });
+    }
+    return json({ ok: true, version: env.CF_VERSION_METADATA?.id ?? null });
+}
+
+function channelFreeRoute(pathname: string): string {
+    if (pathname.startsWith(DATA_PREFIX)) return "data";
+    return pathname;
+}
+
 /**
  * Handles every request under `/api/`; the rest of the site never reaches this module.
  *
@@ -251,8 +272,11 @@ export async function handleApi(
         if (url.pathname === "/api/status") return await status(request, env);
         return refuse("not_found", 404);
     } catch (error) {
-        // Neither the path nor the query is logged: both can name a channel.
-        console.error("api request failed", data ? "data" : url.pathname, error);
+        logPlatformError(
+            new PlatformError(PlatformEvent.ApiFailed, error, {
+                route: channelFreeRoute(url.pathname),
+            }),
+        );
         return refuse("unavailable", 503, data ? GATEWAY_REFUSED : undefined);
     }
 }

src/worker/entry.ts

@@ -1,5 +1,5 @@
 import nitroWorker from "#petal/nitro-worker";
-import { handleApi } from "./api";
+import { HEALTH_PATH, handleApi, health } from "./api";
 import type { Env } from "./env";
 
 // Wrangler looks for Durable Object classes among the exports of the Worker's main module.
@@ -11,6 +11,7 @@ export default {
     ...nitroWorker,
     fetch(request, env, context) {
         const url = new URL(request.url);
+        if (url.pathname === HEALTH_PATH) return health(request, env);
         if (url.pathname.startsWith("/api/")) {
             return handleApi(request, env, context, url);
         }