Skip to content

feat(metrics): add opt-in native edge request metrics - #30

Merged
ALikesToCode merged 1 commit into
integrate/gateway-waves-1-2from
feat/edge-request-metrics
Oct 9, 2026
Merged

ALikesToCode merged 1 commit into
integrate/gateway-waves-1-2from
feat/edge-request-metrics

Conversation

@ALikesToCode

Copy link
Copy Markdown
Owner

Summary

Adds native edge request metrics behind NATIVE_EDGE_METRICS_ENABLED, which defaults to false. While the flag is off, native Worker dispatch, forwarded requests and the Prometheus output are unchanged.

  • With the flag on, authenticated native generations are recorded in the existing D1 usage ledger:
    • covers the codex-easy, linkapi and opencode direct routes;
    • records model, status, duration, token usage and configured cost.
  • Requests forwarded to the Container are not counted again, because the Container ledger keeps ownership of them.
  • Response observation is pull-driven and byte-preserving:
    • no clone, tee or full-body buffer;
    • SSE and JSON parsing is bounded at 64 KiB per event.
  • Each request finalizes exactly once, including on cancellation and transport failure. Finalization uses ctx.waitUntil.
  • A ledger failure never replaces the response.
  • The principal is an opaque SHA-256 identity. No prompt, output, key or key prefix is stored or logged.
  • With PROMETHEUS_ENABLED also on, /v1/metrics/prometheus adds isolate-local native counters with fixed provider and outcome labels, at most 120 series.
  • worker/gateway-lifecycle.mjs adds a static authorize / admit / before_dispatch / observe / finalize hook chain for native dispatch. Hooks are code-registered, with no dynamic loading.

There are no new D1 tables, routes or browser-readable headers. Details are in docs/edge-request-metrics.md.

Integration

The Wave 3 integration PR wires the other Wave 3 features into the native lifecycle:

  • admission leases, cancellation, config revision freshness and rate-limit advice;
  • making those hooks run under their own flags rather than the metrics flag.

Shared registrations

  • Added tests/test_edge_request_metrics_worker.mjs to test:worker.
  • Added worker/gateway-extensions.mjs, worker/gateway-lifecycle.mjs and worker/request-telemetry.mjs to the secret firewall inventory.
  • Added the gateway-extensions.mjs import rewrite to the shared Worker test loader.
  • NATIVE_EDGE_METRICS_ENABLED is read only by the Worker, so it is not forwarded to the Container.

Tests

  • tests/test_edge_request_metrics_worker.mjs: new.
  • Full Python suite: 3,181 passed, 2 skipped. ruff, mypy and bandit are clean.
  • npm run test:worker: 1,247 passed.
  • cf:migrations:check, test:deploy, free routes, client headers and the wrangler deploy --dry-run bundle all pass.
  • tests/test_frontend_build.mjs fails locally because @tailwindcss/cli is not installed. It fails the same way on main.

This is part of Wave 3. It is based on the Wave 2 integration branch (#25), and the Wave 3 integration PR adds the README line.

Record authenticated native Worker generations in the D1 usage ledger and expose isolate-local Prometheus counters behind NATIVE_EDGE_METRICS_ENABLED, default off. Adds a static lifecycle hook chain for native dispatch.
@coderabbitai

coderabbitai Bot commented Oct 8, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration
  • Configuration used: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: a206d667-f270-4e04-9dbe-f8b9b868e867

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Oct 8, 2026 •

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Updated (UTC)
✅ Deployment successful!
View logs
multillm-proxy 220fbde Oct 08 2026, 09:57 PM

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 8, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-08T22:00:17.941123Z 220fbde PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 220fbde9f2

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

async pull(controller) {
try {
const result = await reader.read();
if (result.done) { await finish(); controller.close(); return; }

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Close the response before awaiting ledger finalization

When D1 is slow or unavailable, the terminal read of every enabled native response waits for recordNativeUsage() before closing the downstream stream, so clients using response.text(), response.json(), or reading through EOF inherit D1 latency and can hang even though the complete upstream body has arrived. Since nativeGenerationFetch() already registers the finalization promise with ctx.waitUntil, start finalization without awaiting it before controller.close() so best-effort telemetry cannot delay served responses.

Useful? React with 👍 / 👎.

observer.feed(result.value); controller.enqueue(result.value);
} catch (error) { await finish(options.signal?.aborted ? "canceled" : "transport_error"); controller.error(error); }
},
async cancel(reason) { await finish("canceled"); await reader.cancel(reason); },

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Preserve completion when the OpenCode wrapper cancels

With OPENCODE_EDGE_FETCH=true, NATIVE_EDGE_METRICS_ENABLED=true, and the legacy streaming /opencode/chat/completions route, createOpencodeStreamResponse() returns as soon as it sees [DONE] and unconditionally cancels the observed upstream body in its finally block. This cancellation forces reason="canceled" here even after the observer parsed complete usage and [DONE], so successful requests are persisted as status 499 and inflate error/cancellation metrics; distinguish downstream abandonment from cancellation after a recognized terminal event, or avoid wrapping this internal transform as a client cancellation.

Useful? React with 👍 / 👎.


/** Serialize native telemetry into the existing strict ledger; richer fields need a migration. */
export async function recordNativeUsage(env, event) {
const model = event.model ? `${event.provider}:${event.model}` : null;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Avoid prefixing already-qualified model IDs

When a native request supplies an already-qualified model such as openai:gpt-4, priceUsage() deliberately uses that identifier unchanged, but this serialization stores it as linkapi:openai:gpt-4 (or the corresponding edge provider). That fragments usage summaries under a model name different from the configured pricing key and from the Container accounting path, whose _qualified() helper also leaves IDs containing : unchanged; only prepend event.provider when the model is not already qualified.

Useful? React with 👍 / 👎.

Comment on lines +152 to +154
const known = usage.input_tokens !== null && usage.output_tokens !== null;
const outcome = reason ?? options.outcome ?? (response.status >= 400 || parsed.failed ? "upstream_error"
: parsed.completed && known && hasProviderUsage ? "success" : "unknown");

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Treat embedding output tokens as measured zero

For a native LinkAPI /v1/embeddings response, the standard usage object reports prompt_tokens and total_tokens but has no completion/output-token field. Leaving output_tokens null makes known false, so every otherwise successful embedding response is persisted as status 520 with unknown usage and no configured cost; the existing Container accounting explicitly normalizes embeddings to zero output tokens, and the native finalization path needs the same endpoint-specific normalization.

Useful? React with 👍 / 👎.

Comment on lines +83 to +84
const name = (model.includes(":") ? model : `${provider}:${model}`).toLowerCase();
const price = table?.[name] ?? table?.[`${provider}:*`] ?? table?.["*"];

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Normalize pricing-table keys before lookup

The requested model is lowercased before lookup, but the keys parsed from MODEL_PRICING_USD_PER_MILLION are left unchanged. An existing valid configuration such as {"LINKAPI:MODEL":{"input":1,"output":2}} is normalized and priced by the Container's CostService, while the native path silently records a null cost for the same model; normalize parsed keys to lowercase, as the shared pricing implementation does, so enabling native dispatch does not change pricing based only on key casing.

Useful? React with 👍 / 👎.

Comment on lines +77 to +78
function priceUsage(env, provider, model, usage, known) {
if (!known || !model) return null;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Apply flat image prices without requiring token usage

For a native /images/generations response using a flat-only pricing entry such as {"linkapi:image-model":{"request":0.04}}, standard image payloads commonly contain no token usage. The early known check therefore returns null before the flatOnly handling can run, and finalization records the successful request as status 520 with no cost; this conflicts with the existing MODEL_PRICING_USD_PER_MILLION contract, where request-only prices explicitly do not require provider token counts.

Useful? React with 👍 / 👎.

@ALikesToCode
ALikesToCode merged commit d45aba5 into integrate/gateway-waves-1-2 Oct 9, 2026
5 of 7 checks passed
@ALikesToCode
ALikesToCode deleted the feat/edge-request-metrics branch October 26, 2026 20:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant