Skip to content

feat(routing): add opt-in canary and A/B auto-route cohorts - #54

Merged
ALikesToCode merged 3 commits into
feat/context-pagesfrom
feat/canary-traffic
Oct 9, 2026
Merged

ALikesToCode merged 3 commits into
feat/context-pagesfrom
feat/canary-traffic

Conversation

@ALikesToCode

Copy link
Copy Markdown
Owner

Summary

Adds opt-in canary and A/B cohorts for auto routes. With CANARY_TRAFFIC_ENABLED off (the default), and for any route without an enabled canary policy, auto-route dispatch, headers and storage are unchanged.

  • Configuration:
    • An admin saves the route's candidates and its canary policy together through the existing PUT /admin/auto-routes: enabled, mode (shadow or live), weights (baseline + candidate = 100), salt_revision and approved_candidates. Approved candidates must already be in the route.
    • Invalid configuration returns 400 before saving.
    • Concurrent edits get 409 (expected_updated_at locally, current_revision with durable revisions).
    • Weights never change automatically.
  • Assignment:
    • HMAC-SHA256 under the existing JWT_SECRET, over ["multillm-canary-v1", principal, session, route, salt_revision].
    • The principal comes from authenticated request state, and the caller can't choose it. The session uses the same lookup as prompt cache affinity.
    • No session, no verified principal or no secret means the baseline cohort, never a random per-request choice.
  • Modes:
    • Shadow records the proposed cohort and order, and dispatches exactly the baseline request, with no extra provider call.
    • Live puts approved candidates first for the candidate cohort. Health ordering and the permission, budget, credential, cooldown and deadline checks still run afterwards and win.
  • Visibility: responses on opted-in routes carry X-MultiLLM-Canary-Cohort: <baseline|candidate>; mode=<shadow|live>. Content-free counts by route, cohort and mode are kept per process, up to 800 counters.
  • Storage: migration 0025_canary_traffic.sql stores only the reviewed policy per route and the route's update timestamp. It stores no session, principal or HMAC output. A missing table or an unreadable policy falls back to the baseline with one warning, never a 500.

Note: a session ID is a caller-supplied hint, so a caller can change cohorts by changing their session. This is a traffic split, not an access control, and the doc says so.

Details are in docs/canary-traffic.md.

Shared registrations

  • tests/test_canary_traffic_worker.mjs in test:worker.
  • worker/canary-assignment.mjs in the edge module inventory.
  • CANARY_TRAFFIC_ENABLED in the Container env keys.
  • canary_traffic in REQUIRED_D1_TABLES.
  • X-MultiLLM-Canary-Cohort in both CORS expose lists.

Stacked on #50

Migration numbers must be contiguous in each PR, so this branch is stacked on feat/context-pages (#50, migration 0024) and targets it. Merge #50 first.

Left for the Wave 5 integration

  • Mount handleCanaryRouteOperation in worker/auto-routes-d1.mjs:handleAutoRoutesRequest, so durable (D1) saves can store policies. Local storage works now.
  • Apply prepareCanary to native Worker generations in nativeGenerationFetch.
  • Feed cohort metadata into MetricsService._request_context_metadata and observeNativeResponse.

Tests

  • tests/test_canary_traffic.py and tests/test_canary_traffic_worker.mjs cover:
    • exact session determinism and a shared Python/Worker test vector;
    • weights and salt revisions;
    • baseline for a missing session;
    • zero extra provider calls in shadow mode;
    • eligibility winning over cohort order;
    • verified principal and session lookup;
    • atomic saves with rollback on storage failure;
    • stale-revision 409;
    • unchanged dispatch when disabled.
  • Full Python suite: 3,970 passed, 2 skipped. ruff, mypy and bandit are clean.
  • npm run test:worker: 1,486 passed. npm run test:integration: 7 passed.
  • migrations:check (25 contiguous, additive migrations), test:deploy, free routes, client headers and the wrangler deploy --dry-run bundle all pass.

Apply the migration before enabling durable canary routing: npx wrangler d1 migrations apply multillm-intelligence --remote.

CI notes

  • GitHub python/node jobs: the dependency-audit steps fail on every branch, including main. fix(deps): clear the CI dependency audit advisories #42 fixes that.
  • Workers Builds: fails on this stack because the Wave 3 base adds the AdmissionCoordinator Durable Object migration, which wrangler versions upload cannot apply. This is expected until wrangler deploy runs on main.

This is part of Wave 5. The README line comes with the Wave 5 integration PR.

Behind CANARY_TRAFFIC_ENABLED, an auto route can carry a reviewed canary policy with fixed baseline and candidate weights. Sessions are assigned to a sticky cohort by an HMAC of the authenticated principal, session, route and salt revision; requests without a session stay on the baseline. Shadow mode records the proposed order without changing dispatch, and live mode puts approved candidates first for the candidate cohort while every existing filter still applies. Responses on opted-in routes carry X-MultiLLM-Canary-Cohort. Canary policies are stored in a new canary_traffic table (migration 0025).
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Oct 9, 2026 •

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Updated (UTC)
❌ Deployment failed
View logs
multillm-proxy 6150133 Oct 09 2026, 02:34 AM

@coderabbitai

coderabbitai Bot commented Oct 9, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration
  • Configuration used: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 5540e497-424a-42a8-9a9b-a98f7be432d6

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@ALikesToCode

ALikesToCode commented Oct 9, 2026 •

Copy link
Copy Markdown
Owner Author

Merged the updated feat/context-pages (which now includes the Wave 4 integration glue, cd4ac5f) into this branch. CI after the merge: 4,018 Python passed, 2 skipped; 1,512 Worker (one timing-sensitive test failed while three CI runs shared the machine and passed on a solo rerun); 7 integration; 25 contiguous migrations; deploy, free routes, client headers and the wrangler dry run pass.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant