Skip to content

feat: CLL auth/jsonrpc passthrough + WS call accounting on rewritten feat/websocket-support - #30

Draft
snowkide wants to merge 2 commits into
fix/fallback-tier-subscriptionsfrom
feat/cll-on-websocket-support
Draft

snowkide wants to merge 2 commits into
fix/fallback-tier-subscriptionsfrom
feat/cll-on-websocket-support

Conversation

@snowkide

@snowkide snowkide commented Oct 2, 2026 •

Copy link
Copy Markdown

Summary

Rebased 2026-10-07 onto fix/fallback-tier-subscriptions (#31), the version now running in production. Previously stacked on fix/fallback-tip-leader (#29, closed). Old tip 6dfd658e -> new tip 11108fd2. Both commits applied without conflicts on top of #31's three commits (hedge all-missing leg, fallback heads holdback, filter subscriptions on head tiers). Once #31 merges into feat/websocket-support, retarget this PR there.

Original context: feat/websocket-support was rewritten on 2026-09-28: rebased onto upstream 989cfe60 and squashed to 11 commits (old tip kept as backup/feat-websocket-support-2026-09-28). #24 still sits on the old history. This PR re-applies #24's net change onto the new history and adds WS observability. #24 is not modified.

Commit 1: port of #24 (#20 + #21 + #23)

  • forwardedClientId auth strategy. The secret is also accepted as path /<secret>, ?apikey= and header (NewPayloadFromHttp now takes the URL path).
  • jsonrpc passthrough architecture. It now sits next to upstream's new svm architecture in config, defaults, validation, network IDs and the registry, and it's registered as a no-op ArchitectureHandler because the new architecture registry requires one. Its error extractor claims nothing, so errors still go through the EVM extractor, the same as in prod today.
  • "error": null is treated as success, and empty params are omitted (BTC-family / Stellar).

Commit 2: count every WSS call and add per-client metrics

Today network_request_received_total{transport="ws"}, which CLL's rpc_ws_event_count_total is built from, only counts WS requests that reach Project.Forward or eth_(un)subscribe. WS requests rejected earlier are invisible: invalid, method not allowed, auth failure, project or auth rate limit, network unavailable, and batch parse errors.

Metric Labels Purpose
network_request_received_total + transport ported from #20
ws_requests_total (new) category, user, agent_name, outcome every WS JSON-RPC request counted once, including rejected ones. category="n/a" until authenticated (protects cardinality)
ws_subscription_events_total user, agent_name ported; notifications delivered
websocket_subscription_notifications_dropped_total + user, agent_name base's drop counter, now per key
ws_subscriptions_active (new) kind, user gauge
ws_connections_active / ws_connections_closed_total (new) close_code, initiator connection churn
client_request_duration_seconds (new) network, user, transport, outcome per-key latency. chainlink-erpc drops user from all histograms, so network_request_duration_seconds can't be broken down by key

How the two histories diverged (for review)

Test plan (on the rebased tree, 2026-10-07)

  • go build ./.... go vet on changed packages is clean apart from an existing upstream warning in erpc/query_executor_test.go:341 (from feat: unified selection policy and scoring mechanism erpc/erpc#888)
  • go test ./auth ./common ./util ./indexer/... ./telemetry ./clients ./upstream ./architecture/... pass
  • go test ./erpc -run 'TestWebSocket_|TestAdapter_PerClientMetrics|Subscription' -race pass
  • Label count matches the definition at every call site of each changed or new metric (network_request_received_total +transport, dropped-notifications +user/agent_name, the new ws_* metrics, client_request_duration_seconds). A mismatch would only show up as a panic at runtime
  • Full go test ./erpc (running; result will be posted in a comment)
  • Canary on chainlink-erpc with a non-EVM network (BTC-family / Stellar) over HTTPS and WSS

🤖 Generated with Claude Code

@snowkide

snowkide commented Oct 2, 2026

Copy link
Copy Markdown
Author

Full go test ./erpc/ -count=1 on the PR tree: ok github.com/erpc/erpc/erpc 1023.610s (exit 0). TestNetwork_Forward/ForwardLlamaRPCEndpointRateLimitResponseSingle, which #29 lists as failing on 05126d0, also passed in this run.

snowkide and others added 2 commits October 7, 2026 11:03
…cket-support

Re-applies the net change of #24 (#20 + #21 + #23) onto the 2026-09-28
rewrite of feat/websocket-support (stacked on #29):

- auth: forwardedClientId strategy; secret accepted as path /<secret>,
  ?apikey= query and header (NewPayloadFromHttp takes the URL path)
- architecture: jsonrpc passthrough for non-EVM HAProxy upstreams, now
  coexisting with upstream's svm architecture and registered as a no-op
  ArchitectureHandler (required by the new architecture registry); its
  error extractor defers to the EVM one, as before
- json-rpc: treat "error": null as success; omit empty params

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…cy metrics

- network_request_received_total gains transport (http|ws); WS ingress
  sets it on single and batch requests and on eth_(un)subscribe
- ws_requests_total{category,user,agent_name,outcome}: every JSON-RPC
  request over WebSocket counted once, including requests rejected before
  the network (invalid, method_not_allowed, unauthorized, rate_limited,
  network_unavailable, invalid_batch). category stays "n/a" until the
  client is authenticated so unauthenticated clients cannot mint series
- ws_subscription_events_total{user,agent_name}: notifications delivered
- websocket_subscription_notifications_dropped_total gains user and
  agent_name (labels frozen at subscribe time, no network lookup per drop)
- ws_subscriptions_active{kind,user}, ws_connections_active,
  ws_connections_closed_total{close_code,initiator}
- client_request_duration_seconds{network,user,transport,outcome}: per-user
  end-to-end latency without the category/vendor/upstream dimensions, for
  deployments that drop user from network_request_duration_seconds

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@snowkide
snowkide force-pushed the feat/cll-on-websocket-support branch from 6dfd658 to 11108fd Compare October 7, 2026 09:14
@snowkide
snowkide changed the base branch from fix/fallback-tip-leader to fix/fallback-tier-subscriptions October 7, 2026 09:14
@snowkide

snowkide commented Oct 7, 2026

Copy link
Copy Markdown
Author

Full go test ./erpc/ -count=1 after the rebase onto fix/fallback-tier-subscriptions (tip 11108fd2):

  • Rebased tree: everything passes except TestWebSocket_ShutdownClosesPromptlyAfterInflightRequests (999s, exit 1).
  • Unmodified fix/fallback-tier-subscriptions base, same run: ok github.com/erpc/erpc/erpc 1012.360s.

That test fails on the base too. Running only that test with -count=20: 3/20 fail on this PR, 5/20 fail on the base. The test file is identical on both branches. In each failure, the in-flight eth_getBalance got "result":"0x1" back instead of the delayed 0xabc123 gock mock (ws_server_hardening_test.go:238). So the flaky test is in #31's base and isn't caused by this PR. It looks like another mock answers the request before the delayed eth_getBalance filter mock does.

Still to do: a canary on chainlink-erpc with a non-EVM network (BTC-family / Stellar) over HTTPS and WSS.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant