feat(llm): accurate LLM usage capture; fix TLS and HTTP/2 parsing

[mayankpande88]

Reworks LLM traffic capture so usage numbers are accurate, and fixes the HTTP/2 parsing defects that blocked it.

Why

Measured against an LLM-calling service's own usage metrics, the previous pipeline:

• reported no token usage for a Go service calling Gemini over HTTP/2;
• labelled every request's model unknown;
• saw about a quarter of the requests.

Usage sits at the end of a response body or in the last event of a stream. The generic L7 path keeps one bounded request and the first read of the response, so usage was structurally out of reach.

What changes

TLS single source (eBPF). A TLS connection is seen as plaintext by the uprobes and as ciphertext by the socket syscalls, under the same pid+fd. Once a connection was detected as HTTP/2, its ciphertext went into the same parser. A TLS record header reads as an HTTP/2 frame header, which caused:

• phantom PUSH_PROMISE, PRIORITY and extension frames;
• partial frames that swallowed the real ones after them;
• HPACK desync for the rest of the connection.

Connections handled by a TLS hook are now marked, and their socket-level events are skipped in the kernel (node_agent_l7_tls_ciphertext_skipped_total).

LLM capture channel.

• Identifying connections: by TLS SNI; by API path for gateways and self-hosted servers; then by destination, so per-request connections to a known endpoint are captured from their first write.
• Capture: the kernel copies every read and write on these connections into the L7 ring buffer, so they stay ordered with the generic events that preceded the mark.
• Reassembly: a new llm package rebuilds HTTP/1.1 and HTTP/2 exchanges with net/http and x/net/http2 and decodes gzip, deflate, zstd, SSE and AWS event streams. It handles responses the client stops reading (at SSE [DONE]) and streams cut off by a reset or a closed connection.
• Usage: typed decoders per wire format: OpenAI and compatible, Anthropic, Gemini/Vertex, Bedrock (including InvokeModel token headers), Cohere. Token types are disjoint (input, cached_input, cache_write, output, reasoning), so they sum to billed tokens.

HTTP/2 fixes found along the way:

• Raw kernel time was clamped to zero for any uptime over an hour, so HTTP/2 durations were always 0 and the parser's stale-stream GC never ran.
• After a read truncated by the 4095-byte capture limit, the parser now skips the exact remainder of the cut frame instead of parsing it as frame headers.
• HTTP/1 and HTTP/2 now share one metric series set. Real HTTP/2 durations exposed a duplicate-series collision that failed the whole scrape. /metrics also continues on gather errors now.

Removed:

• the DNS/IP LLM detector;
• the connectionless HTTP/2 path;
• regex usage extraction and the static price table;
• payload accumulation in the shared HTTP/2 parser.

Opt-in: --enable-llm-capture

LLM capture is off by default (--enable-llm-capture / ENABLE_LLM_CAPTURE=true), like Node.js and .NET tracing:

• Off: costs one array lookup per socket read or write in the kernel, and nothing in userspace. The LLM metric families are not registered.
• On: adds a destination check on each new connection's first write, and full capture and parsing of LLM API connections only.

The TLS ciphertext skip and the HTTP/2 fixes are unconditional: they reduce L7 work for everyone. Deployments that want LLM metrics need the flag set; the Helm chart needs a value for it.

Metric changes (breaking for LLM dashboards)

Before	After
container_llm_token_usage_total, container_llm_cached_input_tokens_total	container_llm_tokens_total{gen_ai_token_type}
container_llm_errors_total	derive from container_llm_requests_total{http_response_status_code}
container_llm_cost_usd_total	removed: join tokens with a price list
container_llm_tokens_per_second, container_llm_tool_calls_total, node_agent_llm_sni_tags_total	removed
–	node_agent_llm_capture_total{outcome}, node_agent_llm_capture_drops_total

docs/llm-observability.md is rewritten to match.

Validation

• Replay tests record real Go net/http and x/net/http2 client traffic at the TLS boundary (and at the socket for plain HTTP) against local servers with known usage, then assert exact token counts. Each test was checked to fail when its fix is reverted.
• Deployed from this branch to a test cluster (Linux 6.8, amd64) and measured over 30 minutes with no scrape gaps:
  • against the calling service's own per-pod usage metrics: requests 98.4%, input tokens 99.4%, output tokens 98.4%, cached tokens exact on the busiest model;
  • against an LLM gateway's access log: 98.3% of requests.
  • Cluster-wide: completed external HTTP/2 streams went from 0 to about 2.5k per 30 min, and HPACK errors dropped 55%.
• Not yet validated: kernels older than 6.8, and arm64.

Reviewing

Suggested order:

1. ebpftracer/ebpf/l7/l7.c and llm_capture.c (kernel)
2. llm/ (parsing and usage, tests in replay_test.go)
3. containers/llm_capture.go (glue)
4. the ebpftracer/l7/http2.go and containers/l7.go fixes

ebpftracer/ebpf.go is regenerated from the C sources.

[gemini-code-assist]

Code Review

This pull request introduces a mechanism to skip processing socket-level ciphertext events when a TLS library hook (such as Go crypto/tls or OpenSSL) is already delivering the connection's plaintext. It adds a tls field to connection tracking, updates L7 events to report whether they are TLS plaintext, and introduces a new Prometheus metric node_agent_l7_tls_ciphertext_skipped_total to track skipped ciphertext events. Feedback on the changes points out a high-severity issue in ebpftracer/ebpf/l7/l7.c where mark_tls is called on a stack-allocated connection fallback (conn_on_stack) without persisting it to the active_connections map, which would prevent subsequent ciphertext events from being skipped.

[mayankpande88]

/gemini review

[gemini-code-assist]

Code Review

This pull request refactors the LLM observability pipeline by replacing the previous DNS-based detection with an eBPF-based connection capture mechanism that copies plaintext reads and writes directly to userspace. It introduces a new llm package to handle stream reassembly, decompression, and token usage extraction for multiple providers, alongside new self-observability metrics and eBPF tracer enhancements to skip socket-level ciphertext events on TLS connections. The review feedback highlights critical concurrency issues in containers/container.go where several capture-related methods are called without holding the required container lock, as well as resource leaks in llm/body.go due to unclosed decompression readers.