Skip to content

fix: port upstream ClickHouse parser and detection fixes (B3 of #369) - #377

Open
mayankpande88 wants to merge 5 commits into
mainfrom
port/upstream-b3-l7-parsers
Open

mayankpande88 wants to merge 5 commits into
mainfrom
port/upstream-b3-l7-parsers

Conversation

@mayankpande88

@mayankpande88 mayankpande88 commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Third batch of #369: the ClickHouse parser and detection fixes from upstream coroot-node-agent, all cherry-picked with -x.

Upstream Change
coroot/coroot-node-agent@a8a084f (Nikolay Sivko) A new ClickHouse query parser replaces the one built on ch-go. Queries that carry settings were lost: their span had an empty db.statement, and the empty parse counted as a parse failure. Garbage length prefixes can't turn into large allocations. The ch-go dependency is gone.
coroot/coroot-node-agent@a64920a / b53b38d (Nikolay Sivko) Stricter eBPF ClickHouse detection: the query header (query ids, user, address, interface) and the response structure are validated, so other traffic on ports 9000/8123 isn't taken for ClickHouse.
coroot/coroot-node-agent@556154b (Nikolay Sivko) An offset bound in that parser, which the verifier needs on older kernels.

Plus one fix of our own, from review (08d1e10): the ClickHouse detection functions no longer read past the payload. The query check could not return a wrong answer, but the response check decided short reads from bytes that were not part of the payload. Every read is now checked against the payload size first. This diverges from upstream.

ebpf.go is regenerated.

Not taken from B3:

  • 2d8fb1b (RabbitMQ/ClickHouse misclassification), eBPF half: this fork already classifies any AMQP frame on 5672 as RabbitMQ before ClickHouse detection, which only runs on 9000/8123. Its test case is included with a8a084f.
  • 6d5addb (drop L7 events with unknown protocols): L7Stats here creates no metric vector for an unknown protocol, and observe skips nil vectors.
Engineering detail

Conflicts:

  • a8a084f: this fork had already hardened the old ch-go parser: a LimitReader, header checks and a recover. Upstream's parser replaces all of it.
  • b53b38d: the fork had added an AMQP basic.publish check to is_clickhouse_query. The stricter header validation rejects AMQP frames anyway.
  • Call site: is_clickhouse_response now takes the read size, so its call in trace_exit_read passes ret.

Bounds checks (08d1e10): the e2e below was re-run with them in: same counts, same statement texts. All 40 programs load on 5.10 and 6.1.

CI: gofmt, goimports, vet, golangci-lint, go test (excluding /containers, including the new ClickHouse parser cases) and the build all pass in a Linux container with Go 1.26.5.

Local e2e: I built agent binaries from this branch and from main and ran both side by side on a local Debian 12 VM (kernel 6.1), each sending spans to its own OpenTelemetry collector. A clickhouse-go v2.46 client ran 20 rounds against a local ClickHouse 25.8 on port 9000. Each round sent a plain SELECT, a SELECT with settings (max_execution_time, max_threads) and a query with a syntax error.

  • Query counts: both builds counted 40 ok and 20 failed queries.
  • Spans: this branch's spans carried the text of all three statements. Main's spans for the query with settings had an empty db.statement all 20 times.
  • Program load: both loaded all 40 eBPF programs on 6.1 without errors. This branch also loaded all 40 on a 5.10 kernel.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request refactors the ClickHouse protocol parser by replacing the external ch-go dependency with a custom, allocation-free Go parser to prevent OOMs on corrupted payloads, and updates the corresponding eBPF tracer logic in C. The review feedback highlights critical safety issues in the eBPF code, specifically potential out-of-bounds reads in both is_clickhouse_query and is_clickhouse_response due to missing bounds checks against buf_size before bpf_read operations. Additionally, in the Go parser, it was recommended to remove an unnecessary unconditional slice truncation that could corrupt valid trailing characters, as the subsequent UTF-8 validation loop already handles partial runes.

Comment thread ebpftracer/ebpf/l7/clickhouse.c
Comment thread ebpftracer/ebpf/l7/clickhouse.c
Comment thread ebpftracer/l7/clickhouse.go
def and others added 5 commits October 8, 2026 17:50
(cherry picked from commit a8a084ff1fccfc091272082e0e79a485f1b475a7)

Conflicts were with this fork's own hardening of the ch-go based parser
(LimitReader, header checks, recover); upstream's parser replaces it and
drops the ch-go dependency. The test function also carries the malformed
length case from coroot/coroot-node-agent@2d8fb1b.
(cherry picked from commit b53b38d0dadbf7dcdae9600db960d94088f2f6bc)

Conflict: this fork had added an AMQP basic.publish check to
is_clickhouse_query. The stricter header validation here rejects AMQP
frames anyway, and ClickHouse detection stays limited to ports 9000/8123.
ebpf.go is regenerated in a later commit.
… on older kernels

(cherry picked from commit 556154b5957118723739b5a3ad8e4740d545627a)
is_clickhouse_query and is_clickhouse_response read fixed offsets
(initial_query_id, initial_address, the compression method, the
exception code) without checking them against the payload size. The
query check could not return a wrong answer, since a final bound
rejects anything that read past the end, but the response check
decided short reads from bytes that were not part of the payload.
Check the size before every read. This diverges from upstream
(b53b38d, 556154b).
@mayankpande88
mayankpande88 force-pushed the port/upstream-b3-l7-parsers branch from 08d1e10 to 8cc3943 Compare October 8, 2026 12:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants