Skip to content

fix: register uprobe programs before attaching, add opt-in --instrumentation-delay (B4a of #369) - #378

Open
mayankpande88 wants to merge 3 commits into
mainfrom
port/upstream-b4a-uprobe-order
Open

mayankpande88 wants to merge 3 commits into
mainfrom
port/upstream-b4a-uprobe-order

Conversation

@mayankpande88

Copy link
Copy Markdown
Contributor

Summary

First part of B4 (#369): the uprobe-lifecycle fixes from upstream coroot-node-agent that apply here without its uprobe deduplication.

Change Upstream What it does
Register uprobe programs at load port of coroot/coroot-node-agent@2c72586 Uprobe programs were registered inside the attach loop, interleaved with attaching tracepoints and kprobes. Events flow as soon as the first one is attached, and handleEvents was already running. A process handled before its uprobe program was registered failed to attach, was marked as checked and was never retried. It also meant t.uprobes was written while handleEvents read it. Now every uprobe program is registered right after the collection loads, before anything is attached.
--instrumentation-delay coroot/coroot-node-agent@44e3e8e (Nikolay Sivko) Optional delay before attaching Python GIL and Node.js event-loop instrumentation to a new process. Default 0 (off) here; upstream uses 30s. TLS probes aren't affected.

Not taken:

Engineering detail

Why the delay defaults to 0: upstream added it to save attach cost on short-lived processes. Here Node.js tracing is already opt-in (--enable-nodejs-tracing), so the delay mostly covers the Python GIL probes. With about 300 short-lived Python processes a minute, it saved about 2.6 millicores of agent CPU (2290 vs 2134 ms over 60s), at the price of losing GIL metrics for each process's first 30s.

44e3e8e conflicts: upstream's instrumentDone channel belongs to its deduplication, so it isn't taken. The flag line follows this fork's flag style.

CI: gofmt, goimports, vet, golangci-lint, go test (excluding /containers) and the build all pass in a Linux container with Go 1.26.5.

Local e2e: I built agent binaries from this branch and from main and ran them as systemd services on a local Debian 12 VM (kernel 6.1).

  • Delay off vs 30s, same binary: while about 300 Python processes that each live 2s were spawned over 60s, the agent used 2290 ms of CPU with no delay and 2134 ms with 30s.
  • 30s delay: a long-lived Python process had its probes attached 30s after it started.
  • TLS capture: for an HTTPS client (Python, OpenSSL) that was already running when the agent started, this branch counted 86 requests and main 82 in the same 50s window.

mayankpande88 and others added 3 commits October 8, 2026 15:34
Port of coroot/coroot-node-agent@2c72586 ("register uprobes before
reporting the running processes"). Uprobe programs were registered in
t.uprobes inside the attach loop, interleaved with attaching
tracepoints and kprobes. Events start flowing once the first of those
is attached, and handleEvents (already running) could attach TLS probes
for a process before its program was registered: the attach failed, the
process was marked as checked and never retried. Registering them right
after the collection loads also removes concurrent writes to t.uprobes
while handleEvents reads it.
…instrumentation

(cherry picked from commit 44e3e8ef582f04dabe44fa7ed78d0838c5e43aae)

In this fork the delay covers Python GIL probes and the opt-in Node.js and
.NET instrumentation; TLS probes are attached per connection and are not
delayed. Upstream's instrumentDone channel belongs to its uprobe dedupe
and is not taken.
Upstream delays Python GIL and Node.js event-loop instrumentation by
30s to save the attach cost on short-lived processes. Here Node.js
tracing is already opt-in, and with about 300 short-lived Python
processes a minute the delay saved about 2.6 millicores of agent CPU
(2290 vs 2134 ms over 60s), while hiding GIL metrics for each process's
first 30s. Keep the flag and leave instrumentation immediate by
default.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces an instrumentation delay configuration (--instrumentation-delay) to delay Python GIL and Node.js event loop instrumentation after a process starts. It also updates the eBPF tracer to register uprobe and uretprobe programs before tracepoints or kprobes are attached, ensuring no events are missed. Feedback suggests replacing time.After with time.NewTimer in the delay logic to prevent potential timer leaks if the process context is cancelled before the delay expires.

Comment thread containers/process.go
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants