Skip to content

fix: run on RHEL 8 kernels (probe for BPF ring buffers, fix two Go TLS programs) - #383

Merged
mayankpande88 merged 5 commits into
mainfrom
fix/ringbuf-feature-probe
Oct 9, 2026
Merged

mayankpande88 merged 5 commits into
mainfrom
fix/ringbuf-feature-probe

Conversation

@mayankpande88

@mayankpande88 mayankpande88 commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Summary

The agent now runs on RHEL 8 and its rebuilds (Rocky, Alma, Oracle Linux 8), whose 4.18 kernels backport BPF ring buffers.

  • Startup check: the agent used to exit on any kernel older than 5.8. It now probes for the BPF ring buffer map type instead, so a backported kernel passes and a kernel without ring buffers still gets a clear error. RLIMIT_MEMLOCK is raised before the probe, as the tracer already did before loading, because kernels before 5.11 charge the probe's map to it. Only a definite "not supported" result is fatal; any other probe error (such as missing privileges) logs a warning and leaves the error to the program loader.
  • Go TLS programs: the RHEL 8 verifier rejected go_crypto_tls_read_enter and go_crypto_tls_write_enter ("invalid read from stack"). Both read a file descriptor that a helper fills in only when it succeeds. The variable is now zero-initialized. Newer verifiers accepted the code because they track that the error path returns first.
  • README: the minimum-kernel line mentions backported kernels.
Engineering detail

How the failing programs were found: I loaded each program of the 4.16 amd64 object on its own on Rocky 8.10 (kernel 4.18.0-553) and read the verifier log. Before the fix, 2 of 42 programs failed (the two above); after it, 0 of 42.

Other kernels: with the regenerated objects, every program still loads on 6.1 (5.12 variant, with and without ctx padding, 40 of 40 programs) and on 5.10 (5.6 variant, 40 of 40).

Memlock and the probe: 5.10 still charges BPF maps to RLIMIT_MEMLOCK, and the tracer only raised it later, at load time. With LimitMEMLOCK=0:

  • Probe-only commit: exited with "the kernel does not support BPF ring buffers ... operation not permitted", on a kernel that supports them. Main's version check would have passed there.
  • Fatal only on "not supported": the agent logged a spurious probe warning, then started.
  • Limit raised first (from review): no warning, normal start.

ebpf.go: regenerated with make build in ebpftracer. Only the embedded objects changed (every variant, both architectures).

CI: gofmt, goimports, vet, golangci-lint, go test (excluding /containers) and the build all pass in a Linux container with Go 1.26.5.

Local e2e: I ran agent binaries built from this branch as systemd services on local VMs.

  • Rocky Linux 8.10 (kernel 4.18.0-553, cgroup v1, systemd 239), with Python 3.6 (OpenSSL 1.1.1k) and Go crypto/tls clients against local HTTP and HTTPS servers:
    • Startup: the agent started and loaded every program. The previous commit (probe only) exited with the go_crypto_tls_read_enter verifier error.
    • Python, alternating HTTP and HTTPS: 854 requests counted, against 428 loop iterations (856 requests) printed by the client at scrape time.
    • Python, HTTPS only: 461 counted against 467 iterations printed. The client's last log line ran slightly ahead of the scrape.
    • Go client: 526 counted for 526 sent.
  • Debian 11 (kernel 5.10, x86_64), LimitMEMLOCK=0: the probe-only version exited with the false "does not support BPF ring buffers" error. The final version logs no probe warning, starts, and serves container metrics.

The agent refused any kernel older than 5.8, the release that added BPF
ring buffers. RHEL 8 and its rebuilds run 4.18 with ring buffers
backported, so they were refused although the programs could load.
Probe for the map type instead; kernels without it still get a clear
error before the programs fail to load.
RHEL 8 kernels reject go_crypto_tls_read_enter and go_crypto_tls_write_enter
with "invalid read from stack": fd is filled in by a helper only on its
success path, and the older verifier can't tell the error path returns first.
…rrors

The probe runs before the tracer raises RLIMIT_MEMLOCK, so on kernels that
still charge BPF maps to it, or without privileges, it can fail for reasons
unrelated to ring buffer support. Those now log a warning and leave the
error to the program loader.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request adds support for distribution kernels that backport BPF ring buffers (such as RHEL 8) by replacing the hardcoded kernel version check with a feature probe for ebpf.RingBuf support. It also initializes the fd variable in gotls.c to satisfy the RHEL 8 BPF verifier. Feedback suggests raising the RLIMIT_MEMLOCK limit before performing the feature probe to avoid spurious warnings on older kernels.

Comment thread main.go
Kernels before 5.11 charge the probe's map to it, so a low limit made the
probe fail with a spurious warning (#383 review).
@blue4209211

Copy link
Copy Markdown
Contributor

LGTM. Probing for BPF_MAP_TYPE_RINGBUF instead of checking the version is the right approach. Raising RLIMIT_MEMLOCK before the probe avoids the false failure on 5.10, and treating only ErrNotSupported as fatal is the right split. The zero-initialized fd is a clean fix for the 4.18 verifier.

One note: only Rocky 8.10 (4.18.0-553) was tested. RHEL 8 minors differ in their BPF backports. The probe catches missing ring buffers, but verifier differences in older minors (8.4/8.6) would only show at load time. Worth stating the tested release in the README or release notes ("tested on RHEL 8.10 and rebuilds"), so users on older minors know.

blue4209211
blue4209211 previously approved these changes Oct 9, 2026
# Conflicts:
#	ebpftracer/ebpf.go
@mayankpande88

Copy link
Copy Markdown
Contributor Author

Merged main and regenerated ebpf.go, which conflicted with #377's ClickHouse changes; nothing else changed since your approval. The regenerated objects differ from main only by this PR's gotls.c fix, and all 42 programs of the merged 4.16 object load on Rocky 8.10 (kernel 4.18). The push dismissed the approval: could you re-approve?

@mayankpande88
mayankpande88 merged commit c4f6bb6 into main Oct 9, 2026
7 checks passed
@mayankpande88
mayankpande88 deleted the fix/ringbuf-feature-probe branch October 9, 2026 10:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants