Skip to content

fix(deps): drop cilium/ebpf fork, pin real upstream v0.22.0 - #909

Merged
matthyx merged 1 commit into
mainfrom
fix/ebpf-btf-cache-race
Aug 21, 2026
Merged

fix(deps): drop cilium/ebpf fork, pin real upstream v0.22.0#909
matthyx merged 1 commit into
mainfrom
fix/ebpf-btf-cache-race

Conversation

@matthyx

@matthyx matthyx commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

Overview

trace_network (and potentially other gadgets) intermittently fails to start with:

error="starting operators: starting operator \"oci\": starting operator \"ebpf\":
creating eBPF collection: program ig_trace_net: apply CO-RE relocations:
load BTF for kmod button: rebase split spec: raw BTF differs"

This leaves ContainerProfile.spec.egress/.ingress permanently null while other tracers (execs, opens) work fine. Caught via kubescape/helm-charts#904 failing armosec/system-tests' network_policy_known_servers/network_policy_multiple_replicas E2E tests.

Root cause: the pinned github.com/matthyx/ebpf fork (an intermediate, unmerged snapshot of cilium/ebpf#1988) carries a global weak-pointer kernel BTF cache. A race between one gadget's Cache.Kernel() call and a concurrently-loading gadget's Cache.Module() call causes rebaseDecoder to reject a rebase purely on stale pointer identity — unrelated to which specific kernel module or gadget hits it.

Already root-caused and fixed in the sibling private repo armosec/private-node-agent (PR #520, merged). This PR ports that fix here.

The fix

  • Drops the replace github.com/cilium/ebpf => github.com/matthyx/ebpf ... directive entirely; bumps the direct github.com/cilium/ebpf requirement to real upstream v0.22.0 (no replace needed — the fork's main carries zero unique commits beyond real upstream v0.22.0, confirmed via the fork's own PR that produced this fix).
  • Bumps the replace github.com/inspektor-gadget/inspektor-gadget => github.com/matthyx/inspektor-gadget ... pin to the commit built against real cilium/ebpf v0.22.0 (fixes two internal API breaks inside that fork: btf.NewBuilder's new signature, and ebpf.VariableSpec's field-based redesign). Confirmed this is still the fork's latest commit — no newer one exists.

v0.22.0 is several minors ahead of the previous v0.20.0 pin; this repo's own code doesn't call btf.NewBuilder, CollectionSpec.RewriteMaps, or ebpf.VariableSpec directly (those breaks only affect code inside the vendored inspektor-gadget fork itself, already fixed there), so no node-agent source changes were needed.

How to Test

  • go build ./..., go build -tags stacktrace ./..., and go build -race ./... all pass clean.
  • go vet ./... passes clean.
  • go mod tidy is stable (no further changes) and go mod verify reports all modules verified.
  • A deterministic regression test for the BTF-cache race itself can't meaningfully live in this repo: node-agent never calls cilium/ebpf's BTF/Cache APIs directly (its only direct cilium/ebpf import is rlimit, unrelated) — all BTF/CO-RE/cache logic is reached transitively through the vendored inspektor-gadget fork. The regression test for the race lives upstream in that fork (see armosec/private-node-agent#520's description for the test shape).

Related issues/PRs

🤖 Generated with Claude Code

AI-skills: none | cmds: /oh-my-claudecode:autopilot

Summary by CodeRabbit

  • Chores
    • Updated internal components to newer versions.
    • Improved compatibility and stability through dependency maintenance.

The pinned matthyx/ebpf fork (an intermediate, unmerged snapshot of
cilium/ebpf#1988) carries a global weak-pointer kernel BTF cache: a
race between one gadget's Cache.Kernel() call and a concurrently
loading gadget's Cache.Module() call causes rebaseDecoder to reject a
rebase purely on stale pointer identity. This made trace_network fail
to start intermittently with "apply CO-RE relocations: load BTF for
kmod <mod>: rebase split spec: raw BTF differs", silently leaving
ContainerProfile.spec.egress/.ingress permanently null.

Real upstream cilium/ebpf v0.22.0 already contains #1988's complete,
structurally-correct fix (Cache is fully self-contained per-instance,
no global cache, no weak.Pointer, no identity check to race against),
so this drops the fork replace entirely and pins straight to v0.22.0.

v0.22.0 is several minors ahead and broke two call sites inside the
matthyx/inspektor-gadget fork (btf.NewBuilder's signature, and
VariableSpec's field shape) - fixed there already, so this also bumps
that fork's replace pin to the commit with those compat fixes.

Root-caused and fixed upstream in armosec/private-node-agent#520.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01PdtsXi1wvbfDu9biC21yRu
@matthyx matthyx added ai-assisted Created through Armosec AI tooling (armosec-shared-rules plugin) release Create release labels Aug 21, 2026
@coderabbitai

coderabbitai Bot commented Aug 21, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 199a1ec5-3952-4f15-81cf-a9ba56a95f3c

📥 Commits

Reviewing files that changed from the base of the PR and between 1601737 and 7422132.

⛔ Files ignored due to path filters (1)
  • go.sum is excluded by !**/*.sum
📒 Files selected for processing (1)
  • go.mod

Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review.


📝 Walkthrough

Walkthrough

The Go module manifest updates the eBPF and TOML dependency versions. It also changes the Inspektor Gadget replacement and removes the separate eBPF replacement.

Changes

Go dependency alignment

Layer / File(s) Summary
Dependency versions and replacements
go.mod
Updates github.com/cilium/ebpf to v0.22.0, updates indirect github.com/BurntSushi/toml to v1.6.0, changes the Inspektor Gadget replacement, and removes the eBPF replacement directive.

Estimated code review effort: 1 (Trivial) | ~3 minutes

Merge Risk: ⚪ Minimal · up to 74221

The PR replaces the forked eBPF dependency with upstream v0.22.0 and updates the compatible Inspektor Gadget pin; reported validation checks pass, and no actionable merge-blocking risk remains beyond normal review.

Suggested reviewers: entlein

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes the main dependency changes: removing the eBPF fork and pinning upstream version v0.22.0.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0 files. (1 skipped: 1 unsupported.)
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/ebpf-btf-cache-race

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@matthyx

matthyx commented Aug 21, 2026

Copy link
Copy Markdown
Contributor Author

Reviewed the go.mod/go.sum diff — the dependency swap itself is clean:

  • No leftover matthyx/ebpf references anywhere in go.mod/go.sum after the change.
  • cilium/ebpf v0.22.0 h1/go.mod sums are present and consistent.
  • inspektor-gadget fork pin bumped to the commit with the two compat fixes (btf.NewBuilder, VariableSpec).
  • Rationale in the PR description matches the actual diff (only go.mod/go.sum touched, no source changes needed).

Blocker: the DCO check is failing — the single commit has no Signed-off-by: trailer (it only carries Co-Authored-By: Claude Sonnet 5 <[email protected]>). Please amend the commit to add Signed-off-by: Matthias Bertschy <[email protected]> so the DCO check goes green. (Note: I can't formally approve/request-changes via GitHub review since this is authored by the same account I'm running as — flagging here instead.)

Component tests were still finishing at review time, but everything completed so far is passing, and CodeQL/CodeRabbit/GitGuardian are green. Once DCO is fixed and tests land green, this is good to merge.

@github-actions

Copy link
Copy Markdown

Performance Benchmark Results

Node-Agent Resource Usage
Metric BEFORE AFTER Delta
Avg CPU (cores) 0.164 0.179 +9.2%
Peak CPU (cores) 0.174 0.185 +6.0%
Avg Memory (MiB) 319.228 273.956 -14.2%
Peak Memory (MiB) 324.492 283.008 -12.8%
Dedup Effectiveness

No data available.

@matthyx
matthyx merged commit 4c4de11 into main Aug 21, 2026
70 of 73 checks passed
@matthyx
matthyx deleted the fix/ebpf-btf-cache-race branch August 21, 2026 11:44
@matthyx matthyx moved this to To Archive in KS PRs tracking Aug 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ai-assisted Created through Armosec AI tooling (armosec-shared-rules plugin) release Create release

Projects

Status: To Archive

Development

Successfully merging this pull request may close these issues.

1 participant